help with the crowdsec unifi collection
is there a way too get the collection too work for a UDM-SE they have API access now.
139 Replies
Important Information
Thank you for getting in touch with your support request. To expedite a swift resolution, could you kindly provide the following information? Rest assured, we will respond promptly, and we greatly appreciate your patience. While you wait, please check the links below to see if this issue has been previously addressed. If you have managed to resolve it, please use run the command
/resolve or press the green resolve button below.Log Files
If you possess any log files that you believe could be beneficial, please include them at this time. By default, CrowdSec logs to /var/log/, where you will discover a corresponding log file for each component.
Guide Followed (CrowdSec Official)
If you have diligently followed one of our guides and hit a roadblock, please share the guide with us. This will help us assess if any adjustments are necessary to assist you further.
Screenshots
Please forward any screenshots depicting errors you encounter. Your visuals will provide us with a clear view of the issues you are facing.
ยฉ Created By WhyAydan for CrowdSec โค๏ธ
no, the issue is that CrowdSec doesn't support CEF syslog https://github.com/crowdsecurity/hub/issues/940
GitHub
Unifi OS 3 aren't being parsed ยท Issue #940 ยท crowdsecurity/hub
Describe the bug Unifi OS 3 and newer logs are not being parsed correctly, resulting in detection scenarios such as port scanning not working correctly. To Reproduce Install the unifi collection cs...
Reading into all that, it looks like they are adding a disable magic syslog RFC parsers, slated for 1.6.6?
yup, it should deal with about 90% of the issues until CEF support is added in 1.7.0
So it likely wonโt fully work until 1.7.0
yup it's just a workaround
Did this happen due to the new zone based firewall?
no this has been an issue for a few years
look at the date the issue was opened
Ahh, I see sorry just now switched to full UniFi so was never concerned about it til now
Hi people, sorry if it has already been asked, but I encounter the same issue, can someone give me a quick recap about what is working and what is not, I am a little bit lost
Is CEF support added in the 1.7.0 release?
Or you meant in further 1.7.x release?
Currently Unifi is not working. They have been working on it for a while though...
So there is 3 issues:
- Unifi switching over to CEF means you cannot use our Syslog acquistion (even with disable magic RFC since it not a syslog packet). You must use Rsyslog to collect the entries and write them to a file.
- We do not have a CEF (s00) or a Unifi CEF aware (s01) parser to turn the data into something actionable with scenarios
- Not all logs from unifi are CEF, I have updated the unifi (s00) to work with these logs but its pending to be merged since I needed more data to be sure I can merge it
I have some unifi at home, I can be a guinea pig ๐
I am working on the CEF parser and to some extent the unifi stuff, I have the "suricata" examples but anymore add them to issue please ๐
I have unifi also well UDM but I dont expose anything so all I have is admin logs ๐
same
I enabled some syslog on my firewall rules, so everytime someone each a service hosted behind it logs it
I can give you sample if you need
Just ping me with the sample log you need
For me I have no CEF format available
This is how it looks like on latest version
You dont get to choose its "whatever unifi wants to send depending on the event type" hence why its tricky
Yeah cause my dream machine send in a syslog like format
atleast for firewall rules, hit logs
it sends in syslog for firewall events AFAIK, and intrusion attempts its CEF
Yeah
But I don't know why firewall events doesn't get parsed
Cause most likely its the issue with relayed packets
are you already using rsyslog?
On CrowdSec VM or on unifi ?
VM
or both
I think I do have rsyslog installed, but I used this to configure https://app.crowdsec.net/hub/author/crowdsecurity/collections/unifi and set my CyberSecure to send logs on port 4242
Collections, AppSec Rules & Configurations | CrowdSec Hub
Manage collections, configurations, remediation components, and AppSec rules with CrowdSec Hub. Streamline security with tools and integrations for enhanced protection.
I configured nothing more, that might be the issue
I am still working on the CEF stuff, but you can see in the PR the example rsyslog configuration
Thanks, I will look at it, I am still very new to this domain
but its very WIP and may not work
For my own curiosity, if we use acquid.d with "syslog", it has to comply to RFC right ?
that is why it is advised to use rsyslog with a custom parser ?
Yes as unifi doesnt comply with RFC standards
@iiamloz Are we able to test the new Unifi implemantation (per https://github.com/crowdsecurity/hub/issues/940#issuecomment-3253838553) or do we need to update to some beta version or such first? (Running 1.7.0)
GitHub
Unifi OS 3 aren't being parsed ยท Issue #940 ยท crowdsecurity/hub
Describe the bug Unifi OS 3 and newer logs are not being parsed correctly, resulting in detection scenarios such as port scanning not working correctly. To Reproduce Install the unifi collection cs...
I already merged the firewall stuff
so that will be on 1.7
but the CEF stuff is not merged so if you want to test you are going to have to download things manually and set them up
I can write a quick tldr
yes pls
added tldr to comment
Will set it up and report back
Just to be sure; the rsyslog config & logrotate config are these on the syslog server or on the Unifi device?
syslog server where CrowdSec is installed
I just pushed an updated rsyslog example, im trying to make sure it doesnt break with existing rsyslog configs
@PintjesBier seems the file didnt update in the link go to actual file ๐ https://github.com/crowdsecurity/hub/pull/1457/files#diff-30a462c491efedfff89be5ac747b88961a2b9829c48afef8fd02bb0cb76f40bf
I have set it up (though somewhat janky, due to restrictions on the platform I'm using). Will let it run for a while and see, however I'm unsure if Unifi is outputting the correct logs...
edit: not the correct logs, but IF they're even outputting the logs
Seems like 'logs' from the insights tab are not logged to syslog.
within the unifi section for SIEM, did you enable most / some of the categories?
all of them on debug now... Still nothing afaik
have tog o now
just make sure there nothing blocking like a firewall on the syslog server and/or you adapted the variable to allow unifi IP
I am receiving logs from Unifi. However Unifi is just not publishing the 'security CEF logs' to the syslog server for some reason...
Will test further tommorow
Will it be possible in the future to handle also iptables logs?
I mean out of the box, without requiring too many tinkering
What do you class as tinkering? cause the Rsyslog stuff is going to have to be done just to get the logs to crowdsec, the collection / parsers / scenarios (eventually) will be as simply installing/updating the unifi collection.
It's much more about unifi collection
yeah that will be as simply as
cscli hub update && cscli hub upgrade if you already have it installedIf on the collection page it just gives the rsyslog config, it should be easy
but since its not merged it gotta be manually installed for testing
Sweet
Yeah thats my plan, plus I am going to follow up with a blog post to make sure people find it
Nice
Crowdsec looks very hard
But in fact, it's not that hard when everything clicks
I just got misled, because the collection page for unifi wasn't giving much infos
it like a car, if you just want to drive it, its pretty simply. As soon as you want to make modifications you gotta learn whats under the hood.
Exactly
Yeah cause its not merged yet, as soon as I get working version with people testing it then it will be updated.
But I have to agree, this year unifi change everything at each update
They changed features and layout every single update in the firewall area
Plus I gotta think through the scenarios, so once parsing out of the way then its how we turn the data into something actionable
Yep, the new flows section is nice
Yeah it's much more useful
I also migrated my legacy rules into the zone thing
cause before everything was buried in settings
But in the latest update, they broke the port forwarding thing
nice...
One page is missing, and some links does nothing
Port forwarding is now part of the new rules UI
but if you want to update a rules, if you click "open port forwarding" it does nothing
So you have to find the original rules by hand
Loz, when you say Unifi is sending out security related messages in CEF format... does that include rule hits? Because I receive those in 'normal syslog format' (?)
I think does aren't security related
those*
I think security are the ones also logged in flows
Rule hits are logged in flows... But you probably mean the IDS/IPS detections, no?
Yeah those
But I can't be sure, I couldn't find any documentation on what is logged in which category on the unifi documentation
In the rsyslog configuration there is 2 files, one for syslog lines and one for CEF.
Security Firewall, Honeypot, Intrusion Prevention Threat Detected and Blocked, Honeypot Triggered, Blocked by Firewall
Yeah, it's very very very confusing... I have reached out to Unifi support again and they forwarded it to the dev team... (again)
Ubiquiti Help Center
UniFi System Logs & SIEM Integration
UniFi provides a robust, structured activity logging system that gives you full visibility into your networkโs health, performance, and security. These logs capture key eventsโsuch as connectivity ...
Apparently it's only threat and honeypot
So I assume rules hits still use syslog
Yeah, due to not being able to change my rsyslog config (until I deploy a proper syslog server, which I'm planning to do) I have them both spit out in the same file and have the crowdsec agent run both 'types' on them... For the time being I don't mind the extra overhead.
That should work, right?
for
unifi it should yes, just make sure you have updated the parser to latest that I merged yesterdayI have updated the Unifi parsers. However can't find the CEF parsers on github. Did anything change on those as well?
I need to push a fix for the
unifi-cef, but you can find all files linked in the pull request https://github.com/crowdsecurity/hub/pull/1457alright, looking good after updating ๐ค๐ฝ
push fixes to unifi-cef file, as I took my example from a previous version and it seems unifi added a timestamp to all logs
I noticed, waiting for github tests to complete and will update afterwards! Should I remove the syslog parsing for the time being so we can see if CEF is correctly parsing?
Nah its all good, you just have the double read overhead but we can see if it parsing via
cscli metricsoh yah, forgot there was a parsers metric as well lmao
updated version running now
so my idea for scenarios right now are :
- Trigger bucket for medium and high "UNIFIrisk" rating
- Leaky bucket for low "UNIFIrisk" rating in the ball park of (3/60 seconds)
Looks good to me. However I strongly suggest adding a scenario for rule hits as well...
If an IP gets blocked several times on the same rule (or even different rules) I don't think it has good intentions... I have 5783 'Block internet to gateway' hits since this morning...
but isnt block to internet to gateway simply just a inherit drop rule? or let me know what that rule is cause I dont have that in my UDM cause im not exposing anything
By default they only block 'invalid' traffic to the gateway I think. I have added 'new' traffic as well.
this is the similar rule I guess
which is a "catch all" for any ports that are not forwarded?
You're right
Do you think it would be possible to detect port scan?
A kinda similar scenario as the iptables one's
Unifi detects port scanning
And once again if they would hit the cleanup rules several times on a different port I would guess they're port scanning as well
the base unifi collection thats already released already detects port scanning, as it uses the iptables scenarios
So for unraid I just need do a path for
/var/log/ and point this to the file location on my syslog? In my case on unraid I believe my only choice is syslog-ng containerI also use unraid and use it's built in syslog server. However if I were to redo it, I would setup a dedicated syslog server
Sadly for unraid it seems thereโs either the built in, grafana stack or syslog-ng by linuxserver
You can run every docker there is out there (not only those from the app-store) and you can always setup a VM
But if your unraid server is the only one you have, I would go with a docker
What docker would u recommend? Syslog-ng works but no gui
I haven't looked into the different syslog servers. Just go with whichever you feel most comfortable I guess
Why would u change from the built in?
Because, I don't deem it the NAS's duty to be a syslog server + the built in has pretty much no configuration
Just an update:
still no CEF logs after 1.5 days of running... However no IDS/IPS detections since as well...
Wondering if I could trigger it manually somehow
You can trigger the IPS rules manually if you run this curl command
curl -A BlackSun http://example.comDid it and it shows up in Network app but not in unifi log file
Doesn't seem to trigger anything for me... Not even in the Unifi app lmao
aha, it showed up. But was quite some delay
edit: doesn't seem to get parsed, shows up in the logfile however. This is the CEF line:
nice
CEF: 0 there a space :facepalm:
unifi doing there own thing as usual as stated in RFC, the CEF:<version> should have no space
ahhhh f...
Okay working on fixing it, did it go into the CEF file as expected?
I don't have the seperate file, so can't tell you that...
Okay, I will add to the CEF parser a optional catch for the syslog date time
Awesome Loz! Thanks for these swift updates!
Glad to see this moving forward
Added your examples (with some stuff sanitised) and got it working with the syslog prefixes, so if you update your files to the new ones within the PR it should now parse.
again no scenarios so it wont do anything, but at least for now we just testing parsing
on it!
Got parsed now!
and did you have
unifi-cef installed?yessir
Manually updated from the PR
but you dont see that inside parser metrics? ๐
Huh, the cef-logs which says '1 parsed' (?)
Yes, but there another parser at
s01 called unifi-cef inside the PR, which should have also parsedyeah, I have updated that as well... Don't see any mention of that one...
edit: yes I do
"2"! so at least something were parsed
Yesshhh
Do I need to install the unifi-cef manually? As I cannot find it in the parser list?
I have the unifi-logs installed:
name: crowdsecurity/unifi-logs
file_name: unifi-logs.yaml
path: parsers/s00-raw/crowdsecurity/unifi-logs.yaml
version: "0.2"
dependencies: {}
local_path: /etc/crowdsec/parsers/s00-raw/unifi-logs.yaml
local_version: "0.2"
cscli parsers install child-crowdsecurity/unifi-cef
Error: cscli parsers install: can't find 'child-crowdsecurity/unifi-cef' in parsers
cscli parsers install crowdsecurity/unifi-cef
Error: cscli parsers install: can't find 'crowdsecurity/unifi-cef' in parsers
I have manually created the s00-raw & s01-parse files from the PR (https://github.com/crowdsecurity/hub/pull/1457/files)
They're not published yet, so for now it's manually like this
Im just working on HAProxy stuff, but still got the scenarios burning in the back of my brain ๐
Hey folks, is there anything new regarding unifi integration ?
just waiting for any feedback from people that did what I asked in the issue ๐
What feedback were you looking for? The parsing seems to be running fine. However do I do see these errors from time to time, but I was unable to retrieve the actual log line...
2025-09-27T23:51:04.703866816Z time="2025-09-27T23:51:04Z" level=error msg="failed to parse source_ip xx.xx.xxx: ParseAddr(\"xx.xx.xxx\"): IPv4 address too short"
The IP is my WAN ip...most if the CEF is working ๐
I guess you still have logs going to same file?
Correct
Here's the full parser metric
Might give some clarity
the CEF is extremely low, but I guess that is fine? the firewall logs are great though ๐
I'm having issues with my Unifi installation currently, so it's hard to verify if the number is accurate... It seems low indeed, but I have most things that are normally flagged by IDS/IPS (thus using CEF format) blocked by firewall rules (syslog format). So that would explain it...