CrowdSec•3mo ago

Custom Unifi parser

Hi all, I'm trying to write a s00 Unifi parser but running into some roadblocks. I have setup a test env. (which was a hassle btw.) Using the help of AI I'm trying to get a basic understanding of how these parsers are written. Example log:

Jun 13 16:30:19 UDMP-DTC [WAN_LOCAL-D-2147483647] DESCR="[WAN_LOCAL]Block All Traffic" IN=eth8 OUT= MAC=74:ac:b9:1c:xx:e5:00:17:xx:2b:31:a9:08:00 SRC=130.195.223.2 DST=<WAN_IP> LEN=60 TOS=00 PREC=0x00 TTL=53 ID=26089 DF PROTO=TCP SPT=57263 DPT=54329 SEQ=2894412795 ACK=0 WINDOW=64240 SYN URGP=0 MARK=1a0000

Jun 13 16:30:19 UDMP-DTC [WAN_LOCAL-D-2147483647] DESCR="[WAN_LOCAL]Block All Traffic" IN=eth8 OUT= MAC=74:ac:b9:1c:xx:e5:00:17:xx:2b:31:a9:08:00 SRC=130.195.223.2 DST=<WAN_IP> LEN=60 TOS=00 PREC=0x00 TTL=53 ID=26089 DF PROTO=TCP SPT=57263 DPT=54329 SEQ=2894412795 ACK=0 WINDOW=64240 SYN URGP=0 MARK=1a0000

/home/ubuntu/crowdsec-v1.6.8/tests/hub/parsers/s00-raw/crowdsecurity/s00-unifi-firewall.yaml

name: crowdsecurity/udmp-firewall-raw
description: "Parse Ubiquiti UDMP firewall log lines in syslog format"
stage: s00-raw
filter: "evt.Parsed.program == ''" # Optionally refine this
pattern_syntax:
  MAC: "[0-9a-f]{2}(:[0-9a-f]{2}){5}(,[0-9a-f]{2}(:[0-9a-f]{2}){5})*"
nodes:
  - grok:
      pattern: |
        %{SYSLOGTIMESTAMP:log_ts} %{DATA:host} \[%{DATA:fw_rule}\] DESCR=\"%{DATA:fw_descr}\" IN=%{WORD:in_iface} OUT=%{WORD:out_iface} MAC=%{MAC:mac} SRC=%{IPV4:src_ip} DST=%{IPV4:dst_ip} LEN=%{INT:len1} TOS=%{DATA:tos} PREC=%{DATA:prec} TTL=%{INT:ttl} ID=%{INT:id} PROTO=%{WORD:proto} SPT=%{INT:src_port} DPT=%{INT:dst_port} LEN=%{INT:len2} MARK=%{DATA:mark}
      apply_on: message
      statics:
        - meta: service
          value: "udmp-firewall"
        - target: evt.Time
          expression: "evt.Parsed.log_ts"
        - target: evt.Parsed.source_ip
          expression: "evt.Parsed.src_ip"
        - target: evt.Parsed.destination_ip
          expression: "evt.Parsed.dst_ip"
        - target: evt.Parsed.source_port
          expression: "evt.Parsed.src_port"
        - target: evt.Parsed.destination_port
          expression: "evt.Parsed.dst_port"
        - target: evt.Parsed.proto
          expression: "evt.Parsed.proto"
        - target: evt.StrTime
          expression: evt.Parsed.log_ts

/home/ubuntu/crowdsec-v1.6.8/tests/hub/parsers/s00-raw/crowdsecurity/s00-unifi-firewall.yaml

name: crowdsecurity/udmp-firewall-raw
description: "Parse Ubiquiti UDMP firewall log lines in syslog format"
stage: s00-raw
filter: "evt.Parsed.program == ''" # Optionally refine this
pattern_syntax:
  MAC: "[0-9a-f]{2}(:[0-9a-f]{2}){5}(,[0-9a-f]{2}(:[0-9a-f]{2}){5})*"
nodes:
  - grok:
      pattern: |
        %{SYSLOGTIMESTAMP:log_ts} %{DATA:host} \[%{DATA:fw_rule}\] DESCR=\"%{DATA:fw_descr}\" IN=%{WORD:in_iface} OUT=%{WORD:out_iface} MAC=%{MAC:mac} SRC=%{IPV4:src_ip} DST=%{IPV4:dst_ip} LEN=%{INT:len1} TOS=%{DATA:tos} PREC=%{DATA:prec} TTL=%{INT:ttl} ID=%{INT:id} PROTO=%{WORD:proto} SPT=%{INT:src_port} DPT=%{INT:dst_port} LEN=%{INT:len2} MARK=%{DATA:mark}
      apply_on: message
      statics:
        - meta: service
          value: "udmp-firewall"
        - target: evt.Time
          expression: "evt.Parsed.log_ts"
        - target: evt.Parsed.source_ip
          expression: "evt.Parsed.src_ip"
        - target: evt.Parsed.destination_ip
          expression: "evt.Parsed.dst_ip"
        - target: evt.Parsed.source_port
          expression: "evt.Parsed.src_port"
        - target: evt.Parsed.destination_port
          expression: "evt.Parsed.dst_port"
        - target: evt.Parsed.proto
          expression: "evt.Parsed.proto"
        - target: evt.StrTime
          expression: evt.Parsed.log_ts

35 Replies

CrowdSec•3mo ago

Important Information

This post has been marked as resolved. If this is a mistake please press the red button below or type /unresolve

PintjesBierOP•3mo ago

/home/ubuntu/crowdsec-v1.6.8/tests/hub/.tests/s00-unifi/config.yaml

parsers:
    - ./parsers/s00-raw/crowdsecurity/s00-unifi-firewall.yaml
scenarios:
postoverflows:
log_file: s00-unifi.log
log_type: unifi

/home/ubuntu/crowdsec-v1.6.8/tests/hub/.tests/s00-unifi/config.yaml

parsers:
    - ./parsers/s00-raw/crowdsecurity/s00-unifi-firewall.yaml
scenarios:
postoverflows:
log_file: s00-unifi.log
log_type: unifi

The output:

╭─🎯 ubuntu on crowdsec-testing in ~/crowdsec-v1.6.8/tests/hub via 🎁 v0.1.0 via 🐍 pyenv                                                                                                                                         5s446ms
╰─✗  csdev --cscli "$(dirname $PWD)/cscli" --crowdsec "$(dirname $PWD)/crowdsec" hubtest run s00-unifi
Running test 's00-unifi'
WARNING Line 0/1 is missing evt.StrTime. It is most likely a mistake as it will prevent your logs to be processed in time-machine/forensic mode.  file=/home/ubuntu/crowdsec-v1.6.8/tests/hub/.tests/s00-unifi/results/parser-dump.yaml
WARNING Assert file '/home/ubuntu/crowdsec-v1.6.8/tests/hub/.tests/s00-unifi/parser.assert' is empty, generating assertion: 

len(results) == 2
len(results["s00-raw"]["crowdsecurity/udmp-firewall-raw"]) == 1
results["s00-raw"]["crowdsecurity/udmp-firewall-raw"][0].Success == false
len(results["success"][""]) == 0

Error: please fill your assert file(s) for test 's00-unifi', exiting

╭─🎯 ubuntu on crowdsec-testing in ~/crowdsec-v1.6.8/tests/hub via 🎁 v0.1.0 via 🐍 pyenv                                                                                                                                         5s446ms
╰─✗  csdev --cscli "$(dirname $PWD)/cscli" --crowdsec "$(dirname $PWD)/crowdsec" hubtest run s00-unifi
Running test 's00-unifi'
WARNING Line 0/1 is missing evt.StrTime. It is most likely a mistake as it will prevent your logs to be processed in time-machine/forensic mode.  file=/home/ubuntu/crowdsec-v1.6.8/tests/hub/.tests/s00-unifi/results/parser-dump.yaml
WARNING Assert file '/home/ubuntu/crowdsec-v1.6.8/tests/hub/.tests/s00-unifi/parser.assert' is empty, generating assertion: 

len(results) == 2
len(results["s00-raw"]["crowdsecurity/udmp-firewall-raw"]) == 1
results["s00-raw"]["crowdsecurity/udmp-firewall-raw"][0].Success == false
len(results["success"][""]) == 0

Error: please fill your assert file(s) for test 's00-unifi', exiting

Sadly AFAIK I'm unable to find an example of these parser.assert files...

CrowdSec•3mo ago

Resolving Custom Unifi parser This has now been resolved. If you think this is a mistake please run /unresolve

TravelerVFX•3mo ago

This update is available using cscli? (sorry for the beginner question)

PintjesBierOP•3mo ago

Not yet, I have submitted a PR to the Hub's repo. Once that is approved it should be available for everyone to add.

TravelerVFX•3mo ago

Thanks a lot for the fix and clarification !

PintjesBierOP•3mo ago

Like I have mentioned tho on the github thread: if you have the following: - Unifi IDS/IPS logs - Unifi WAN rule hit logs - Unifi login attempt logs Please send them to me for further testing and expanding what is being parsed.

Lepresidente•2mo ago

thought i'll come chat here, so this currently will never work unifi-logs in S00 will never fire since syslogs-logs fires non-syslogs-logs first

Lepresidente•2mo ago

@Loz do you see any issues with me updating this line to include unifi in the filter From what I've seen the parser loads alphabetically and the unifi-logs parser is never reached as this non sys logs returns true https://github.com/crowdsecurity/hub/blob/0af7ad090edc801526a777892da27f9b1fb1c730/parsers/s00-raw/crowdsecurity/syslog-logs.yaml#L35

GitHub

hub/parsers/s00-raw/crowdsecurity/syslog-logs.yaml at 0af7ad090edc8...

Main repository for crowdsec scenarios/parsers. Contribute to crowdsecurity/hub development by creating an account on GitHub.

PintjesBierOP•2mo ago

The type in the acquis is also unifi, so shouldn't that bypass this? Will get back to your github feedback tomorrow btw 🙂

Lepresidente•2mo ago

Yea i got it all working my side using your grok filter, also got the cef stuff into the existing crowdsec unifi-logs if needed. You need to use rsyslog though unifi-logs never fires if crowdsec is using the builtin internal syslog server

PintjesBierOP•2mo ago

I'm in talks with Unifi about their logging, depending on their answers I will create whatever parser is needed for both CEF and normal Update, just received an email:

We got an update from the development team on this. This will be fixed in the 9.3.30 Network Application version.

We got an update from the development team on this. This will be fixed in the 9.3.30 Network Application version.

Bump @Loz

PintjesBierOP•2mo ago

Would be nice to have some feedback on this as well... https://github.com/crowdsecurity/hub/pull/1395

GitHub

Unifi logs V0.5 by PintjesB · Pull Request #1395 · crowdsecurity/hub

Initial push of Unifi parser + scenario. Please let me know what needs to change.

PintjesBierOP•2mo ago

bump

iiamloz•2mo ago

Hey 👋 I have seen this and I will review it but the first thing this is a s01 parser which means its going to get really messy when you try to support multiple logs from unifi but thank you for contacting unifi and managing a communication with them. I will see if I can look into it today, but most likely need to switch to s00

iiamloz•2mo ago

https://github.com/crowdsecurity/hub/pull/1408

GitHub

enhance: Attempt s00 unifi by LaurenceJJones · Pull Request #1408 ...

s00 unifi to handle the firewall logs, formats to iptables parsers and uses the iptables sceanrio Since unifi not longer does the old logs I have removed them and only uses the newest syslog logs. ...

iiamloz•2mo ago

I will track back through the thread now to see applicaton logs like ssh

PintjesBierOP•2mo ago

Awesome, I'll be on vacay for 3 weeks but if anything else is needed I can have a look after.

iiamloz•2mo ago

Have an awesome vacation! and ill keep doing updates and ill get the team to review as we heading into holidays so we got some extra bandwidth

PintjesBierOP•2mo ago

Also note, communication with Unifi stated that: - IDS/IPS logs will be implemented in RAW format to syslog soon - Failed auth logs are not forwarded to syslog However, I have noticed people who selfhosted the controller do get auth logs Think they claimed it would be implemented starting from 9.3.30 however I do not see the logs appearing yet...

iiamloz•2mo ago

as in the gui panel logs?

PintjesBierOP•2mo ago

Logs like these (in RAW format):

unifi-firewall.log.8.gz:2025-06-09T18:37:21-04:00 UniFi CEF: 0|Ubiquiti|UniFi Network|9.1.120|Intrusion Prevention|Threat Detected and Blocked|6|msg=IPS Alert 2: Misc Attack. Signature ET DROP Dshield Block Listed Source group 1. From: 193.163.125.254:50228, to: 192.168.10.254:6281, protocol: TCP signature_type=ET signature_id=2402000

unifi-firewall.log.8.gz:2025-06-09T18:37:21-04:00 UniFi CEF: 0|Ubiquiti|UniFi Network|9.1.120|Intrusion Prevention|Threat Detected and Blocked|6|msg=IPS Alert 2: Misc Attack. Signature ET DROP Dshield Block Listed Source group 1. From: 193.163.125.254:50228, to: 192.168.10.254:6281, protocol: TCP signature_type=ET signature_id=2402000

Should be sent to syslog as of 9.3.30, however I can't seem to find them I have messaged Unifi support

iiamloz•2mo ago

so from timestamp onwards? cause I guess unifi-firewall.log.8.gz is not included?

PintjesBierOP•2mo ago

I'm not following, what do you mean?

iiamloz•2mo ago

cause the CEF that the other users have shown was just

0|Ubiquiti|UniFi Network|9.1.120|Intrusion Prevention|Threat Detected and Blocked|6|msg=IPS Alert 2: Misc Attack. Signature ET DROP Dshield Block Listed Source group 1. From: 193.163.125.254:50228, to: 192.168.10.254:6281, protocol: TCP signature_type=ET signature_id=2402000

0|Ubiquiti|UniFi Network|9.1.120|Intrusion Prevention|Threat Detected and Blocked|6|msg=IPS Alert 2: Misc Attack. Signature ET DROP Dshield Block Listed Source group 1. From: 193.163.125.254:50228, to: 192.168.10.254:6281, protocol: TCP signature_type=ET signature_id=2402000

I guess 2025-06-09T18:37:21-04:00 UniFi CEF: is the raw prefix and unifi-firewall.log.8.gz is the file they grabbed it from?

PintjesBierOP•2mo ago

ohhh yeah yeah correct Can anyone else confirm they can see IDS/IPS logs in 9.3.33 (EA)? @TravelerVFX I'm on vacay rn, can check in about 2 weeks

TravelerVFX•2mo ago

Sorry, I'm still on version 9.2.87

PintjesBierOP•5w ago

9.3.43 seems to have included the revampedlogging system. Seems like they're still pushing hard for CEF format though... But it's progress nevertheless Hi, @iiamloz is there anything we/I can help you with for this?

iiamloz•5w ago

i am fighting with the tests atm

iiamloz•5w ago

https://tenor.com/view/hasbulla-hasbik-cute-meme-influencer-gif-21732737

Tenor

PintjesBierOP•5w ago

Anything I can help with?

bondskin•4w ago

I am on 9.3.45 and still not seeing ips/ids events on my UDM-SE, hence not able to send them to rsyslog/crowdsec. Very frustrating

PintjesBierOP•4w ago

Did you enable them in settings?

bondskin•4w ago