CrowdSec•7mo ago

Custom Unifi parser

Hi all, I'm trying to write a s00 Unifi parser but running into some roadblocks. I have setup a test env. (which was a hassle btw.) Using the help of AI I'm trying to get a basic understanding of how these parsers are written. Example log:

Jun 13 16:30:19 UDMP-DTC [WAN_LOCAL-D-2147483647] DESCR="[WAN_LOCAL]Block All Traffic" IN=eth8 OUT= MAC=74:ac:b9:1c:xx:e5:00:17:xx:2b:31:a9:08:00 SRC=130.195.223.2 DST=<WAN_IP> LEN=60 TOS=00 PREC=0x00 TTL=53 ID=26089 DF PROTO=TCP SPT=57263 DPT=54329 SEQ=2894412795 ACK=0 WINDOW=64240 SYN URGP=0 MARK=1a0000

Jun 13 16:30:19 UDMP-DTC [WAN_LOCAL-D-2147483647] DESCR="[WAN_LOCAL]Block All Traffic" IN=eth8 OUT= MAC=74:ac:b9:1c:xx:e5:00:17:xx:2b:31:a9:08:00 SRC=130.195.223.2 DST=<WAN_IP> LEN=60 TOS=00 PREC=0x00 TTL=53 ID=26089 DF PROTO=TCP SPT=57263 DPT=54329 SEQ=2894412795 ACK=0 WINDOW=64240 SYN URGP=0 MARK=1a0000

/home/ubuntu/crowdsec-v1.6.8/tests/hub/parsers/s00-raw/crowdsecurity/s00-unifi-firewall.yaml

name: crowdsecurity/udmp-firewall-raw
description: "Parse Ubiquiti UDMP firewall log lines in syslog format"
stage: s00-raw
filter: "evt.Parsed.program == ''" # Optionally refine this
pattern_syntax:
  MAC: "[0-9a-f]{2}(:[0-9a-f]{2}){5}(,[0-9a-f]{2}(:[0-9a-f]{2}){5})*"
nodes:
  - grok:
      pattern: |
        %{SYSLOGTIMESTAMP:log_ts} %{DATA:host} \[%{DATA:fw_rule}\] DESCR=\"%{DATA:fw_descr}\" IN=%{WORD:in_iface} OUT=%{WORD:out_iface} MAC=%{MAC:mac} SRC=%{IPV4:src_ip} DST=%{IPV4:dst_ip} LEN=%{INT:len1} TOS=%{DATA:tos} PREC=%{DATA:prec} TTL=%{INT:ttl} ID=%{INT:id} PROTO=%{WORD:proto} SPT=%{INT:src_port} DPT=%{INT:dst_port} LEN=%{INT:len2} MARK=%{DATA:mark}
      apply_on: message
      statics:
        - meta: service
          value: "udmp-firewall"
        - target: evt.Time
          expression: "evt.Parsed.log_ts"
        - target: evt.Parsed.source_ip
          expression: "evt.Parsed.src_ip"
        - target: evt.Parsed.destination_ip
          expression: "evt.Parsed.dst_ip"
        - target: evt.Parsed.source_port
          expression: "evt.Parsed.src_port"
        - target: evt.Parsed.destination_port
          expression: "evt.Parsed.dst_port"
        - target: evt.Parsed.proto
          expression: "evt.Parsed.proto"
        - target: evt.StrTime
          expression: evt.Parsed.log_ts

/home/ubuntu/crowdsec-v1.6.8/tests/hub/parsers/s00-raw/crowdsecurity/s00-unifi-firewall.yaml

name: crowdsecurity/udmp-firewall-raw
description: "Parse Ubiquiti UDMP firewall log lines in syslog format"
stage: s00-raw
filter: "evt.Parsed.program == ''" # Optionally refine this
pattern_syntax:
  MAC: "[0-9a-f]{2}(:[0-9a-f]{2}){5}(,[0-9a-f]{2}(:[0-9a-f]{2}){5})*"
nodes:
  - grok:
      pattern: |
        %{SYSLOGTIMESTAMP:log_ts} %{DATA:host} \[%{DATA:fw_rule}\] DESCR=\"%{DATA:fw_descr}\" IN=%{WORD:in_iface} OUT=%{WORD:out_iface} MAC=%{MAC:mac} SRC=%{IPV4:src_ip} DST=%{IPV4:dst_ip} LEN=%{INT:len1} TOS=%{DATA:tos} PREC=%{DATA:prec} TTL=%{INT:ttl} ID=%{INT:id} PROTO=%{WORD:proto} SPT=%{INT:src_port} DPT=%{INT:dst_port} LEN=%{INT:len2} MARK=%{DATA:mark}
      apply_on: message
      statics:
        - meta: service
          value: "udmp-firewall"
        - target: evt.Time
          expression: "evt.Parsed.log_ts"
        - target: evt.Parsed.source_ip
          expression: "evt.Parsed.src_ip"
        - target: evt.Parsed.destination_ip
          expression: "evt.Parsed.dst_ip"
        - target: evt.Parsed.source_port
          expression: "evt.Parsed.src_port"
        - target: evt.Parsed.destination_port
          expression: "evt.Parsed.dst_port"
        - target: evt.Parsed.proto
          expression: "evt.Parsed.proto"
        - target: evt.StrTime
          expression: evt.Parsed.log_ts

38 Replies

CrowdSec•7mo ago

Important Information

This post has been marked as resolved. If this is a mistake please press the red button below or type /unresolve

PintjesBierOP•7mo ago

/home/ubuntu/crowdsec-v1.6.8/tests/hub/.tests/s00-unifi/config.yaml

parsers:
    - ./parsers/s00-raw/crowdsecurity/s00-unifi-firewall.yaml
scenarios:
postoverflows:
log_file: s00-unifi.log
log_type: unifi

/home/ubuntu/crowdsec-v1.6.8/tests/hub/.tests/s00-unifi/config.yaml

parsers:
    - ./parsers/s00-raw/crowdsecurity/s00-unifi-firewall.yaml
scenarios:
postoverflows:
log_file: s00-unifi.log
log_type: unifi

The output:

╭─🎯 ubuntu on crowdsec-testing in ~/crowdsec-v1.6.8/tests/hub via 🎁 v0.1.0 via 🐍 pyenv                                                                                                                                         5s446ms
╰─✗  csdev --cscli "$(dirname $PWD)/cscli" --crowdsec "$(dirname $PWD)/crowdsec" hubtest run s00-unifi
Running test 's00-unifi'
WARNING Line 0/1 is missing evt.StrTime. It is most likely a mistake as it will prevent your logs to be processed in time-machine/forensic mode.  file=/home/ubuntu/crowdsec-v1.6.8/tests/hub/.tests/s00-unifi/results/parser-dump.yaml
WARNING Assert file '/home/ubuntu/crowdsec-v1.6.8/tests/hub/.tests/s00-unifi/parser.assert' is empty, generating assertion: 

len(results) == 2
len(results["s00-raw"]["crowdsecurity/udmp-firewall-raw"]) == 1
results["s00-raw"]["crowdsecurity/udmp-firewall-raw"][0].Success == false
len(results["success"][""]) == 0

Error: please fill your assert file(s) for test 's00-unifi', exiting

╭─🎯 ubuntu on crowdsec-testing in ~/crowdsec-v1.6.8/tests/hub via 🎁 v0.1.0 via 🐍 pyenv                                                                                                                                         5s446ms
╰─✗  csdev --cscli "$(dirname $PWD)/cscli" --crowdsec "$(dirname $PWD)/crowdsec" hubtest run s00-unifi
Running test 's00-unifi'
WARNING Line 0/1 is missing evt.StrTime. It is most likely a mistake as it will prevent your logs to be processed in time-machine/forensic mode.  file=/home/ubuntu/crowdsec-v1.6.8/tests/hub/.tests/s00-unifi/results/parser-dump.yaml
WARNING Assert file '/home/ubuntu/crowdsec-v1.6.8/tests/hub/.tests/s00-unifi/parser.assert' is empty, generating assertion: 

len(results) == 2
len(results["s00-raw"]["crowdsecurity/udmp-firewall-raw"]) == 1
results["s00-raw"]["crowdsecurity/udmp-firewall-raw"][0].Success == false
len(results["success"][""]) == 0

Error: please fill your assert file(s) for test 's00-unifi', exiting

Sadly AFAIK I'm unable to find an example of these parser.assert files...

CrowdSec•7mo ago

Resolving Custom Unifi parser This has now been resolved. If you think this is a mistake please run /unresolve

TravelerVFX•6mo ago

This update is available using cscli? (sorry for the beginner question)

PintjesBierOP•6mo ago

Not yet, I have submitted a PR to the Hub's repo. Once that is approved it should be available for everyone to add.

TravelerVFX•6mo ago

Thanks a lot for the fix and clarification !

PintjesBierOP•6mo ago

Like I have mentioned tho on the github thread: if you have the following: - Unifi IDS/IPS logs - Unifi WAN rule hit logs - Unifi login attempt logs Please send them to me for further testing and expanding what is being parsed.

Lepresidente•6mo ago

thought i'll come chat here, so this currently will never work unifi-logs in S00 will never fire since syslogs-logs fires non-syslogs-logs first

Lepresidente•6mo ago

@Loz do you see any issues with me updating this line to include unifi in the filter From what I've seen the parser loads alphabetically and the unifi-logs parser is never reached as this non sys logs returns true https://github.com/crowdsecurity/hub/blob/0af7ad090edc801526a777892da27f9b1fb1c730/parsers/s00-raw/crowdsecurity/syslog-logs.yaml#L35

GitHub

hub/parsers/s00-raw/crowdsecurity/syslog-logs.yaml at 0af7ad090edc8...

Main repository for crowdsec scenarios/parsers. Contribute to crowdsecurity/hub development by creating an account on GitHub.

PintjesBierOP•6mo ago

The type in the acquis is also unifi, so shouldn't that bypass this? Will get back to your github feedback tomorrow btw 🙂

Lepresidente•6mo ago

Yea i got it all working my side using your grok filter, also got the cef stuff into the existing crowdsec unifi-logs if needed. You need to use rsyslog though unifi-logs never fires if crowdsec is using the builtin internal syslog server

PintjesBierOP•6mo ago

I'm in talks with Unifi about their logging, depending on their answers I will create whatever parser is needed for both CEF and normal Update, just received an email:

We got an update from the development team on this. This will be fixed in the 9.3.30 Network Application version.

We got an update from the development team on this. This will be fixed in the 9.3.30 Network Application version.

Bump @Loz

PintjesBierOP•6mo ago

Would be nice to have some feedback on this as well... https://github.com/crowdsecurity/hub/pull/1395

GitHub

Unifi logs V0.5 by PintjesB · Pull Request #1395 · crowdsecurity/hub

Initial push of Unifi parser + scenario. Please let me know what needs to change.

PintjesBierOP•6mo ago

bump

Loz•6mo ago

Hey 👋 I have seen this and I will review it but the first thing this is a s01 parser which means its going to get really messy when you try to support multiple logs from unifi but thank you for contacting unifi and managing a communication with them. I will see if I can look into it today, but most likely need to switch to s00

Loz•6mo ago

https://github.com/crowdsecurity/hub/pull/1408

GitHub

enhance: Attempt s00 unifi by LaurenceJJones · Pull Request #1408 ...

s00 unifi to handle the firewall logs, formats to iptables parsers and uses the iptables sceanrio Since unifi not longer does the old logs I have removed them and only uses the newest syslog logs. ...

Loz•6mo ago

I will track back through the thread now to see applicaton logs like ssh

PintjesBierOP•6mo ago

Awesome, I'll be on vacay for 3 weeks but if anything else is needed I can have a look after.

Loz•6mo ago

Have an awesome vacation! and ill keep doing updates and ill get the team to review as we heading into holidays so we got some extra bandwidth

PintjesBierOP•6mo ago

Also note, communication with Unifi stated that: - IDS/IPS logs will be implemented in RAW format to syslog soon - Failed auth logs are not forwarded to syslog However, I have noticed people who selfhosted the controller do get auth logs Think they claimed it would be implemented starting from 9.3.30 however I do not see the logs appearing yet...

Loz•6mo ago

as in the gui panel logs?

PintjesBierOP•6mo ago

Logs like these (in RAW format):

unifi-firewall.log.8.gz:2025-06-09T18:37:21-04:00 UniFi CEF: 0|Ubiquiti|UniFi Network|9.1.120|Intrusion Prevention|Threat Detected and Blocked|6|msg=IPS Alert 2: Misc Attack. Signature ET DROP Dshield Block Listed Source group 1. From: 193.163.125.254:50228, to: 192.168.10.254:6281, protocol: TCP signature_type=ET signature_id=2402000

unifi-firewall.log.8.gz:2025-06-09T18:37:21-04:00 UniFi CEF: 0|Ubiquiti|UniFi Network|9.1.120|Intrusion Prevention|Threat Detected and Blocked|6|msg=IPS Alert 2: Misc Attack. Signature ET DROP Dshield Block Listed Source group 1. From: 193.163.125.254:50228, to: 192.168.10.254:6281, protocol: TCP signature_type=ET signature_id=2402000

Should be sent to syslog as of 9.3.30, however I can't seem to find them I have messaged Unifi support

Loz•6mo ago

so from timestamp onwards? cause I guess unifi-firewall.log.8.gz is not included?

PintjesBierOP•6mo ago

I'm not following, what do you mean?

Loz•6mo ago

cause the CEF that the other users have shown was just

0|Ubiquiti|UniFi Network|9.1.120|Intrusion Prevention|Threat Detected and Blocked|6|msg=IPS Alert 2: Misc Attack. Signature ET DROP Dshield Block Listed Source group 1. From: 193.163.125.254:50228, to: 192.168.10.254:6281, protocol: TCP signature_type=ET signature_id=2402000

0|Ubiquiti|UniFi Network|9.1.120|Intrusion Prevention|Threat Detected and Blocked|6|msg=IPS Alert 2: Misc Attack. Signature ET DROP Dshield Block Listed Source group 1. From: 193.163.125.254:50228, to: 192.168.10.254:6281, protocol: TCP signature_type=ET signature_id=2402000

I guess 2025-06-09T18:37:21-04:00 UniFi CEF: is the raw prefix and unifi-firewall.log.8.gz is the file they grabbed it from?

PintjesBierOP•6mo ago

ohhh yeah yeah correct Can anyone else confirm they can see IDS/IPS logs in 9.3.33 (EA)? @TravelerVFX I'm on vacay rn, can check in about 2 weeks

TravelerVFX•6mo ago

Sorry, I'm still on version 9.2.87

PintjesBierOP•5mo ago

9.3.43 seems to have included the revampedlogging system. Seems like they're still pushing hard for CEF format though... But it's progress nevertheless Hi, @iiamloz is there anything we/I can help you with for this?

Loz•5mo ago

i am fighting with the tests atm

Loz•5mo ago

https://tenor.com/view/hasbulla-hasbik-cute-meme-influencer-gif-21732737

Tenor

PintjesBierOP•5mo ago

Anything I can help with?

bondskin•5mo ago

I am on 9.3.45 and still not seeing ips/ids events on my UDM-SE, hence not able to send them to rsyslog/crowdsec. Very frustrating

PintjesBierOP•5mo ago

Did you enable them in settings?

bondskin•5mo ago

I enabled all options in Console and Network . I am seeing events (eg logon, changing settings? but nothing from the firewall.

PintjesBierOP•4mo ago

Mhm, I just got back from vacay. Will look into it tomorrow and contact Unifi if needed I thought I was able to find them @bondskin How are your logs looking right now? And what version are you running? @TravelerVFX idem ditto

bondskin•4mo ago

Hi @PintjesBier , yes, but I suspect this being related to the SWAG mod in my environment time=2025-08-27T09:00:05+02:00 level=info msg=172.18.0.1 - [Wed, 27 Aug 2025 09:00:05 CEST] "POST /v1/usage-metrics HTTP/1.1 400 5.028665ms "crowdsec-nginx-bouncer/v1.1.3" " time=2025-08-27T09:01:25+02:00 level=info msg=127.0.0.1 - [Wed, 27 Aug 2025 09:01:25 CEST] "POST /v1/usage-metrics HTTP/1.1 201 21.205034ms "crowdsec/v1.6.11-d64ee2ae-docker" " time=2025-08-27T09:12:38+02:00 level=info msg=Sent 3 usage metrics time=2025-08-27T09:14:44+02:00 level=info msg=172.18.0.1 - [Wed, 27 Aug 2025 09:14:44 CEST] "POST /v1/usage-metrics HTTP/1.1 201 4.314083ms "crowdsec-firewall-bouncer/v0.0.34-debian-pragmatic-arm64-4144555453620958398aee64253dfd90bbc1f698" " time=2025-08-27T09:15:05+02:00 level=error msg=Failed to bind json: json: cannot unmarshal object into Go struct field AllMetrics.remediation_components.feature_flags of type []string func=UsageMetrics time=2025-08-27T09:15:05+02:00 level=info msg=172.18.0.1 - [Wed, 27 Aug 2025 09:15:05 CEST] "POST /v1/usage-metrics HTTP/1.1 400 1.258852ms "crowdsec-nginx-bouncer/v1.1.3" " time=2025-08-27T09:22:02+02:00 level=info msg=capi metrics: sending

bondskin•4mo ago

as a summary: swag mod (v1.1.3) does not deliver metrics, but the stand-alone host firewall bouncer (v0.0.34) does

PintjesBierOP•4mo ago

I mean the Unifi logs, to help Loz figure out the new parser 🫣

Gaming

Programming

Custom Unifi parser

Did you find this page helpful?