CrowdSec•2mo ago

How to retrieve the User-Agent in a Postoverflow scenario?

Hello, I’m trying to set up a Postoverflow whitelist to whitelist an IP if it triggers an alert on a specific UA, but it doesn’t seem to be working. Do you have any idea why?

name: aukfood/whitelist_screaming_frog_ua
description: "Whitelist Screaming Frog SEO Spider UA from IP 192.192.192.192"
whitelist:
  reason: "Legitimate SEO crawler tool from IP 192.192.192.192 excluded from HTTP bad-user-agent detection"
  expression:
    - 'evt.Meta.source_ip == "192.192.192.192" &&
       evt.Meta.http_user_agent == "Screaming Frog SEO Spider/22.2"'

name: aukfood/whitelist_screaming_frog_ua
description: "Whitelist Screaming Frog SEO Spider UA from IP 192.192.192.192"
whitelist:
  reason: "Legitimate SEO crawler tool from IP 192.192.192.192 excluded from HTTP bad-user-agent detection"
  expression:
    - 'evt.Meta.source_ip == "192.192.192.192" &&
       evt.Meta.http_user_agent == "Screaming Frog SEO Spider/22.2"'

line: 192.192.192.192 - - [17/Oct/2025:09:52:41 +0200] "GET /robots.txt HTTP/1.1" 200 1020 "-" "Screaming Frog SEO Spider/22.2" ├ s00-raw | ├ 🔴 crowdsecurity/syslog-logs | └ 🟢 crowdsecurity/non-syslog (+5 ~8) ├ s01-parse | └ 🟢 crowdsecurity/apache2-logs (+21 ~2) ├ s02-enrich | ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~2) | ├ 🔴 crowdsecurity/enrich-user-agent-year | ├ 🟢 crowdsecurity/geoip-enrich (+13) | ├ 🟢 crowdsecurity/http-logs (+7) | ├ 🟢 mywhitelists (unchanged) | ├ 🟢 crowdsecurity/nextcloud-whitelist (unchanged) | ├ 🟢 crowdsecurity/public-dns-allowlist (unchanged) | └ 🟢 crowdsecurity/whitelists (unchanged) ├-------- parser success 🟢 ├ Scenarios ├ 🟢 crowdsecurity/http-bad-user-agent └ 🟢 crowdsecurity/http-crawl-non_statics

8 Replies

CrowdSec•2mo ago

Important Information

This post has been marked as resolved. If this is a mistake please press the red button below or type /unresolve

🅱🅱🆄🅳🅳🅷🅰OP•2mo ago

I had tried that as well.

       evt.Overflow.Alert.Source.IP == "192.192.192.192" &&
       any(evt.Overflow.Alert.Events, { string(.Meta) contains "Screaming Frog SEO Spider" })'

       evt.Overflow.Alert.Source.IP == "192.192.192.192" &&
       any(evt.Overflow.Alert.Events, { string(.Meta) contains "Screaming Frog SEO Spider" })'

Loz•2mo ago

@bbuddha should be using the # replace no?

       evt.Overflow.Alert.Source.IP == "80.15.14.127" &&
       any(evt.Overflow.Alert.Events, { #.GetMeta('http_user_agent') contains "Screaming Frog SEO Spider" })'

       evt.Overflow.Alert.Source.IP == "80.15.14.127" &&
       any(evt.Overflow.Alert.Events, { #.GetMeta('http_user_agent') contains "Screaming Frog SEO Spider" })'

🅱🅱🆄🅳🅳🅷🅰OP•2mo ago

I'll try. By the way, with cscli explain can we see if a whitelist postoverflow is triggered?

Loz•2mo ago

not currently, postoverflows are not supported via cscli explain there is an edge case, where if you provide enough logs to trigger a scenario then it will show for one explain within the bulk but its kinda bad

🅱🅱🆄🅳🅳🅷🅰OP•2mo ago

hahahahaha yes ok

🅱🅱🆄🅳🅳🅷🅰OP•2mo ago

for i in $(seq 1 50); do   echo '192.192.192.192 - - [20/Oct/2025:12:00:00 +0200] "GET /robots.txt HTTP/1.1" 200 1020 "-" "Screaming Frog SEO Spider/22.2"'; done | cscli explain -f- --type apache2

for i in $(seq 1 50); do   echo '192.192.192.192 - - [20/Oct/2025:12:00:00 +0200] "GET /robots.txt HTTP/1.1" 200 1020 "-" "Screaming Frog SEO Spider/22.2"'; done | cscli explain -f- --type apache2

thx

CrowdSec•2mo ago

Resolving How to retrieve the User-Agent in a Postoverflow scenario? This has now been resolved. If you think this is a mistake please run /unresolve

Gaming

Programming

How to retrieve the User-Agent in a Postoverflow scenario?

Did you find this page helpful?