Whitelist user agent from file

Hi,
I try to write a whitelist parser, to whitelist user agent from a file. (stored in parsers/s02-enrich).
This is the parser that I write :

name: si/si_wl_useragent_ai
description: "Whitelist UA AI"
whitelist:
  reason: "Whitelist UA AI"
  expression:
    - "any(File('useragent_ai.txt'), { evt.Parsed.http_user_agent contains # })"
data:
  - source_url: https://data.srvsi.com/useragent_ai.txt
    dest_file: useragent_ai.txt
    type: string

name: si/si_wl_useragent_ai
description: "Whitelist UA AI"
whitelist:
  reason: "Whitelist UA AI"
  expression:
    - "any(File('useragent_ai.txt'), { evt.Parsed.http_user_agent contains # })"
data:
  - source_url: https://data.srvsi.com/useragent_ai.txt
    dest_file: useragent_ai.txt
    type: string

And some extract from the file :

GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot

GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot

The parser is loaded correctly :

time="2025-06-10T05:10:07+02:00" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s02-enrich/si-useragent-ai.yaml stage=s02-enrich

time="2025-06-10T05:10:07+02:00" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s02-enrich/si-useragent-ai.yaml stage=s02-enrich

I have probably missed something, any idea on how I can fix this ?

CrowdSec•6/10/25, 6:38 AM

Important Information

Thank you for getting in touch with your support request. To expedite a swift resolution, could you kindly provide the following information? Rest assured, we will respond promptly, and we greatly appreciate your patience. While you wait, please check the links below to see if this issue has been previously addressed. If you have managed to resolve it, please use run the command

/resolve

/resolve

or press the green resolve button below.

Log Files

If you possess any log files that you believe could be beneficial, please include them at this time. By default, CrowdSec logs to /var/log/, where you will discover a corresponding log file for each component.

Guide Followed (CrowdSec Official)

If you have diligently followed one of our guides and hit a roadblock, please share the guide with us. This will help us assess if any adjustments are necessary to assist you further.

Screenshots

Please forward any screenshots depicting errors you encounter. Your visuals will provide us with a clear view of the issues you are facing.

•

6/10/25, 6:38 AM

SSich Hi, I try to write a whitelist parser, to whitelist user agent from a file. (sto...

Loz•6/10/25, 8:34 AM

and the file exist in ?

SichOP•6/10/25, 8:36 AM

Hi, yes the file exist

Loz•6/10/25, 8:36 AM

if you add to the yaml of the whitelist do you get any usable feedback in the log?

SichOP•6/10/25, 8:42 AM

I will test that

SichOP•6/10/25, 8:49 AM

ok, tested, so much spam in the log !

Apparently he read the file correctly.
But I continu to get bot banned even with the good user agent.

SichOP•6/10/25, 8:49 AM

SichOP•6/10/25, 9:04 AM

if you tell me that everything is fine, I will just do a cscli decions delete --all, and watch my logs and see if something interesting happen...

Loz•6/10/25, 9:04 AM

its looks good to me, the only thing is its a contains so it has to match case.

SichOP•6/10/25, 9:11 AM

currently scanned by chatgpt, but no scenarios triggered, I will continu to watch.
thx for your help.

SichOP•6/10/25, 11:13 AM

ok, I think I understand, thoses IP are banned inside CAPI.
Exemple for 20.171.207.115, it's in 20.171.207.0/24, who is a openai range.
https://openai.com/gptbot-ranges.txt
But the IP is blocked by CAPI : https://app.crowdsec.net/cti/20.171.207.115
And on my side, it's blocked through CAPI :

Maybe you should take care of that ? To not ban those IPs with the CAPI ?