Create a local collection

Hello, I am trying to create a local collection (to take into account local scenario), but unfortunately it does not work (the scenarios of collection are ignored for log processing).

My collection appear in the list of collections (

cscli collections list

cscli collections list

) but its details are incomplete with

cscli collections inspect

cscli collections inspect

.

Here is my collection (

pierre42100/bots-user-agents

pierre42100/bots-user-agents

is a custom scenario that I created based on

crowdsecurity/http-bad-user-agent

crowdsecurity/http-bad-user-agent

name: pierre42100/custom
parsers:
  - crowdsecurity/traefik-logs
collections:
  # Rely on Traefik
  - crowdsecurity/base-http-scenarios
description: "Custom collection for custom scenarios"
author: pierre42100
scenarios:
  - pierre42100/bots-user-agents
tags:
  - custom
  - traefik
  - http
  - bruteforce

name: pierre42100/custom
parsers:
  - crowdsecurity/traefik-logs
collections:
  # Rely on Traefik
  - crowdsecurity/base-http-scenarios
description: "Custom collection for custom scenarios"
author: pierre42100
scenarios:
  - pierre42100/bots-user-agents
tags:
  - custom
  - traefik
  - http
  - bruteforce

Here is the output of the

cscli collections list

cscli collections list

 Name                               📦 Status          Version  Local Path               
 crowdsecurity/base-http-scenarios  ✔️  enabled        1.2      /etc/crowdsec/collections/base-http-scenarios.yaml 
 crowdsecurity/http-cve             ✔️  enabled        2.9      /etc/crowdsec/collections/http-cve.yaml            
 crowdsecurity/traefik              ✔️  enabled        0.1      /etc/crowdsec/collections/traefik.yaml             
 pierre42100/custom                 🏠  enabled,local           /etc/crowdsec/collections/custom_collection.yaml

 Name                               📦 Status          Version  Local Path               
 crowdsecurity/base-http-scenarios  ✔️  enabled        1.2      /etc/crowdsec/collections/base-http-scenarios.yaml 
 crowdsecurity/http-cve             ✔️  enabled        2.9      /etc/crowdsec/collections/http-cve.yaml            
 crowdsecurity/traefik              ✔️  enabled        0.1      /etc/crowdsec/collections/traefik.yaml             
 pierre42100/custom                 🏠  enabled,local           /etc/crowdsec/collections/custom_collection.yaml

And the output of

cscli collections inspect pierre42100/custom

cscli collections inspect pierre42100/custom

: (dependencies are empty, but it should not)

type: collections
name: pierre42100/custom
file_name: custom_collection.yaml
dependencies: {}
local_path: /etc/crowdsec/collections/custom_collection.yaml
downloadpath: ""
up_to_date: true
tainted: false
installed: true
local: true

Current metrics:

type: collections
name: pierre42100/custom
file_name: custom_collection.yaml
dependencies: {}
local_path: /etc/crowdsec/collections/custom_collection.yaml
downloadpath: ""
up_to_date: true
tainted: false
installed: true
local: true

Current metrics:

What should I do to address this?

Thank you in advance for your help!

CrowdSec•12/8/25, 10:37 AM

Important Information

This post has been marked as resolved. If this is a mistake please press the red button below or type

/unresolve

/unresolve

•

12/8/25, 11:18 AM

PPierre HUBERT Hello, I am trying to create a local collection (to take into account local scen...

Loz•12/8/25, 10:51 AM

Yes there is no such thing as local collections, you have to manually install the scenario inside the

/etc/crowdsec/scenarios

/etc/crowdsec/scenarios

folder

Pierre HUBERTOP•12/8/25, 10:54 AM

Okay great thank you for the quick feedback! I already did this, I created the following files:

crowdsec-5bf9b9489d-pg9hq:/# cat /etc/crowdsec/scenarios/bots_user_agents.yaml 
type: leaky
format: 2.0
#debug: true
name: pierre42100/bots-user-agents
description: "Detect usage of bad User Agents"
filter: 'evt.Meta.log_type in ["http_access-log", "http_error-log"] && RegexpInFile(evt.Parsed.http_user_agent, "bots_user_agents.regex.txt")'
capacity: 1
leakspeed: 1m
groupby: "evt.Meta.source_ip"
blackhole: 2m
labels:
  confidence: 1
  spoofable: 0
  classification:
    - attack.T1595
  behavior: "http:scan"
  label: "Bad User Agent"
  service: http
  remediation: true
crowdsec-5bf9b9489d-pg9hq:/# cat /var/lib/crowdsec/data/bots_user_agents.regex.txt 
\bGPTBot\b
\bmeta-externalagent\b
\bAmazonbot\b
\bAliyunSecBot\b
\bClaudeBot\b
\bSemrushBot\b
\bSERankingBacklinksBot\b
\bBytespider\b

crowdsec-5bf9b9489d-pg9hq:/# cat /etc/crowdsec/scenarios/bots_user_agents.yaml 
type: leaky
format: 2.0
#debug: true
name: pierre42100/bots-user-agents
description: "Detect usage of bad User Agents"
filter: 'evt.Meta.log_type in ["http_access-log", "http_error-log"] && RegexpInFile(evt.Parsed.http_user_agent, "bots_user_agents.regex.txt")'
capacity: 1
leakspeed: 1m
groupby: "evt.Meta.source_ip"
blackhole: 2m
labels:
  confidence: 1
  spoofable: 0
  classification:
    - attack.T1595
  behavior: "http:scan"
  label: "Bad User Agent"
  service: http
  remediation: true
crowdsec-5bf9b9489d-pg9hq:/# cat /var/lib/crowdsec/data/bots_user_agents.regex.txt 
\bGPTBot\b
\bmeta-externalagent\b
\bAmazonbot\b
\bAliyunSecBot\b
\bClaudeBot\b
\bSemrushBot\b
\bSERankingBacklinksBot\b
\bBytespider\b

But how can I check that the scenario is taken into account then?

PPierre HUBERT Okay great thank you for the quick feedback! I already did this, I created the f...

Loz•12/8/25, 10:54 AM

restart crowdsec then

grep "pierre42100/bots-user-agents" /var/log/crowdsec.log

grep "pierre42100/bots-user-agents" /var/log/crowdsec.log

Loz•12/8/25, 10:55 AM

grep "bots_user_agents.yaml" /var/log/crowdsec.log

grep "bots_user_agents.yaml" /var/log/crowdsec.log

Pierre HUBERTOP•12/8/25, 10:55 AM

I get this "not downloading local item"

Loz•12/8/25, 10:56 AM

yeah cause it local it doesnt need to download anything

Pierre HUBERTOP•12/8/25, 10:56 AM

But my scenario, unlike the others of the hub, is not part of a collection. Is it automatically used?

Loz•12/8/25, 10:57 AM

Yes if it already exists in the folder then it automatically enabled

Loz•12/8/25, 10:57 AM

you can run

cscli scenarios list

cscli scenarios list

Pierre HUBERTOP•12/8/25, 10:57 AM

Loz•12/8/25, 11:00 AM

so just to confirm your scenario triggers when a single ip sends 2 requests with "bot" user agents

cause an overflow is capacity + 1

Pierre HUBERTOP•12/8/25, 11:01 AM

Yes absolutely

Loz•12/8/25, 11:01 AM

just confirming as some users get confused and think capacity means overflow trigger

Pierre HUBERTOP•12/8/25, 11:02 AM

Yes my goal here is -> when ChatGPT or any other bots try to request my server at least twice, a captcha is automatically required (configured in profiles)

Loz•12/8/25, 11:03 AM

yeah as long as your profiles is matching the scenario then lgtm

Pierre HUBERTOP•12/8/25, 11:04 AM

Here is my profile:

crowdsec-5678ddc69f-rqzc4:/# cat /etc/crowdsec/profiles.yaml 
name: default_ip_remediation
#debug: true
filters:
  - Alert.Remediation == true && Alert.GetScope() == "Ip"
decisions:
  - type: captcha
    duration: 24h
on_success: break
---
name: default_range_remediation
#debug: true
filters:
  - Alert.Remediation == true && Alert.GetScope() == "Range"
decisions:
  - type: captcha
    duration: 24h
on_success: break

crowdsec-5678ddc69f-rqzc4:/# cat /etc/crowdsec/profiles.yaml 
name: default_ip_remediation
#debug: true
filters:
  - Alert.Remediation == true && Alert.GetScope() == "Ip"
decisions:
  - type: captcha
    duration: 24h
on_success: break
---
name: default_range_remediation
#debug: true
filters:
  - Alert.Remediation == true && Alert.GetScope() == "Range"
decisions:
  - type: captcha
    duration: 24h
on_success: break

Pierre HUBERTOP•12/8/25, 11:04 AM

And when I try the

explain

explain

command my scenario does not seem to be triggered

message.txt10.37KB

Loz•12/8/25, 11:05 AM

RegexpInFile(evt.Parsed.http_user_agent, "bots_user_agents.regex.txt")'

RegexpInFile(evt.Parsed.http_user_agent, "bots_user_agents.regex.txt")'

Loz•12/8/25, 11:05 AM

evt.Meta

evt.Meta

cause

    |        └ create evt.Meta.http_user_agent : Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com GPTBot)

    |        └ create evt.Meta.http_user_agent : Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com GPTBot)

Loz•12/8/25, 11:06 AM

that means our default doesnt work for traefik

Loz•12/8/25, 11:09 AM

hmmm

Loz•12/8/25, 11:09 AM

looking at the parser it does

Loz•12/8/25, 11:09 AM

im blind yeah it should be

Loz•12/8/25, 11:10 AM

probably the regex is not hitting or the datafile is blank

Pierre HUBERTOP•12/8/25, 11:10 AM

Okay I will investigate this then, because yes

crowdsecurity/http-bad-user-agent

crowdsecurity/http-bad-user-agent

works with Traefik (I got it triggered multiple times already)

Loz•12/8/25, 11:11 AM

ohh

Loz•12/8/25, 11:11 AM

I see the problem

Loz•12/8/25, 11:11 AM

you didnt actually define the file as a data property

Loz•12/8/25, 11:11 AM

using the helper doesnt load the file

Loz•12/8/25, 11:11 AM

you need to define

data:
  - dest_file: bad_user_agents.regex.txt
    type: regexp
    strategy: LRU
    size: 40
    ttl: 10s

data:
  - dest_file: bad_user_agents.regex.txt
    type: regexp
    strategy: LRU
    size: 40
    ttl: 10s

source_url can be removed just define the filename on disk.

Pierre HUBERTOP•12/8/25, 11:12 AM

Yes I removed it because the file is local (I placed it manually). But it remains mandatory? I'll try and come back to you!

PPierre HUBERT Yes I removed it because the file is local (I placed it manually). But it remain...

Loz•12/8/25, 11:13 AM

Yes as the helper doesnt load the file, the data property loads the file into memory. The helper simply check for the file in memory and uses it or errors if not loaded.

but explain doesnt show crowdsec errors.

Pierre HUBERTOP•12/8/25, 11:14 AM

Pierre HUBERTOP•12/8/25, 11:14 AM

I made the change and now I have detections

Loz•12/8/25, 11:14 AM

https://tenor.com/view/boom-alonzo-training-day-bang-explosion-gif-7668092698670143505

Tenor

Pierre HUBERTOP•12/8/25, 11:14 AM

Thank you a lot !!! It worked!

Loz•12/8/25, 11:15 AM

just note depending on your remediation component solely using captcha just allows them to hammer over and over. you might want to add another profile to check active decisions and convert to a ban and block at firewall level for those to reduce load.

Loz•12/8/25, 11:15 AM

shown here: https://docs.crowdsec.net/docs/next/local_api/profiles/captcha_profile

Pierre HUBERTOP•12/8/25, 11:18 AM

Thank you again for this help. Yes I shall escalate remediations to ban after multiple attempts. I am deploying Crowdsec progressively and will look at this in near future! Thank you again!

CCrowdSec Empty message

CrowdSecAPP•12/8/25, 11:18 AM

Resolving Create a local collection

CrowdSec•12/8/25, 11:18 AM

This has now been resolved. If you think this is a mistake please run

/unresolve

/unresolve