CrowdSec Runtime Error: High CPU Load and Memory Errors Causing Restarts
Hi team,
I’m encountering severe performance and stability issues when running CrowdSec with the AppSec component under high traffic conditions. Here are the details:
⸻
🔧 Setup:
• CrowdSec Engine: v1.6.9 (Docker)
• Subscribed to Console & Free Blocklist
• AppSec Component: Enabled with appsec-default rules using 4 routines
• Remediation: Nginx bouncer
• Traffic Load: ~1000 requests/sec
- Server: 8 CPU cores 16GB RAM
⸻
⚠️ Issue Summary:
After enabling the AppSec component:
• CPU usage spikes to 20–30% constantly.
• CrowdSec eventually consumes ~5GB of memory, then crashes and restarts.
• Observing frequent Nginx errors related to Lua timers.
⸻
🔍 Error Logs & Behavior:
NGINX Logs:
[error] lua entry thread aborted: runtime error: /usr/lib/crowdsec/lua/crowdsec.lua:305: Failed to create the timer: too many pending timers
Container logs before breaking:
fatal error: runtime: out of memory
...
runtime.stack: ... (full Go OOM trace follows)
Appsec config:
APPSEC_URL=http://127.0.0.1:7422
APPSEC_FAILURE_ACTION=passthrough
APPSEC_CONNECT_TIMEOUT=100
APPSEC_SEND_TIMEOUT=100
APPSEC_PROCESS_TIMEOUT=500
ALWAYS_SEND_TO_APPSEC=false
SSL_VERIFY=false
Could you help diagnose this?
• Is there a known leak or issue with AppSec timers or memory handling in this version?
• Any tuning suggestions to prevent crashes?
• Would upgrading or changing AppSec rules improve stability?
Thanks for your support — happy to provide more details or test patches!
13 Replies
Important Information
This post has been marked as resolved. If this is a mistake please press the red button below or type
/unresolve© Created By WhyAydan for CrowdSec ❤️
How did you install the remediation, can you see which version it is? as this was an issue we had in one of the versions but we thought we fixed it.
Does your service have a lot of file uploads?
I’m running CrowdSec NGINX Bouncer version 1.1.0 on Debian 11, installed through the apt package manager.
Due to dependency conflicts, I’m unable to install the latest version without first upgrading to Debian 12.
Yes, there are some file uploads as well.
The timer issue was fixed in 1.1.1 (unfortunately, the fix makes it incompatible with nginx 1.18 due to a bug in the debian version of nginx, so we bumped the minimum required version in 1.1.2)
Regarding the file uploads, what's the average size ?
Could the high CPU load and memory consumption be caused by a timer-related issue?
Would upgrading the machine along with a newer version of NGINX help mitigate this?
The file uploads are relatively small, approximately 50 KB to 150 KB each.
Yes as nginx will create a new timer meaning cpu/ram is used to store the timer and this does cause exhaustion in Nginx itself. It wouldnt cause CrowdSec to OOM but the first step to debugging would be to ensure all components are up to date to ensure we are not debugging a fixed issue.
After upgrading to Debian 12 (with NGINX v1.22 and nginx bouncer v1.1.2 now), the timer-related errors have been resolved. We’re no longer seeing memory exhaustion issues or frequent docker container restarts (previously occurring 3–4 times/hour).
However, we’re still noticing relatively high CPU usage, with around 20–25% more CPU load being consumed by a single routine in the AppSec component.
Do you have any suggestions for configuration optimizations to reduce the CPU impact of CrowdSec?
How many requests per second are you processing ?
Which rules/collections have you enabled ?
Approximately 700–1000 requests/sec.
For AppSec, we’re using the appsec-default appsec-config with the following enabled rules:
In-band rules:
• crowdsecurity/base-config
• crowdsecurity/vpatch-*
• crowdsecurity/generic-*
Out-of-band rules:
• crowdsecurity/experimental-*
• crowdsecurity/appsec-generic-test
I'd need to try to do some more advanced benchmarking to see where exactly the time is spent, but I don't find it particularly high if you are analyzing 1k requests per second.
Were you using another WAF previously that used less CPU ? (we have definitely things to optimize, but again, I'm not really shocked at a 25% CPU usage)
We weren’t using another WAF. The 20% additional CPU usage isn’t a major issue, but I wanted to double-check in case there are known optimizations that could help reduce the load.
The new crowdsec WAF component is really an amazing tool to add another security level on our infra. Keep going and provide more appsec rules for well known CVEs.
Thank you! glad to see you got value out of it, there are a few internal optimization that we can do inside the codebase but on a user end pretty much only loading what you need / general maintence is pretty much what you can do for now, obviously increasing the routines would save time spent with an increase in cpu usage, but since the problem isnt a bottle neck than this is fine for now.
Resolving CrowdSec Runtime Error: High CPU Load and Memory Errors Causing Restarts
This has now been resolved. If you think this is a mistake please run
/unresolve