crowdsec-nginx-bouncer memory leak?
On a debian bookworm system with nginx (version 1.22.1-9+deb12u2) I try to install and run crowdsec-nginx-bouncer. As soon as the crowdsec-nginx-bouncer is configured, nginx gets regularly killed by oom.
The crowdsec lapi is running on a different machine.
As an example a "nginx -t" only takes 2 seconds to complete without crowdsec-nginx-bouncer and with installed/configured/enabled crowdsec-nginx-bouncer the command "nginx -t" takes at least 1 minute.
A "nginx -s reload" acts quickly (under 3 seconds) without crowdsec-nginx-bouncer and with installed/configured/enabled crowdsec-nginx-bouncer the nginx service gets killed by "out of memory".
Normally, the nginx process takes about 1 gigabyte of ram on this machine. The machine has 6 gigabytes of memory. With crowdsec-nginx-bouncer and reloading nginx with "nginx -s reload" it quickly jumps up to 4 gigabytes and then gets oomed.
Does anybody have the same problem? How do I debug this? Where could be the error? Thank you for your help.
8 Replies
Important Information
This post has been marked as resolved. If this is a mistake please press the red button below or type
/unresolve© Created By WhyAydan for CrowdSec ❤️
There was an old mod lua bug (https://github.com/openresty/lua-nginx-module/issues/1005) that could lead to very high memory usage, but it has been fixed a while back (I don't know which version of mod_lua ships with bookworm)
Can you try to comment the
lua_ssl_trusted_certificate directive in /etc/nginx/conf.d/crowdsec-nginx.conf and restart/reload ? (this directive is only required if you are using captcha remediation / querying LAPI over HTTPS)GitHub
A "memory trick" in
ngx_http_lua_limit_data_segment leads to redu...Nginx version: 1.11.9 Lua Nginx Module version: 0.10.6 Openssl version: 1.1.0e OS: Linux 3.19.0-80-generic Recently I added lua_ssl_trusted_certificate directive to main config section of one of th...
Oookey, thank you very much! This seems to be the problem! So in this case, I can't use yet the captcha remediation and/or a HTTPS connection to a LAPI on a debian bookworm system.
Debian bookworm ships with
libnginx-mod-http-lua version 0.10.23-1
Just for my understanding. This means it will probably work with the lua_ssl_trusted_certificate with a new debian release (which is just around the corner)?Just to confirm, you have removed the directive and it works fine now ?
actually, reading the issue in more details, it's actually a glibc issue.
But I'm not sure what caused this issue to be triggered on your system, we haven't had report of it for a long time.
Do you have a lot of trusted certificates on the system ? (more than the default certs)
No, only the default certs. It's a pretty standard debian installation.
Should I create a bug report on github?
Resolving crowdsec-nginx-bouncer memory leak?
This has now been resolved. If you think this is a mistake please run
/unresolveI digged a little bit further. On the machine I have a lot of letsencrypt certificates ... Now I just supply a concatenated certificate with all letsencrypt root certifcates to
lua_ssl_trusted_certificate. As I use hcaptcha (which relies on Letsencrypt aswell), this seems to work.GitHub
Memory leak with crowdsec-nginx-bouncer and nginx? (because of lua_...
On a Debian bookworm system with nginx (version 1.22.1-9+deb12u2) I try to install and run crowdsec-nginx-bouncer. As soon as the crowdsec-nginx-bouncer is configured, nginx gets regularly killed b...