CrowdSec•6mo ago

CrowdSec NGINX Bouncer internal server error

After upgrading from 1.0.9 to 1.1.0 I started getting intermittent http 500 errors. The bouncer worked fine for a few hours until I started to get intermittent notification that some of my services were down (with an http 500 error code).

2025/03/24 09:31:51 [error] 960177#960177: *12552450 lua entry thread aborted: runtime error: /usr/lib/crowdsec/lua/crowdsec.lua:305: Failed to create the timer: too many pending timers
stack traceback:
coroutine 0:
        [C]: in function 'error'
        /usr/lib/crowdsec/lua/crowdsec.lua:305: in function 'SetupStream'
        /usr/lib/crowdsec/lua/crowdsec.lua:341: in function 'allowIp'
        /usr/lib/crowdsec/lua/crowdsec.lua:553: in function 'Allow'
        access_by_lua(conf.d/crowdsec_nginx.conf:19):6: in main chunk, client: 0.0.0.0, server: example.com, request: "GET / HTTP/1.1", host: "example.com"

2025/03/24 09:31:51 [error] 960177#960177: *12552450 lua entry thread aborted: runtime error: /usr/lib/crowdsec/lua/crowdsec.lua:305: Failed to create the timer: too many pending timers
stack traceback:
coroutine 0:
        [C]: in function 'error'
        /usr/lib/crowdsec/lua/crowdsec.lua:305: in function 'SetupStream'
        /usr/lib/crowdsec/lua/crowdsec.lua:341: in function 'allowIp'
        /usr/lib/crowdsec/lua/crowdsec.lua:553: in function 'Allow'
        access_by_lua(conf.d/crowdsec_nginx.conf:19):6: in main chunk, client: 0.0.0.0, server: example.com, request: "GET / HTTP/1.1", host: "example.com"

2025/03/24 09:20:50 [error] 960177#960177: *12195385 lua entry thread aborted: runtime error: /usr/lib/crowdsec/lua/crowdsec.lua:290: Failed to create the timer: too many pending timers
stack traceback:
coroutine 0:
        [C]: in function 'error'
        /usr/lib/crowdsec/lua/crowdsec.lua:290: in function </usr/lib/crowdsec/lua/crowdsec.lua:256>, context: ngx.timer, client: 0.0.0.0, server: 0.0.0.0:443

2025/03/24 09:20:50 [error] 960177#960177: *12195385 lua entry thread aborted: runtime error: /usr/lib/crowdsec/lua/crowdsec.lua:290: Failed to create the timer: too many pending timers
stack traceback:
coroutine 0:
        [C]: in function 'error'
        /usr/lib/crowdsec/lua/crowdsec.lua:290: in function </usr/lib/crowdsec/lua/crowdsec.lua:256>, context: ngx.timer, client: 0.0.0.0, server: 0.0.0.0:443

I've downgraded to 1.0.8 in the meantime

23 Replies

CrowdSec•6mo ago

Important Information

Thank you for getting in touch with your support request. To expedite a swift resolution, could you kindly provide the following information? Rest assured, we will respond promptly, and we greatly appreciate your patience. While you wait, please check the links below to see if this issue has been previously addressed. If you have managed to resolve it, please use run the command /resolve or press the green resolve button below.

Log Files

If you possess any log files that you believe could be beneficial, please include them at this time. By default, CrowdSec logs to /var/log/, where you will discover a corresponding log file for each component.

Guide Followed (CrowdSec Official)

If you have diligently followed one of our guides and hit a roadblock, please share the guide with us. This will help us assess if any adjustments are necessary to assist you further.

Screenshots

Please forward any screenshots depicting errors you encounter. Your visuals will provide us with a clear view of the issues you are facing.

blotus•6mo ago

Hey, Could you maybe try to run nginx with debug enabled for the error log ? It will log how many timers are existing before creating new one. If you cannot run in debug mode, you can also try to edit /usr/lib/crowdsec/lua/crowdsec.lua and replace the ngx.DEBUG by ngx.INFO line 298 (this will log every UPDATE_FREQUENCY seconds)

GNU Plus Windows UserOP•6mo ago

That will take a while for me to do and really clutter up my logs. Just FYI I'm using stream mode with UPDATE_FREQUENCY set to 2 I'll report back once I reproduced and have the logs

blotus•6mo ago

ok, no worries, I'll try to see if I can reproduce how many workers do you run ? in theory, the max number of timer active at any time should be 2 * workers count

GNU Plus Windows UserOP•6mo ago

mine is set to auto, the VM has 4 CPU cores assigned to it.

blotus•6mo ago

ok so 4 workers by default, the lua module allows up to 1024 pending timers and 256 running if you look at /var/log/crowdsec_api.log, do you see requests to /decisions/stream that take more than 2 seconds ?

GNU Plus Windows UserOP•6mo ago

No, but I've noticed that the requests are around 200ms for each request for both versions

blotus•6mo ago

and how many requests per second does nginx receive ? (on average, and burst if you know)

GNU Plus Windows UserOP•6mo ago

not a whole lot, its a very low traffic site. sometimes the only visitors are bots.

blotus•6mo ago

ok nothing weird happened before the error occurred ? No error in the logs or something ?

GNU Plus Windows UserOP•6mo ago

nope everything was working fine

blotus•6mo ago

just to be sure, is lua_max_pending_timers set anywhere in your nginx config ?

GNU Plus Windows UserOP•6mo ago

no I've upgraded back to 1.1.0 and I've noticed the pending timers are slowly ticking up (It's been about 20 or so minutes)

pending timers: 15

Not sure if this is an issue, but I'll keep my eye on it.

blotus•6mo ago

I saw them rising on my test machine as well, but it was going up and down It has been running for a few days, and I'm at 6 currently

GNU Plus Windows UserOP•6mo ago

mine is at 892 now 100% my issue

blotus•6mo ago

Which nginx version are you running / which lua module version ?

GNU Plus Windows UserOP•6mo ago

OS: Ubuntu 24.04 Nginx Version: 1.24.0 Nginx Modules: * libModSecurity3 3.0.14 * Brotli * Lua > 1:0.10.26-2 * NDK > 1:0.3.3-1build1 * http-headers-more-filter > 1:0.37-2build1 fyi I downgraded to 1.0.8 since it's obvious stuff's about to break. I can go back to 1.1.0 if you need me to test something.

blotus•6mo ago

We have a PR (https://github.com/crowdsecurity/lua-cs-bouncer/pull/105 and https://github.com/crowdsecurity/cs-nginx-bouncer/pull/76) that will change the way the timer are creating, simplifying the code and avoiding trying to do it each time we receive a request, so in theory it should fix the number of timers to 2*worker count (I haven't tested it yet, and the tests seem broken, but I will let you know once we managed to test it in depth/released a new version)

GitHub

feat: call SetupStream and SetupMetrics from nginx configuration by...

call SetupStream and SetupMetrics from nginx configuration

blotus•5mo ago

Hey @GNU Plus Windows User We have released a new RC of the bouncer that will start the timers during nginx startup, which should hopefully fix your issue. If you want to try it, go ahead and let us know if it seems to be better

GNU Plus Windows UserOP•5mo ago

Are all the files in the t directory tests? I can't find them in my local install

blotus•5mo ago

yes

GNU Plus Windows UserOP•5mo ago

Just upgraded to the release candidate, nothing's caught on fire so far. I think the issue is fixed now I'll report back if something breaks

blotus•5mo ago

great, thanks for the feedback

Gaming

Programming

CrowdSec NGINX Bouncer internal server error

Did you find this page helpful?