CrowdSec•2mo ago

High CPU load since restart haproxy bouncers

Hi, I have high CPU load since I restarted host with haproxy bouncers. I use a SQLITE database. The pprof result says slow request sqlite. I tried to migrate to a mariadb database, and it doesn't solve the CPU load. I have sometimes failed request with http code 500

time="2025-07-07T12:04:58+02:00" level=info msg="10.0.3.240 - [Mon, 07 Jul 2025 12:04:58 CEST] \"GET /v1/decisions/stream?startup=true HTTP/1.1 500 2m0.001530468s \"crowdsec-haproxy-bouncer/v1.0.0\" \""

time="2025-07-07T12:04:58+02:00" level=info msg="10.1.2.240 - [Mon, 07 Jul 2025 12:04:58 CEST] \"GET /v1/decisions/stream?startup=true HTTP/1.1 500 2m0.001225788s \"crowdsec-haproxy-bouncer/v1.0.0\" \""

time="2025-07-07T12:04:58+02:00" level=info msg="10.4.0.240 - [Mon, 07 Jul 2025 12:04:58 CEST] \"GET /v1/decisions/stream?startup=true HTTP/1.1 500 2m0.002370166s \"crowdsec-haproxy-bouncer/v1.0.0\" \""

time="2025-07-07T12:04:58+02:00" level=info msg="10.1.0.240 - [Mon, 07 Jul 2025 12:04:58 CEST] \"GET /v1/decisions/stream?startup=true HTTP/1.1 500 2m0.001156967s \"crowdsec-haproxy-bouncer/v1.0.0\" \""

time="2025-07-07T12:04:58+02:00" level=info msg="10.0.3.240 - [Mon, 07 Jul 2025 12:04:58 CEST] \"GET /v1/decisions/stream?startup=true HTTP/1.1 500 2m0.001530468s \"crowdsec-haproxy-bouncer/v1.0.0\" \""

time="2025-07-07T12:04:58+02:00" level=info msg="10.1.2.240 - [Mon, 07 Jul 2025 12:04:58 CEST] \"GET /v1/decisions/stream?startup=true HTTP/1.1 500 2m0.001225788s \"crowdsec-haproxy-bouncer/v1.0.0\" \""

time="2025-07-07T12:04:58+02:00" level=info msg="10.4.0.240 - [Mon, 07 Jul 2025 12:04:58 CEST] \"GET /v1/decisions/stream?startup=true HTTP/1.1 500 2m0.002370166s \"crowdsec-haproxy-bouncer/v1.0.0\" \""

time="2025-07-07T12:04:58+02:00" level=info msg="10.1.0.240 - [Mon, 07 Jul 2025 12:04:58 CEST] \"GET /v1/decisions/stream?startup=true HTTP/1.1 500 2m0.001156967s \"crowdsec-haproxy-bouncer/v1.0.0\" \""

These come from my differents hosts with bouncer. I observed that request with parameter startup=false does return 200 .. About mariadb, I have sometimes this error:

2025-07-07 12:08:59 411 [Warning] Aborted connection 411 to db: 'crowdsec' user: 'crowdsec' host: 'localhost' (Got an error writing communication packets)

2025-07-07 12:08:59 411 [Warning] Aborted connection 411 to db: 'crowdsec' user: 'crowdsec' host: 'localhost' (Got an error writing communication packets)

Can you provide me advice to debug more in depth ?

10 Replies

CrowdSec•2mo ago

Important Information

This post has been marked as resolved. If this is a mistake please press the red button below or type /unresolve

alendroitOP•2mo ago

The request 411 is :

411 Execute  SELECT `decisions`.`id`, `decisions`.`created_at`, `decisions`.`updated_at`, `decisions`.`until`, `decisions`.`scenario`, `decisions`.`type`, `decisions`.`start_ip`, `decisions`.`end_ip`, `decisions`.`start_suffix`, `decisions`.`end_suffix`, `decisions`.`ip_size`, `decisions`.`scope`, `decisions`.`value`, `decisions`.`origin`, `decisions`.`simulated`, `decisions`.`uuid`, `decisions`.`alert_decisions` FROM `decisions` LEFT JOIN `decisions` AS `t1` ON `t1`.`value` = `decisions`.`value` AND `t1`.`type` = `decisions`.`type` AND `t1`.`scope` = `decisions`.`scope` AND `t1`.`until` > `decisions`.`until` WHERE ((`decisions`.`until` < '2025-07-07 10:07:10.258885992' AND `t1`.`until` IS NULL) AND NOT `decisions`.`simulated`) AND `decisions`.`scope` IN ('Ip', 'Range') ORDER BY `decisions`.`id` ASC

411 Execute  SELECT `decisions`.`id`, `decisions`.`created_at`, `decisions`.`updated_at`, `decisions`.`until`, `decisions`.`scenario`, `decisions`.`type`, `decisions`.`start_ip`, `decisions`.`end_ip`, `decisions`.`start_suffix`, `decisions`.`end_suffix`, `decisions`.`ip_size`, `decisions`.`scope`, `decisions`.`value`, `decisions`.`origin`, `decisions`.`simulated`, `decisions`.`uuid`, `decisions`.`alert_decisions` FROM `decisions` LEFT JOIN `decisions` AS `t1` ON `t1`.`value` = `decisions`.`value` AND `t1`.`type` = `decisions`.`type` AND `t1`.`scope` = `decisions`.`scope` AND `t1`.`until` > `decisions`.`until` WHERE ((`decisions`.`until` < '2025-07-07 10:07:10.258885992' AND `t1`.`until` IS NULL) AND NOT `decisions`.`simulated`) AND `decisions`.`scope` IN ('Ip', 'Range') ORDER BY `decisions`.`id` ASC

And in crowdsec.log

time="2025-07-07T12:59:01+02:00" level=warning msg="QueryExpiredDecisionsWithFilters : context canceled"
time="2025-07-07T12:59:01+02:00" level=error msg="unable to query expired decision for 'bouncer1-haproxy' : expired decisions: unable to query"

time="2025-07-07T12:59:01+02:00" level=warning msg="QueryExpiredDecisionsWithFilters : context canceled"
time="2025-07-07T12:59:01+02:00" level=error msg="unable to query expired decision for 'bouncer1-haproxy' : expired decisions: unable to query"

I manage to retrieve normal CPU load after rebuild from scratch de LAPI. I re enroll all my machine and bouncer in fresh install. By doing that I losted all my previous data. Here is two more question: - Here is a trick to reimport all previous data in the fresh DB ? - Here is a procedure to follow when reload/restart haproxy service to not affect crowdsec service ?

blotus•2mo ago

How many decisions do you have ? The initial query is a bit heavier as it will retrieve everything, but it should not be that slow

alendroitOP•2mo ago

1519 including CAPI

blotus•2mo ago

sorry, I meant before you reinstalled everything

alendroitOP•2mo ago

origin crowdsec: 294 197 😮 total in db: 608 053

blotus•2mo ago

that's a bit on the high side If you have many bouncers querying LAPI at the same time, it could be an explanation

alendroitOP•2mo ago

I have 5 bouncers. Is there a way to delay query ? All bouncers are querying at the same time

blotus•2mo ago

you'd have to restart the bouncers one by one

CrowdSec•2mo ago

Resolving High CPU load since restart haproxy bouncers This has now been resolved. If you think this is a mistake please run /unresolve

Gaming

Programming

High CPU load since restart haproxy bouncers

Did you find this page helpful?