Haproxy bouncer try to pull expired decisions from LAPI

Hello, I have crowdsec api error with haproxy bouncer. Here is the log

time="2025-02-20T10:04:03+01:00" level=error msg="unable to query expired decision for 'node01-haproxy' : expired decisions: unable to query"
time="2025-02-20T10:04:09+01:00" level=warning msg="QueryExpiredDecisionsWithFilters : context canceled"

time="2025-02-20T10:04:03+01:00" level=error msg="unable to query expired decision for 'node01-haproxy' : expired decisions: unable to query"
time="2025-02-20T10:04:09+01:00" level=warning msg="QueryExpiredDecisionsWithFilters : context canceled"

I already tried to restart both crowdsec and haproxy on 'node01-haproxy' and also crowdsec LAPI itself. The error occurred after the haproxy service was reloaded on 'node01-haproxy'. The crowdsec LAPI cpu is at 100% since then. Someone already had this problem ? Is there a way to debug more in depth ?

5 Replies

CrowdSec•7mo ago

Important Information

Thank you for getting in touch with your support request. To expedite a swift resolution, could you kindly provide the following information? Rest assured, we will respond promptly, and we greatly appreciate your patience. While you wait, please check the links below to see if this issue has been previously addressed. If you have managed to resolve it, please use run the command /resolve or press the green resolve button below.

Log Files

If you possess any log files that you believe could be beneficial, please include them at this time. By default, CrowdSec logs to /var/log/, where you will discover a corresponding log file for each component.

Guide Followed (CrowdSec Official)

If you have diligently followed one of our guides and hit a roadblock, please share the guide with us. This will help us assess if any adjustments are necessary to assist you further.

Screenshots

Please forward any screenshots depicting errors you encounter. Your visuals will provide us with a clear view of the issues you are facing.

blotus•7mo ago

Hello, How many decisions (non-expired and expired) do you have in the database ? (you can check with cscli decisions list -a, it should show even the expired ones). The error makes me think the database request took too long, and so the request made by haproxy timed out. If the CPU is still at 100%, can you get a cpu profile ? On the LAPI, run curl 127.0.0.1:6060/debug/pprof/profile\?debug=1 -o cpu.pprof (it will take 30s to run)

alendroitOP•7mo ago

How many decisions => a lot of CAPI and 12 other Result of curl 127.0.0.1:6060/debug/pprof/profile\?debug=1 -o cpu.pprof cpu back to normal

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   618  100   618    0     0     20      0  0:00:30  0:00:30 --:--:--   163

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   618  100   618    0     0     20      0  0:00:30  0:00:30 --:--:--   163

EDIT: 50520 total decision (cscli decision list -a -o raw | wc -l)

blotus•7mo ago

Could you share the cpu.pprof file ? 50k decisions is not that much, so it's a bit surprising LAPI took that much CPU (or long enough for the request to timeout)

alendroitOP•7mo ago

cpu.pprof

Gaming

Programming

Haproxy bouncer try to pull expired decisions from LAPI

Did you find this page helpful?