Failed sending alert to LAPI, Invalid character.
Hi, I do get this error when looking at the docer logs:
time="2025-09-21T15:00:00+02:00" level=error msg="while pushing to api : failed sending alert to LAPI: API error: invalid character '\x1f' looking for beginning of value"
My Setup has a OPNsense which acts a the LAPI and a Debian Server with Crowdsec and Nginx Proxy Manager Container. The setup was functional before and when I check the OPNSense I can see alerts from a day before. I'm not really sure if I did something that broke it or if onlye some alerts get pushed to the opnsense... I added a new service to my nginx proxy manager today and thats why I looked into crowdsec to check if everything is working.
I cleared all the logs, also tested a manual ban from the cscli console of the debian server and this got pushed to the opnsense without a problem.
I can't really find anything whenn googling for the error message.
7 Replies
Important Information
Thank you for getting in touch with your support request. To expedite a swift resolution, could you kindly provide the following information? Rest assured, we will respond promptly, and we greatly appreciate your patience. While you wait, please check the links below to see if this issue has been previously addressed. If you have managed to resolve it, please use run the command
/resolve or press the green resolve button below.Log Files
If you possess any log files that you believe could be beneficial, please include them at this time. By default, CrowdSec logs to /var/log/, where you will discover a corresponding log file for each component.
Guide Followed (CrowdSec Official)
If you have diligently followed one of our guides and hit a roadblock, please share the guide with us. This will help us assess if any adjustments are necessary to assist you further.
Screenshots
Please forward any screenshots depicting errors you encounter. Your visuals will provide us with a clear view of the issues you are facing.
© Created By WhyAydan for CrowdSec ❤️
looks like it's related to error in processing journal logs? https://github.com/crowdsecurity/crowdsec/issues/3890
GitHub
Journalctl acquisition · Issue #3890 · crowdsecurity/crowdsec
What happened? it seems that the journalctl acquisition module has been probably ported from fmt.Sprintf and now crowdsec/pkg/acquisition/modules/journalctl/journalctl.go Line 202 in 4e08b88 j.src ...
okay i looked into it a little bit and don't think it is related. I run tcpdump on the opnsense and it seems that the crowdsec client on the debian server sends a POST request to /v1/alerts with a gzip compressed json file. The LAPI on the OPNSense however does not expect the json to be compressed and responsd with this errormessage: {"message":"invalid character '\x1f' looking for beginning of value"}
The headers on the initial Post Request are correct:
Content-Encoding: gzip
Content-Type: application/json
Accept-Encoding: gzip
Okay I did some further troubleshooting and I think it is related to this change in version 1.7.0:
- https://github.com/crowdsecurity/crowdsec/pull/3709
There it sates: This also enables gzip compression for bodies bigger than 5KB for LP->LAPI and LAPI->CAPI communication, as the metrics payload can get quite big.
I went back to version v1.6.11 and the error message is gone and it is working again. Not sure if this is a problem with the new implementation or with the plugin on OPNsense though.
GitHub
usage metrics: report acquisition + parsers metrics to LAPI by blot...
Send more detailed metrics about acquisition and parsing to LAPI/CAPI to help users troubleshoot.
New usage metrics:
read: Track how many lines were read from a datasource (with datasource_type, a...
I did create a Issue on the opnsense Github: https://github.com/opnsense/plugins/issues/4947
GitHub
CrowdSec LAPI fails to handle gzipped requests from Agent >= v1.7.0...
Hello, Since the CrowdSec Agent v1.7.0, the client uses Content-Encoding: gzip for requests larger than 5KB to the LAPI, as introduced in crowdsecurity/crowdsec#3709 The server-side endpoint manage...
most likely @Hackstronaut you lapi is on a older version (
1.6.11) and the other is 1.7.0? if so this may cause issues as both LAPI and other deployments should stick to the same version. On opnsense it lags behind as we have to push to freshports which then opnsense does rolling updates so we cannot control when it updates.
howver, I strongly suggest it is the original issue that @KaszpiR linked, did you check to make sure you wasnt using an journalctl acquisitions? (which is very common on debian 12 for example as it ships with no rsyslog now)No, I'm not using any journalctl acquisitions. But the problem is the same yes, it fails at parsing the json but the reason seems different. At the moment with Version 1.6.11 it is working so i wait until the OPNSense Plugin is updated.
I have these versions installed:
- os-crowdsec (OPNsense Plugin): 1.0.12
- crowdsecurity/crowdsec Docker Container = v1.7.0 (With this configuration I do have the problem)
With a downgrade to 1.6.11 in the docker container it is working again.
Should I mark this as solved or wait until a new OPNSense version is out and test it again?