CrowdSec Agent UnmarshalJSON: unexpected end of JSON input with Traefik logs on Kubernetes (Containe
Hi everyone,
I'm facing a persistent issue with my CrowdSec agent on Kubernetes.
My Setup:
CrowdSec: Deployed via Helm chart.
Container Runtime: containerd is configured (container_runtime: containerd in values.yaml).
Log Source: Traefik access logs.
Traefik Configuration: Access logs are set to json format (--accesslog.format=json).
Problem: The CrowdSec agent continuously throws UnmarshalJSON : unexpected end of JSON input errors, as shown in the attached screenshot.
What I've already tried:
Ensuring Traefik outputs JSON: Confirmed Traefik is generating JSON access logs (--accesslog.format=json).
Reducing Traefik log size: Configured Traefik to include only essential fields in JSON logs (--accesslog.fields.defaultmode=drop and keeping specific fields) to make lines shorter.
CrowdSec acquisition: Configured CrowdSec agent's acquisition to target Traefik pods (namespace: traefik, podName: traefik-, program: traefik).
Verifying log content on node: When I tail -f the Traefik log files (/var/log/containers/traefik-qa-.log) directly on the Kubernetes node, I can see the JSON lines, but they appear truncated at the end, confirming the 'unexpected end of JSON input' error. (Example: {"ClientAddr":"10.244.116.192:13977",...,"OriginD)
My understanding:
It seems the Kubernetes container runtime (containerd) is truncating the Traefik JSON log lines before they are fully written, causing invalid JSON for CrowdSec to parse. The standard solution is to increase MaxLogSize in containerd/kubelet config on the nodes.
My question:
Is there any alternative solution or a recommended workaround within the CrowdSec or Traefik configuration (or at the application level) to handle these truncated JSON logs, if direct modifications to MaxLogSize on the Kubernetes cluster nodes are difficult or not feasible? Or, are there specific CrowdSec parsing/acquisition strategies for containerd logs that can tolerate or recover from this kind of truncation?
3 Replies
Important Information
Thank you for getting in touch with your support request. To expedite a swift resolution, could you kindly provide the following information? Rest assured, we will respond promptly, and we greatly appreciate your patience. While you wait, please check the links below to see if this issue has been previously addressed. If you have managed to resolve it, please use run the command
/resolve or press the green resolve button below.Log Files
If you possess any log files that you believe could be beneficial, please include them at this time. By default, CrowdSec logs to /var/log/, where you will discover a corresponding log file for each component.
Guide Followed (CrowdSec Official)
If you have diligently followed one of our guides and hit a roadblock, please share the guide with us. This will help us assess if any adjustments are necessary to assist you further.
Screenshots
Please forward any screenshots depicting errors you encounter. Your visuals will provide us with a clear view of the issues you are facing.
© Created By WhyAydan for CrowdSec ❤️
You could reconfigure traefik to log to a pvc mounted in the traefik container, and then have a custom sidecar or another pods to read those logs and process it further. That could be a normal volume or hostpath shared among the containers in the same or different pods.
This way coul could have a minimal logs going to containerd and the more rich logs in separate location ,to avoid issues with containerd.
Or you could reconfigure containerd via daemonset (or cloud -init) to allow processing longer lines if required.
So let say you have a hostpath shared - it needs to be shar d between traefik and crowdsec agent, and of course you need to configure traefik to log to s dedicated file there, maybe updated per hour or whatever, and crowdsec to read from it and be aware it is a multiline json file in a given format. Instead of hostpath you could also use a dedicated volumes for that. Either way it would be a direct path between traefik and crowdsec agent.
Another option is to do something similar to the above setup but then just send logs to the central location using for example promtail/fluentd/fluentbit and so on. But then the crowdsec agent would have to fetxh logs from that central location ( or intermediate aggregation point for faster processing and less buffering) and then invoke actions based on those logs ( in my use case it is a loki instance and running specific query by he agent to fetch logs which are ingress specific only or depends on your scenarios etc)
Thanks for the help! I'm going to make this change