ingress nginx kubernetes + custom logs
The logs are being parsed, but no further events occur.
Nginx has custom logs, but I created a parser.
Could you please advise me on how to resolve this issue?
25 Replies
Important Information
This post has been marked as resolved. If this is a mistake please press the red button below or type
/unresolveΒ© Created By WhyAydan for CrowdSec β€οΈ
I have followed the troubleshooting steps, but unfortunately the issue still persists.
Can you provide the full output with
-v passed to the explain command, this will show the log line details so if you have any PII you would need to sed the output or sanitize before sending.here is the output of log processing
and custom parser
Okay parsing seems fine, but there no matching scenarios, so if you run
cscli scenarios list from within the pod (agent) do you see any installed?but if I check the raw log from Docker, the transition to other parsers does not happen. Could that be the issue?
scenarios
That is not docker logs that is
containerd
so you should be running cscli explain ... --type containerd --labels "program:nginx"
ref: https://app.crowdsec.net/hub/author/crowdsecurity/log-parsers/cri-logs
Also your parser needs to use evt.Parsed.message instead of evt.Line.Raw as the raw line include the containerd prefixesupdate parser
cri log - cscli explain ... --type containerd --labels "program:nginx"
nginx log - cscli explain ... --type nginx
kubectl -n crowdsec logs pods/crowdsec-agent-q4txc --tail 10
Defaulted container "crowdsec-agent" out of: crowdsec-agent, wait-for-lapi-and-register (init)
time="2025-09-02T09:36:07Z" level=error msg="JsonExtractObject: key proxy does not exist"
time="2025-09-02T09:36:07Z" level=error msg="JsonExtractObject: key upstream does not exist"
time="2025-09-02T09:36:07Z" level=error msg="JsonExtractObject: key http does not exist"
time="2025-09-02T09:36:07Z" level=error msg="JsonExtractObject: key http does not exist"
time="2025-09-02T09:36:07Z" level=error msg="JsonExtractObject: key proxy does not exist"
time="2025-09-02T09:36:07Z" level=error msg="JsonExtractObject: key upstream does not exist"
create evt.Parsed.http_referer : [32mhttps://new.site.online/ship <-- from "http": {"referer": "https://new.site.online/ships/celestyal-discovery","user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36","accept_encoding": "gzip, deflate, br, zstd"}
((((((
let me cook up a parser using
UnmarshalJSON
its faster, and more runtime efficient
Hey @bearded
what is the intention from these groks?
this is an attempt to not process logs containing the string Error obtaining Endpoints for Service
ahhh, okay my updated parser will just ignore them anyways
this line appeared out of desperation and attempts to fix log parsing (
This works for me and output is
ohh wait I just need to update it
updated it, by default all numbers are floats so we have to wrap status in
int() callThe parser worked perfectly! Thank you!
What was my mistake? Why didnβt it move on to the next parser?
and...
crowdsec-agent-6vn8b:/# cscli decisions list
No active decisions
time="2025-09-02T10:56:43Z" level=warning msg="failed to run filter : unexpected end of JSON input (1:42)\n | evt.Parsed.program startsWith 'nginx' && UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, 'nginx') in [nil, '']\n | .........................................^" id=wandering-snow name=crowdsecurity/custom-nginx-logs stage=s01-parse
Hooray! Youβre amazing! We managed to fix it! HUGE THANKS!
We needed to override the variable in the Helm chart container_runtime: containerd.
Unfortunately, I couldnβt find any documentation on this issue.
Sorry yes I should have added this part, as I did mention it wasnt docker π
in the agent logs there are errors
is this normal or should it be fixed?
time="2025-09-02T12:54:04Z" level=error msg="UnmarshalJSON : invalid character 'W' looking for beginning of value" line="W0902 12:54:04.346346 7 controller.go:1107] Error obtaining Endpoints for Service "logs/devnull": no object matching key "logs/devnull" in local store"
time="2025-09-02T12:54:04Z" level=warning msg="failed to run filter : invalid character 'W' looking for beginning of value (1:42)\n | evt.Parsed.program startsWith 'nginx' && UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, 'nginx') == nil\n | .........................................^" id=throbbing-sound name=crowdsecurity/custom-nginx-logs stage=s01-parse
Probably you have non json logs being printed?
if so you can add to the filter like
so in short we check the message starts with
{ indicating a json log, depending on it also we can check for stdout or stderr from containerdyes, the agents are monitoring nginx through which many services pass. It turned out that there are non-normalized logs there.
cool! the fix works
once again, huge thanks
Resolving ingress nginx kubernetes + custom logs
This has now been resolved. If you think this is a mistake please run
/unresolve