Wrong Grok Pattern for Custom Parser
Hi !
I created a custom parser that have to match 2 log templates :
This one:
10.6.0.1 - - [28/Feb/2025:09:20:15 +0000] "GET /health HTTP/1.1" 200 3 "-" "kube-probe/1.30" "-" "10.6.0.1" 107 0.000 - - - - - - - "- -”
And this one: 1.11.10.11 - - [03/Dec/2024:13:57:32 +0000] "PUT /xx/v1/xx/x HTTP/1.1" 200 1468 "https://xx.fr" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.6668.29 Safari/537.36" "1.11.10.11, 1.11.10.12" "1.11.10.13" 3887 1.053 - 10.120.0.196:82 1468 0.000 1.053 1.054 200 "TLSv1.3 1301"
Here is my custom parser configuration:
%{IP:source_ip} - - \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status_code} %{NUMBER:bytes} "%{DATA:referer}" "%{DATA:user_agent}" "%{DATA:forwarded_for}" "%{IP:upstream_ip}" %{NUMBER:response_time_ms:int} %{NUMBER:request_time:float} - %{IP:upstream_host}:%{NUMBER:upstream_port:int} %{NUMBER:upstream_bytes:int} %{NUMBER:upstream_request_time:float} %{NUMBER:upstream_response_time:float} %{NUMBER:total_time:float} %{NUMBER:final_status} "%{DATA:tls_version_cipher}”
When looking to my grafana dashboard, I see cs_node_hits_ok_total metrics for the following grok pattern:
%{IP:source_ip} - - \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status_code} %{NUMBER:bytes}
But I have cs_node_hits_ko_total metrics with the grok pattern in the configuration I shared previously.
Can you help me?
Kind regards13 Replies
Important Information
Thank you for getting in touch with your support request. To expedite a swift resolution, could you kindly provide the following information? Rest assured, we will respond promptly, and we greatly appreciate your patience. While you wait, please check the links below to see if this issue has been previously addressed. If you have managed to resolve it, please use run the command
/resolve or press the green resolve button below.Log Files
If you possess any log files that you believe could be beneficial, please include them at this time. By default, CrowdSec logs to /var/log/, where you will discover a corresponding log file for each component.
Guide Followed (CrowdSec Official)
If you have diligently followed one of our guides and hit a roadblock, please share the guide with us. This will help us assess if any adjustments are necessary to assist you further.
Screenshots
Please forward any screenshots depicting errors you encounter. Your visuals will provide us with a clear view of the issues you are facing.
© Created By WhyAydan for CrowdSec ❤️
since the kubeprobe doesnt really have those information it wont accept
- as acceptable for IP or NUMBER
so you can instead mark the trailing as optional
I zoom on the change
Since wrapping the data in ()? means it optional to parse
ooh wait I messed up, fixed
also
I saw in your original
” is not an " even though they look the same near the tls cipher but that was causing an issueALright so I can test this one ?
Oh you're right ! It is because I copy/paste in my Notion app that inserts trailing
I had also tested on this website and the pattern I wrote was correct
Yes that one I tested and worked
and which type do you expect to match which parser name, might be useful to show the full parser
idk why the grok pattern doesn't work on crowdsec
how did you proceed to test ?
I tested via grokdebugger so it could mean another issue
can you dump the raw parser in DM's
ok
I sent it