Should Prometheus Work in Multi-Server Setup?
Really quick question that I couldn't find the answer to anywhere. I have a multi-server setup pushing all logs to my main machine/security engine. 3-4 machines are pushing into the Sec Engine.
On the main sec engine, prometheus is working without issue in the default
config.yml. On a secondary, but important machine, I have prometheus set up the same as default but it isn't working. I get a scrape config timeout from Prometheus and and error in docker logs that it couldn't mind to the address. I've exposed the 6060 port in my docker compose file.
My hunch is that prometheus metrics are dependent on the LAPI being up and running. Can anyone confirm or deny if this is the case? Should metrics be available on any machine regardless of the rest of the configuration?
Thanks in advance!5 Replies
Important Information
Thank you for getting in touch with your support request. To expedite a swift resolution, could you kindly provide the following information? Rest assured, we will respond promptly, and we greatly appreciate your patience. While you wait, please check the links below to see if this issue has been previously addressed. If you have managed to resolve it, please use run the command
/resolve or press the green resolve button below.Log Files
If you possess any log files that you believe could be beneficial, please include them at this time. By default, CrowdSec logs to /var/log/, where you will discover a corresponding log file for each component.
Guide Followed (CrowdSec Official)
If you have diligently followed one of our guides and hit a roadblock, please share the guide with us. This will help us assess if any adjustments are necessary to assist you further.
Screenshots
Please forward any screenshots depicting errors you encounter. Your visuals will provide us with a clear view of the issues you are facing.
© Created By WhyAydan for CrowdSec ❤️
Probably lapi accepts connections from localhost only.
I rmember i had some weird issue that even swtting listen on 0.0.0.0 ended in permission denied, so i added proxy pass on nginx 😉
Can you show your prometheus configuraiton on the connecting machines?
as long as you set
enabled to true within the prometheus configuration block it should work
but in short no prometheus is not dependant on LAPI being enabledthat's exactly my configuration - enable true, listen 0.0.0.0 and port 6060. Let me see if there are any more relevant logs.
Okay, i figured it out, but it doesn't quite make sense. On my sec engine machine, running
cscli machines list I saw that none of the machines were sending a heart beat. From one of the machines I ran cscli lapi status and I got an affirmative message that I could interact with the LAPI on the sec engine.
All of a sudden every other machine showed valid and recent heartbeats and from prometheus I could see the target was online. 🤷🏻♂️CNI?