Proposal: Native Prometheus Metrics Support
Hi,
I wanted to ask if anyone has considered the possibility of implementing native support for exposing Prometheus metrics, without requiring a plugin. I believe this would be a valuable feature, as it would allow metrics to be exported via Telegraf and sent to InfluxDB or VictoriaMetrics.
I’m willing to work on the implementation myself if someone can provide guidance to ensure the code remains consistent with the existing standards.
Thanks,
Andrea
22 Replies
What do you mean "without a plugin"?
I mean without the need of installing a third party plugin and to be implemented directly into signalk. Right now the prometheus plugin it isn't working and probably not mantained anymore.
I don't think we would add it to the server proper, since it's pretty platform specific
but prometheus metrics now days is a standard, and not platform specific. If you guys don't have time to do it I can do a PR myself.
I just want to make sure it is something we want to add to the server before you start any work. In my opinion, it belongs in a plugin
But @Teppo Kurki should chime in
Agreed, unless i am missing something here? There are plenty of plugins that either send sk data to another service, over some protocol
Or expose a new service/protocol
I will write the plugin if you think it can’t be a useful feature for most users. The thing is that most of the plugins that I saw are platform related. With Prometheus you can have a standard of exporting metrics to third party services without the need of developing a specific plugin for every technology. The implementation on the website is pretty straightforward just needed to have an endpoint /metrics that expose the date in plaintext and standard Prometheus format.
What would be the difference to https://www.npmjs.com/package/@ib236/signalk-prometheus-exporter
npm
@ib236/signalk-prometheus-exporter
Signal K server plugin expose a Prometheus pull end point. Latest version: 0.0.3, last published: 6 years ago. Start using @ib236/signalk-prometheus-exporter in your project by running
npm i @ib236/signalk-prometheus-exporter. There are no other projects in the npm registry using @ib236/signalk-prometheus-exporter.The plugin i believe isn't maintained anymore and it isn't working, is it a problem of my instace of signalk?
I’ve never used P myself
Apart from it not working 😆 what would be the difference?
As I am not familiar with P trying to understand things..
Just as a background: in retrospect some of the components that are part of the server proper could/should have been plugins, just that plugin mechanism came after…
So plugin vs built in is not a value judgment
Thanks for the context! The main idea behind implementing directly on the server is to provide an official, standardized, maintainable way to expose metrics without needing to fork or create a new project (plugin) each time.
In my industry (Cybersecurity) P is widely used to monitor metrics.
I could have started by building a standalone plugin, but I thought it would be better to implement it directly so it can be officially maintained by the community.
A plugin can be all of those qualities, like several already are, for example signalk to nmea0183 and 2000 plugins
Please don’t get stuck on direct vs plugin - if i were doing this myself i would make it a plugin
I just wanted to understand why it can’t be a main feature. That’s it if you telling me that developing a plugin is a better choice I trust you, you know your product better than anyone 😅
I thought it was more a server side feature, like api requests. That’s it
Lots of plugins do that.
And lots of benefits to doing stuff like this in a plugin.
No worries I don’t want to fight for it. I do want just the feature and help with the development. But the only thing is that I don’t understand why should be better to have a plugin that does that.
Think of the server as a “framework” for implementing things like this. We don’t implement anything like this in the server proper.
That’s what plugins are for
The only external interfaces in the server now are NMEA and that’s because they came before the plugin architecture.
plugin pluses:
- separately updatable, without a server release
- declarative configuration UI or you can build a custom one
- installed on demand instead of everything for everybody - what you don't use is not using install bandwidth, diskspace, UI real estate or mental load ("what is this mqtt/prometheus/noforeignland/UDP/influx/yourtechhere and why should i care?") - people do ask for excluding some of the default plugins, and for some their disk space usage is nontrivial for embedded systems
minuses
- default config ui is clunky
- no backpressure mechanism
- less (documented) access to server internals
like i said some parts of the server would be better as plugins, to facilitate on demand install and individual updates
I brought this up a while back. Specifically including observability into Signalk to monitor performance. Prometheus is just a backend for metrics data. There are several other backend systems that can do the same thing. I’m curious what metrics @Andrea is looking for and why specifically using Prometheus over Influx or another data source. For performance metrics or actually Signalk deltas?
It’s not about using the Prometheus backend itself. The key point is having the ability to expose metrics in the modern Prometheus-compatible format. I noticed that you already provide metrics as JSON through the API at the endpoint
signalk/v1/api/vessels/self. Why not also add an endpoint that serves data in Prometheus format? This way, the metrics could be easily consumed by InfluxDB (via the Telegraf agent), Prometheus (using its native pull model), VictoriaMetrics (via Telegraf), and other monitoring systems.
I’m running two instances of VictoriaMetrics on two different databases: one on the boat for local access, and another at home via a site-to-site connection for redundancy. I collect all metrics from the boat to act as a “black box” in case something happens, to help debug physical problems, and to maintain a backlog of sailing performance data. With VictoriaMetrics, I can then create Grafana dashboards to visualize and analyze everything effectively.
Btw last night I developed a very simple script that converts, the JSON to Prometheus like metrics: https://github.com/AndreaF17/signalk-metrics-exporterIf you think to use 3rd party apps like Telegraf (I use that personally), then you could just read SK JSON in, parse need parts and use output plugin to make compatible to P.
https://github.com/influxdata/telegraf/tree/master/plugins/outputs/prometheus_client
Zero coding, just configurations.
If you want to get data to InfluxdB, then there are plugins available already, for both 1.x and 2.x and no need to use any middle app.
Very specific data formats might be nice, but use cases are probably very limited and audience too.
fwiw i think a plugin to add support for ”sk over prometheus” is a good idea - not unlike ”sk over mqtt”
Hello, I’m’running the signalk-prometheus-exporter plugin for months, and it works pretty well. I submitted a PR 10 months ago that the developer has accepted it. But unfortunately, he doesn’t published the new version to NPM.
I think it would be a better idea to fork this plugin, and maintain it, instead of creating a new one. I can help if needed.
In my use case, I collect all data through VictoriaMetrics VMagent, and store them to a VictoriaMetrics DB in my home. The big advantage of vmagent is that it can have local cache, that keep data even for long time offline period. Moreover, it can run on arm32, for example on a CerboGX, which is hard to do with Telegraf.