A few questions about metrics
Hi there!
A while ago we set up metrics collection using Prometheus and visualization using Grafana.
We set the prometheus level to "full", and we've only just noticed - using a network monitoring tool - that some of our machines (the ones running ispconfig, and configured to read each and every log of all the websites) are basically constantly sending 5-10Mbps of traffic to our Prometheus server. Because we run well over a hundred machines, this means that the gigabit connection on our monitoring server gets overwhelmed.
For example, on a machine that's been running for a while, the metrics are ~300MB.
We filter some of these metrics out on the prometheus side (e.g. all the go_ metrics, and also the whitelist hits).
As far as I could tell, there is currently no way for us to filter these on the client side.
Currently, we are also experimenting with just using the
Using aggregated metrics, on another long-running server, the metrics size, and thus the network traffic are looking much better:
But, this too, has some data that we don't really need.
Is there any possibility that in the future there would be functionality to control exactly what data gets pushed to metrics, so that we don't need to filter it on the Prometheus side, after already pushing all the data over the network?
Sorry if my explanation is a bit fumbled.
A while ago we set up metrics collection using Prometheus and visualization using Grafana.
We set the prometheus level to "full", and we've only just noticed - using a network monitoring tool - that some of our machines (the ones running ispconfig, and configured to read each and every log of all the websites) are basically constantly sending 5-10Mbps of traffic to our Prometheus server. Because we run well over a hundred machines, this means that the gigabit connection on our monitoring server gets overwhelmed.
For example, on a machine that's been running for a while, the metrics are ~300MB.
We filter some of these metrics out on the prometheus side (e.g. all the go_ metrics, and also the whitelist hits).
As far as I could tell, there is currently no way for us to filter these on the client side.
Currently, we are also experimenting with just using the
aggregatedUsing aggregated metrics, on another long-running server, the metrics size, and thus the network traffic are looking much better:
But, this too, has some data that we don't really need.
Is there any possibility that in the future there would be functionality to control exactly what data gets pushed to metrics, so that we don't need to filter it on the Prometheus side, after already pushing all the data over the network?
Sorry if my explanation is a bit fumbled.
