184c7eb918
Because of relabelling, an endpoint can only select a subset of series that go through WriteStorage Having a highestTimestamp at WriteStorage level yields wrong values if the corresponding sample won't even make it to a remote queue. Currently PrometheusRemoteWriteBehind is based on that, and would fire if an endpoint is only interested in a subset of series that take time to appear. A "prometheus_remote_storage_queue_highest_timestamp_seconds" that only takes into account samples in the queue is introduced, and used in PrometheusRemoteWriteBehind and dashboards in documentation/prometheus-mixin Same applies to samplesIn/dataIn, QueueManager should know more about when to update those; when data is enqueued. That makes dataDropped unnecessary, thus help simplify the logic in QueueManager.calculateDesiredShards() Signed-off-by: machine424 <ayoubmrini424@gmail.com>
Prometheus Mixin
This is work in progress. We aim for it to become a good role model for alerts and dashboards eventually, but it is not quite there yet.
The Prometheus Mixin is a set of configurable, reusable, and extensible alerts and dashboards for Prometheus.
To use them, you need to have jsonnet (v0.13+) and jb installed. If you
have a working Go development environment, it's easiest to run the following:
$ go install github.com/google/go-jsonnet/cmd/jsonnet@latest
$ go install github.com/google/go-jsonnet/cmd/jsonnetfmt@latest
$ go install github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest
Note: The make targets lint and fmt need the jsonnetfmt binary, which is
available from v.0.16.0 in the Go implementation of jsonnet. If your jsonnet version is older than 0.16.0 you have to either upgrade or install the C++ version of
jsonnetfmt if you want to use make lint
or make fmt.
Next, install the dependencies by running the following command in this directory:
$ jb install
You can then build a prometheus_alerts.yaml with the alerts and a directory
dashboards_out with the Grafana dashboard JSON files:
$ make prometheus_alerts.yaml
$ make dashboards_out
For more advanced uses of mixins, see https://github.com/monitoring-mixins/docs.