There are still some dependencies that we need to make work to fully
deactivate the legacyImports in the future. I'll start opening PRs
against those other repositories.
This could have been achieved either by switching to stringData, or doing
`std.base64(std.encodeUTF8($._config.alertmanager.config))` as per
google/jsonnet#575
I went with the former, because it's:
1. Easier to read existing config
2. Consistent with the way jsonnet object-based config is written just above
Expose the Thanos Sidecar's HTTP metrics and gRPC StoreAPI interfaces via a dedicated service. ClusterIP is set to none to allow for full discovery of all endpoints behind the service via in-cluster DNS. This allows for a new ServiceMonitor to then scrape the metrics available on the Sidecar's HTTP port.
A new service has been used so as to separate out the metrics that Prometheus makes available, and the metrics the Thanos Sidecar makes available. The Thanos Mixins from thanos-io/thanos default to a job label of 'thanos-sidecar', and hence the service here has had this label applied.
kube-apiserver has a histogram etcd_request_duration_seconds that
measures latency between the kube-apiserver and etcd instance.
This metrics is currently dropped by cluster-prometheus. Enable
this metrics so we have visibility into etcd latency.
We ensured that this does not enable other unwanted metrcis
count by(name) ({name=~"etcd_request.+"})
etcd_request_duration_seconds_bucket
etcd_request_duration_seconds_count
etcd_request_duration_seconds_sum
Previously the alert would fire when the number of Alertmanager pods
didn't match the number of replicas defined in the Alertmanager spec
even though all the running pods had the same configuration hash. This
type of issue is already covered by KubeStatefulSetUpdateNotRolledOut
(and possibly KubePodNotReady), having AlertmanagerConfigInconsistent
also active in this situation creates unnecessary noise.
With this change, the alert expression only returns when Alertmanager
pods have different configuration hash values irrespective of the number
of pod replicas. The message annotation has also been enhanced to report
the configuration hash for each pod.
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
Ignore kubelet pod filesystem mounts of the form:
/var/lib/kubelet/pods/1b260ce7-e75d-44d4-8409-922d2bd0851f/volumes...
Metrics for these volumes are available via the kubelet_volume_stats*
metrics.
