12389 Commits

Author SHA1 Message Date
Bryan Boreham
dcd024a095 storage/remote: speed up StoreSeries by re-using labels.Builder
Relabeling can take a pre-populated `Builder` instead of making a new
one every time. This is much more efficient.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-01-29 18:49:55 +00:00
Bryan Boreham
d9483bb77c storage/remote: add BenchmarkStoreSeries
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-01-29 16:54:12 +00:00
Marco Pracucci
501bc6419e
Add ShardedPostings() support to TSDB (#10421)
This PR is a reference implementation of the proposal described in #10420.

In addition to what described in #10420, in this PR I've introduced labels.StableHash(). The idea is to offer an hashing function which doesn't change over time, and that's used by query sharding in order to get a stable behaviour over time. The implementation of labels.StableHash() is the hashing function used by Prometheus before stringlabels, and what's used by Grafana Mimir for query sharding (because built before stringlabels was a thing).

Follow up work
As mentioned in #10420, if this PR is accepted I'm also open to upload another foundamental piece used by Grafana Mimir query sharding to accelerate the query execution: an optional, configurable and fast in-memory cache for the series hashes.

Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-29 11:57:27 +00:00
Leegin
19efd0a675
api: Serialize discovered labels into JSON directly in dropped targets (#13484)
Converted maps into labels.Labels to avoid a lot of copying of data which leads to very high memory consumption while opening the /service-discovery endpoint in the Prometheus UI

Signed-off-by: Leegin <114397475+Leegin-darknight@users.noreply.github.com>
2024-01-29 10:20:20 +00:00
Leegin
e3040bfabc
api: Serialize discovered and target labels into JSON directly (#13469)
Converted maps into labels.Labels to avoid a lot of copying of data which leads to very high memory consumption while opening the /service-discovery endpoint in the Prometheus UI

Signed-off-by: Leegin <114397475+Leegin-darknight@users.noreply.github.com>
2024-01-29 10:19:02 +00:00
Marco Pracucci
cbbbd6e70a
Remove superfluous nil check in Group.metrics
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-29 10:21:57 +01:00
Marco Pracucci
046cd7599f
Introduced sequentialRuleEvalController
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-29 10:19:18 +01:00
Marco Pracucci
23f89c18b2
Improved RuleConcurrencyController interface doc
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-29 10:18:29 +01:00
Marco Pracucci
2764c46531
Added more test cases to TestDependenciesEdgeCases
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-29 10:18:03 +01:00
Marco Pracucci
52bc568d04
Add more test cases to TestDependenciesEdgeCases
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-29 10:17:13 +01:00
Marco Pracucci
21a03dc018
Simplify the design to update concurrency controller once the rule evaluation has done
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-29 10:16:31 +01:00
Marco Pracucci
1bb341fa51
Improve doc
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-29 10:15:50 +01:00
Marco Pracucci
6bbb03bd00
Fixed auto-generated doc
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-29 10:12:37 +01:00
Marco Pracucci
ac1c6eb3ef
Fix typo in CLI flag description
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-29 10:11:50 +01:00
Danny Kopping
7aa3b10c3f
Block until all rules, both sync & async, have completed evaluating
Updated & added tests
Review feedback nits
Return empty map if not indeterminate
Use highWatermark to track inflight requests counter
Appease the linter
Clarify feature flag

Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
2024-01-29 10:08:41 +01:00
Danny Kopping
f922534c4d
Refactoring for performance, and to allow controller to be overridden
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
2024-01-29 10:08:41 +01:00
Danny Kopping
5bda33375a
Rename flag
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
2024-01-29 10:08:41 +01:00
Danny Kopping
94cdfa30cd
Refactoring
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
2024-01-29 10:08:41 +01:00
Danny Kopping
0dc7036db3
Optimising dependencies/dependents funcs to not produce new slices each request
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
2024-01-29 10:08:41 +01:00
Danny Kopping
e7758d187e
Refactor concurrency control
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
2024-01-29 10:08:39 +01:00
Danny Kopping
ed2933ca60
Add feature flag
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
2024-01-29 10:08:07 +01:00
Danny Kopping
940f83a540
Implementation
NOTE:
Rebased from main after refactor in #13014

Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
2024-01-29 10:07:15 +01:00
Björn Rabenstein
bdc3cfdd5d
Merge pull request #13477 from prometheus/release-2.50
Merge release-2.50 branch back into main
2024-01-27 01:10:57 +01:00
Björn Rabenstein
88d07c43cb
Merge pull request #13474 from fpetkovski/hist-last-over-time
Fix last_over_time for native histograms
2024-01-26 23:28:58 +01:00
Bartlomiej Plotka
5b35381e21
Merge pull request #13466 from prometheus/beorn7/maintainers
Name @beorn7 and @krajorama as maintainers for native histograms
2024-01-26 16:52:15 +01:00
Julien Pivotto
960b6266e2
Merge pull request #13447 from ptodev/update-puppetdb-metric-label
Change metric label for Puppet SD from 'http' to 'puppetdb'
2024-01-26 15:40:15 +01:00
Filip Petkovski
a577a0a542
Fix last_over_time for native histograms
The last_over_time retains a histogram sample without making a copy.
This sample is now coming from the buffered iterator used for windowing functions,
and can be reused for reading subsequent samples as the iterator progresses.

I would propose copying the sample in the last_over_time function, similar to
how it is done for rate, sum_over_time and others.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2024-01-26 15:02:40 +01:00
Goutham Veeramachaneni
fb552d290d
Merge pull request #13464 from aknuds1/arve/fix-update-copy
otlptranslator/update-copy.sh: Fix sed command lines
2024-01-26 17:52:49 +05:30
Bryan Boreham
66237c1996 tsdb: use cheaper Mutex on series
Mutex is 8 bytes; RWMutex is 24 bytes and much more complicated. Since
`RLock` is only used in two places, `UpdateMetadata` and `Delete`,
neither of which are hotspots, we should use the cheaper one.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-01-26 11:01:39 +00:00
Yury Molodov
a598ddfbc4
ui/fix: correct url handling for stacked graphs (#13460)
Signed-off-by: Yury Moladau <yurymolodov@gmail.com>
2024-01-26 09:25:23 +01:00
Björn Rabenstein
0fe34f8f8d
Merge pull request #13449 from fatsheep9146/native-histogram-support-exemplar
scrape: Prometheus server support exemplar in native histogram
2024-01-25 21:34:17 +01:00
beorn7
fe35f2dac3 Name @beorn7 and @krajorama as maintainers for native histograms
I have been the de-facto maintainer for native histograms from the
beginning. So let's put this into MAINTAINERS.md.

In addition, I hereby proposose George Krajcsovits AKA Krajo as a
co-maintainer. He has contributed a lot of native histogram code, but
more importantly, he has contributed substantially to reviewing other
contributors' native histogram code, up to a point where I was merely
rubberstamping the PRs he had already reviewed. I'm confident that he
is ready to to be granted commit rights as outlined in the
"Maintainers" section of the governance:
https://prometheus.io/governance/#maintainers

According to the same section of the governance, I will announce the
proposed change on the developers mailing list and will give some time
for lazy consensus before merging this PR.

Signed-off-by: beorn7 <beorn@grafana.com>
2024-01-25 15:25:04 +01:00
Arve Knudsen
de28494434 Make update-copy.sh work for both OSX and GNU sed
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-01-25 15:11:03 +01:00
Ben Kochie
c3b8ef1694
Rollback k8s.io requirements (#13462)
Rollback k8s.io Go modules to v0.28.6 to avoid forcing upgrade of Go to
1.21. This allows us to keep compatibility with the currently supported
upstream Go releases.

Signed-off-by: SuperQ <superq@gmail.com>
2024-01-25 14:18:47 +01:00
Arve Knudsen
660df3488d otlptranslator/update-copy.sh: Fix sed command lines
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-01-25 13:56:04 +01:00
Augustin Husson
599ad93245
Merge pull request #13463 from prometheus/nexucis/merge-back
Merging back release v2.49.1 to main
2024-01-25 13:47:54 +01:00
Augustin Husson
52fa5863a2
Merge branch 'main' into nexucis/merge-back 2024-01-25 13:31:16 +01:00
Bryan Boreham
b9eab6e4b8
tsdb: simplify internal series delete function (#13261)
Lifting an optimisation from Agent code, `seriesHashmap.del` can use
the unique series reference, doesn't need to check Labels.
Also streamline the logic for deleting from `unique` and `conflicts` maps,
and add some comments to help the next person.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-01-25 11:57:54 +01:00
Bryan Boreham
74b73d1e2c
Labels: Add DropMetricName function, used in PromQL (#13446)
This function is called very frequently when executing PromQL functions,
and we can do it much more efficiently inside Labels.

In the common case that `__name__` comes first in the labels, we simply
re-point to start at the next label, which is nearly free.

`DropMetricName` is now so cheap I removed the cache - benchmarks show
everything still goes faster.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-01-25 11:48:49 +01:00
Bryan Boreham
3f30ad3cc2
Merge pull request #13015 from bboreham/smaller-txring
tsdb: make transaction isolation data structures smaller
2024-01-25 10:48:15 +00:00
dependabot[bot]
1069c1f495
build(deps): bump bufbuild/buf-push-action (#13357)
Bumps [bufbuild/buf-push-action](https://github.com/bufbuild/buf-push-action) from 342fc4cdcf29115a01cf12a2c6dd6aac68dc51e1 to a654ff18effe4641ebea4a4ce242c49800728459.
- [Release notes](https://github.com/bufbuild/buf-push-action/releases)
- [Commits](342fc4cdcf...a654ff18ef)

---
updated-dependencies:
- dependency-name: bufbuild/buf-push-action
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-25 10:24:37 +01:00
dependabot[bot]
97c186e2c8
build(deps): bump actions/upload-artifact from 3.1.3 to 4.0.0 (#13355)
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3.1.3 to 4.0.0.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](a8a3f3ad30...c7d193f32e)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-25 10:24:14 +01:00
dependabot[bot]
7f837694d9
build(deps): bump the go-opentelemetry-io group with 1 update (#13453)
Bumps the go-opentelemetry-io group with 1 update: [go.opentelemetry.io/collector/semconv](https://github.com/open-telemetry/opentelemetry-collector).


Updates `go.opentelemetry.io/collector/semconv` from 0.92.0 to 0.93.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-collector/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-collector/blob/main/CHANGELOG-API.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-collector/compare/v0.92.0...v0.93.0)

---
updated-dependencies:
- dependency-name: go.opentelemetry.io/collector/semconv
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-opentelemetry-io
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-25 10:20:35 +01:00
dependabot[bot]
94ced726e2
build(deps): bump the k8s-io group with 2 updates (#13454)
Bumps the k8s-io group with 2 updates: [k8s.io/api](https://github.com/kubernetes/api) and [k8s.io/client-go](https://github.com/kubernetes/client-go).


Updates `k8s.io/api` from 0.28.4 to 0.29.1
- [Commits](https://github.com/kubernetes/api/compare/v0.28.4...v0.29.1)

Updates `k8s.io/client-go` from 0.28.4 to 0.29.1
- [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md)
- [Commits](https://github.com/kubernetes/client-go/compare/v0.28.4...v0.29.1)

---
updated-dependencies:
- dependency-name: k8s.io/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: k8s-io
- dependency-name: k8s.io/client-go
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: k8s-io
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-25 10:20:14 +01:00
Augustin Husson
9a9e73b34a
update all go dependencies (#13438)
Signed-off-by: Augustin Husson <husson.augustin@gmail.com>
2024-01-24 16:53:32 +01:00
Arve Knudsen
ba7012ec6a
TestHeadLabelValuesWithMatchers: Add test case (#13414)
Add test case to TestHeadLabelValuesWithMatchers, while fixing a couple
of typos in other test cases. Also enclosing some implicit sub-tests in a
`t.Run` call to make them explicitly sub-tests.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-01-24 10:47:56 +01:00
Ziqi Zhao
033c9b229d mirror metrics.proto change & generate code
Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
2024-01-24 14:01:23 +08:00
Paulin Todev
53589fde4f
Change metric label for Puppetdb from 'http' to 'puppetdb'
Signed-off-by: Paulin Todev <paulin.todev@gmail.com>
2024-01-23 19:24:13 +00:00
Björn Rabenstein
89523cf9b3
doc: Mark mad_over_time as experimental (#13440)
We forgot to do that in
https://github.com/prometheus/prometheus/pull/13059

Signed-off-by: beorn7 <beorn@grafana.com>
2024-01-23 17:05:34 +01:00
Filip Petkovski
583f3e587c
Optimize histogram iterators (#13340)
Optimize histogram iterators

Histogram iterators allocate new objects in the AtHistogram and
AtFloatHistogram methods, which makes calculating rates over long
ranges expensive.

In #13215 we allowed an existing object to be reused
when converting an integer histogram to a float histogram. This commit follows
the same idea and allows injecting an existing object in the AtHistogram and
AtFloatHistogram methods. When the injected value is nil, iterators allocate
new histograms, otherwise they populate and return the injected object.

The commit also adds a CopyTo method to Histogram and FloatHistogram which
is used in the BufferedIterator to overwrite items in the ring instead of making
new copies.

Note that a specialized HPoint pool is needed for all of this to work 
(`matrixSelectorHPool`).

---------

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
2024-01-23 17:02:14 +01:00