Commit Graph

1368 Commits

Author SHA1 Message Date
Alan Protasio
25aee26a57
Improving "Sparse postings" intersection (#13971)
Lets take the given example:

P1: [2, 5, 9, 18, 21]
P2: [3, 7, 14, 19, 21]
P3: [1, 21]

Currently, we would only advance through P1 and P2 until discovering
an intersection and then checking P3. In essence, the traversal order
was: 2, 3, 5, 7, 9, 14, 18, 19, 21 (intersection found).

With the proposed change, P3 is also examined even if P1 and P2
haven't found an intersection yet. This adjustment allows for the
possibility of skipping some iterations.

Post-change, the traversal order becomes: 2, 3, 21 (3 iterations instead of 9).

Signed-off-by: alanprot <alanprot@gmail.com>
2025-08-05 12:22:54 +01:00
Julius Volz
2e709c6567
Merge pull request #16695 from sujalshah-bit/block_endpoint
api: Create `/status/tsdb/blocks` endpoint.
2025-07-31 18:15:49 +02:00
George Krajcsovits
3f59fe1a80
fix(chunkenc): appending histograms with empty buckets (#16893)
* test(chunkenc): appending histograms with empty buckets and gaps

Append such native histograms that have empty buckets and gaps
between the indexes of those buckets.

There is a special case for appending counter native histograms to a chunk in TSDB: if we append a histogram that is missing some buckets that are already in chunk, then usually that's a counter reset. However if the missing bucket is empty, meaning its value is 0, then we don't consider it missing.

For this case to trigger , we need to write empty buckets into the chunk. Normally native histograms are compacted when we emit them , so this is very rare and compact make sure that there are no multiple continuous empty buckets with gaps between them.

The code that I've added in #14513 did not take into account that you can bypass compact and write histograms with many empty buckets, with gaps between them. These are still valid, so the code has to account for them.

Main fix in the expandIntSpansAndBuckets and expandFloatSpansAndBuckets function. I've also refactored them for clarity. Consequently needed to fix insert and adjustForInserts to also allow gaps between inserts.

I've added some new test cases (data driven test would be nice here, too many cases). And removed the deprecated old function that didn't deal with empty buckets at all.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>
Co-authored-by: Björn Rabenstein <beorn@grafana.com>
2025-07-24 18:01:02 +02:00
machine424
9a0bbb60bc test(tsdb): disable TestDelayedCompaction/delayed_compaction_enabled on windows
as flaky because of Time imprecision

fixes https://github.com/prometheus/prometheus/issues/16450

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2025-07-22 15:30:05 +01:00
Charles Korn
46acc974c0
fix(remote): Unregister metrics emitted by remote.WriteStorage when closed (#16868)
* Unregister metrics emitted by `remote.WriteStorage` when closed

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Address PR feedback: add test

Signed-off-by: Charles Korn <charles.korn@grafana.com>

---------

Signed-off-by: Charles Korn <charles.korn@grafana.com>
2025-07-17 11:32:15 +02:00
socialsister
869c946370 chore: fix some minor issues in comments
Signed-off-by: socialsister <seekseat@qq.com>
2025-07-16 11:24:42 +01:00
Nicolás Pazos
b43a07248f tsdb tests: fix mockIndex implementation
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2025-07-10 15:59:38 -03:00
machine424
846acc10bb chore(tsdb): remove NewLeveledCompactorWithChunkSize constructor as unused, library users ca can redefine it on their side
Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2025-07-09 17:10:13 +01:00
George Krajcsovits
1d79f0f47e
chore(tsdb): add a few more testcases for unlock of unlocked mtx 16332 (#16848)
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2025-07-09 16:24:46 +02:00
Banana Duck
89f011ba13
fix: unlock of unlocked mutex (#16332)
* fix: unlock on unlocked mutex

Signed-off-by: Usama Alhanaqtah <a.usama@yandex.ru>

* test coverage

Signed-off-by: Usama Alhanaqtah <a.usama@yandex.ru>

---------

Signed-off-by: Usama Alhanaqtah <a.usama@yandex.ru>
Co-authored-by: alhanaqtah.usama <alhanaqtah.usama@DEV-254.local>
2025-07-09 15:37:55 +02:00
liangmulu
b1a7df2c0c chore: fix some minor issues in comments
Signed-off-by: liangmulu <liangmulu@outlook.com>
2025-07-09 18:05:41 +08:00
Ahmed Hassan
01be7bfb2e add NumFloatSamples to TSDB block stats
Signed-off-by: Ahmed Hassan <afayekhassan@gmail.com>
2025-07-07 13:48:18 -07:00
sujal shah
4408a6bcaf api: Create /status/tsdb/blocks endpoint.
this endpoint serves blocks data to the client.

Signed-off-by: sujal shah <sujalshah28092004@gmail.com>
2025-07-04 03:13:54 +05:30
Ahmed Hassan
6d77b47d13 add numHistogramSamples to block stats
Signed-off-by: Ahmed Hassan <afayekhassan@gmail.com>
2025-07-02 19:52:04 -07:00
machine424
5ac1e6a656
fix(test): fall back to default direct I/O requirements in tests when statx isn't supported by using a higher lever util
Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2025-05-29 09:35:33 +02:00
Charles Korn
ab1b1db128
tsdb: fix issue where a new segment file is created for every chunk if WithSegmentSize not called (#16635)
* tsdb: fix issue where a new segment file is created for every chunk

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Address PR feedback

Signed-off-by: Charles Korn <charles.korn@grafana.com>

---------

Signed-off-by: Charles Korn <charles.korn@grafana.com>
2025-05-28 18:21:59 +02:00
Ayoub Mrini
317acb3d68
refactor: use the built-in max/min to simplify the code (#16617)
Signed-off-by: carrychair <linghuchong404@gmail.com>
2025-05-27 14:42:50 +02:00
Ayoub Mrini
2edc3ed6c5
feat(tsdb): introduce --use-uncached-io feature flag and allow using it for chunks writing (#15365)
Signed-off-by: machine424 <ayoubmrini424@gmail.com>
Signed-off-by: Ayoub Mrini <ayoubmrini424@gmail.com>
2025-05-21 14:42:30 +02:00
carrychair
e83dc66bdb refactor: use the built-in max/min to simplify the code
Signed-off-by: carrychair <linghuchong404@gmail.com>
2025-05-20 14:36:39 +08:00
György Krajcsovits
772d5ab433
Merge branch 'main' into krajo/intern-custom-values 2025-05-20 08:23:15 +02:00
hardlydearly
ba4b058b7a refactor: use slices.Contains to simplify code
Signed-off-by: hardlydearly <799511800@qq.com>
2025-05-09 08:27:10 +02:00
György Krajcsovits
6c646657d5
perf(chunkenc): intern the custom values for native histograms
The custom values are the "le" bucket boundaries of native histograms
with custom buckets. They are never modified. It is ok to not copy them
when iterating a chunk, just reference them.

If we will ever have a function that modifies the custom values, like
'trim' for example. That function will have to make a copy on write.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2025-05-07 14:40:45 +02:00
Andre Branchizio
b07b552139
[PERF] TSDB: Pass down label value limit into implementation (#16158)
* allow limiting label values calls

Signed-off-by: Andre Branchizio <andrejbranch@gmail.com>
2025-05-06 18:54:48 +01:00
Dimitar Dimitrov
7e49b91d9a
tsdb/errors.MultiError: support errors.As (#16544)
* tsdb/errors.MultiError: implement Unwrap

the multierror was hiding some errors in Mimir. I also added unit tests because I had them handy from a similar change I and yuri did in XXX and some time ago

---------

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2025-05-06 13:45:16 +00:00
Arve Knudsen
e7e3ab2824
Fix linting issues found by golangci-lint v2.0.2 (#16368)
* Fix linting issues found by golangci-lint v2.0.2

---------

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2025-05-03 19:05:13 +02:00
Bryan Boreham
ca416c580c
Merge branch 'main' into slicelabels
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2025-05-02 10:31:57 +01:00
Yuchen Wang
5630a3906a
fix typo (#16480)
Signed-off-by: Yuchen Wang <yuchen.wang@databricks.com>
2025-04-25 09:27:58 +02:00
Bryan Boreham
8487ed8145
Merge pull request #16440 from bboreham/faster-benchmark-loadwls
[TESTS] TSDB: Faster WAL benchmarks
2025-04-22 15:59:03 +01:00
Bryan Boreham
a11772234d
Merge pull request #16333 from colega/fix-series-create-gc-race
fix: race condition between series creation and garbage collection
2025-04-17 12:15:11 +01:00
machine424
a825d448da feat(tsdb/(head|agent)): dereference the pools at the end of the WL replay to
not wait for an extra GC cycle until the built-in cleanup mechanism
kicks in

See https://github.com/prometheus/prometheus/pull/15778

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2025-04-17 13:06:08 +02:00
Bryan Boreham
1d4b1d76a5 [TESTS] More efficient label creation in BenchmarkLoadWLs
Use the Builder abstraction instead of going via a map.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2025-04-16 18:02:47 +01:00
Bryan Boreham
848df13d3a [TESTS] Faster WAL Benchmarks by reusing buffer
Less garbage collection.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2025-04-16 17:58:09 +01:00
Lukasz Mierzwa
bb76966992 Use stringlabels by default
This removes the stringlabels build tag, makes that implementation the default one, and moves the old labels implementation under the slicelabels build tag.
Fixes #16064.

Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com>
2025-04-15 17:52:24 +01:00
Alex Le
bce72b93d9
tsdb: Introduced new constructor for LeveledCompactor to take in metrics (#16408)
* Introduced new constructor for LeveledCompactor to take in metrics

Signed-off-by: Alex Le <leqiyue@amazon.com>

* Added Metrics to LeveledCompactorOptions

Signed-off-by: Alex Le <leqiyue@amazon.com>

---------

Signed-off-by: Alex Le <leqiyue@amazon.com>
2025-04-11 09:17:45 +01:00
Alex Le
701d13abf9
Make sure LeveledCompactor respect context cancellation during the time opening blocks (#16407)
Signed-off-by: Alex Le <leqiyue@amazon.com>
2025-04-08 09:04:23 +01:00
Oleg Zaytsev
f5f91a9ca4
defer a.unmarkCreatedSeriesAsPendingCommit()
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2025-03-31 10:06:28 +02:00
Ben Kochie
a721daf981
Log WAL segment loading time (#16336)
Improve readability of "WAL segment loaded" by logging the duration
of each load. This helps make it easier to spot slow WAL file load
times.

Signed-off-by: SuperQ <superq@gmail.com>
2025-03-31 06:05:14 +02:00
Ryan Wu
7d73c1d3f8
refactor[discovery, tsdb]: simplify error handling and remove redundant checks (#16328)
* refactor: simplify error handling and remove redundant checks

Signed-off-by: Ryan Wu <rongjun0821@gmail.com>

* Add the comment for return of reloading blocks failure

Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com>
Signed-off-by: Ryan Wu <rongjun0821@gmail.com>

* Add the comment for return of reloading blocks failure

Signed-off-by: Ryan Wu <rongjun0821@gmail.com>

---------

Signed-off-by: Ryan Wu <rongjun0821@gmail.com>
Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com>
2025-03-27 12:20:59 +01:00
Oleg Zaytsev
e4fe8d8684
Create memSeries with pendingCommit=true
This fixes TestHead_RaceBetweenSeriesCreationAndGC.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2025-03-27 11:11:57 +01:00
Oleg Zaytsev
df33f1aace Add TestHead_RaceBetweenSeriesCreationAndGC
This test consistently fails missing ~10 series.
If it doesn't fail on your machine, just increase totalSeries, that's
how race conditions work.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2025-03-27 10:56:24 +01:00
Matthieu MOREL
5fa1146e21
chore: enable gci linter (#16245)
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2025-03-22 15:46:13 +00:00
Patryk Prus
2f43a5a3ab
TSDB: don't process exemplars older than minValidTime during WAL replay
Signed-off-by: Patryk Prus <p@trykpr.us>
2025-03-19 16:24:08 -04:00
Ganesh Vernekar
bc595263c1
Merge pull request #16231 from pr00se/multiref-improvements
TSDB: Handle metadata/tombstones/exemplars for duplicate series during WAL replay
2025-03-19 16:15:50 -04:00
pudongair
308c8c48c1
chore: fix some comments (#16237)
Signed-off-by: pudongair <744355276@qq.com>
2025-03-19 16:28:34 +01:00
Ziqi Zhao
f6903bcc22
Let HistogramAppender.appendable return CounterResetHeader instead of… (#16195)
Let HistogramAppender.appendable return CounterResetHeader instead of boolean

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
Signed-off-by: Björn Rabenstein <github@rabenste.in>

---------

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
Signed-off-by: Björn Rabenstein <github@rabenste.in>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
2025-03-18 17:40:27 +01:00
Patryk Prus
e4e1b515bc
TSDB: Handle metadata/tombstones/exemplars for duplicate series during WAL replay
Signed-off-by: Patryk Prus <p@trykpr.us>
2025-03-18 12:22:33 -04:00
Fiona Liao
37c2ebb5fd
Make out-of-order native histograms flag a no-op and always enable (#16207)
* Remove experimental out-of-order native histogram flag

This feature has been available in Prometheus since September 2024,
and has no known issues. Therefore proposing to remove the flag
entirely and always have it on. Note that there are still two
settings that need to be configured (out-of-order time window > 0
and native histograms enabled) for this feature to work.

Signed-off-by: Fiona Liao <fiona.liao@grafana.com>

* Update CHANGELOG

Signed-off-by: Fiona Liao <fiona.liao@grafana.com>

* Keep feature flag with warning

Signed-off-by: Fiona Liao <fiona.liao@grafana.com>

* Update CHANGELOG

Signed-off-by: Fiona Liao <fiona.liao@grafana.com>

* Update tsdb/head_append.go

Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>

* Update CHANGELOG.md

Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>

* Update tsdb/head_append.go

Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>

* Additional cleanup of comments and test names

Signed-off-by: Fiona Liao <fiona.liao@grafana.com>

---------

Signed-off-by: Fiona Liao <fiona.liao@grafana.com>
Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
2025-03-18 10:59:02 +00:00
Patryk Prus
86eeaf1886
Skip writing series records uniformly across the benchmark, so we skip some OOO series as well
Signed-off-by: Patryk Prus <p@trykpr.us>
2025-03-17 15:17:53 -04:00
Patryk Prus
2147538d1e
Add missing series refs to benchmark
Signed-off-by: Patryk Prus <p@trykpr.us>
2025-03-17 15:17:53 -04:00
Patryk Prus
401dbacf2e
Add counters for unknown series references during WAL/WBL replay
Signed-off-by: Patryk Prus <p@trykpr.us>
2025-03-17 15:17:53 -04:00