1522 Commits

Author SHA1 Message Date
Julien
9d38077e50
Merge pull request #18080 from ldufr/ldufresne/retention-size-percentage
Add percentage based retention
2026-02-24 15:50:36 +01:00
Laurent Dufresne
c76e78d0a4 Added test for percentage-based retention
Signed-off-by: Laurent Dufresne <laurent.dufresne@grafana.com>
2026-02-24 15:28:45 +01:00
Laurent Dufresne
971143edac Added Retention.Percentage to config file with runtime config reloading
Signed-off-by: Laurent Dufresne <laurent.dufresne@grafana.com>
2026-02-24 15:28:20 +01:00
Jérôme LOYET
696679e50c Add storage.tsdb.retention.percentage config
Signed-off-by: Jérôme LOYET <822436+fatpat@users.noreply.github.com>
Signed-off-by: Laurent Dufresne <laurent.dufresne@grafana.com>
2026-02-24 15:27:45 +01:00
Atıl Sensalduz
7a9c057727
Merge pull request #18180 from atilsensalduz/bugfix/tsdb-writeMetaFile-fd-leak
fix(tsdb): close file descriptor on json.MarshalIndent failure in writeMetaFile
2026-02-24 12:25:00 +00:00
harsh kumar
f312fde4a2
test: Enable parallel execution for chunk write queue tests (#17338)
* test(tsdb): Enable parallel execution for chunk write queue tests

Signed-off-by: Harsh <harshmastic@gmail.com>
2026-02-24 12:37:20 +01:00
Matthieu MOREL
addc3dcb47 chore: enable staticcheck linter and update golangci-lint to 2.10.1
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2026-02-18 15:58:16 +00:00
Julien Pivotto
7d0a39ac93 chore(lint): enable wg.Go
Since our minimum supported go version is now go 1.25, we can use wg.Go.

Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
2026-02-17 15:21:51 +01:00
Bartlomiej Plotka
e4c997c126
Merge pull request #18085 from aknuds1/arve/fix-test-block-ranges
tsdb: fix flaky TestBlockRanges by using explicit compaction
2026-02-17 09:08:38 +00:00
Kyle Eckhart
ae062151cd
tsdb/wlog: Remove any temproary checkpoints when creating a Checkpoint (#17598)
* RemoveTmpDirs function to tsdbutil
* Refactor db to use RemoveTmpDirs and no longer cleanup checkpoint tmp dirs
* Use RemoveTmpDirs in wlog checkpoint to cleanup all checkpoint tmp folders
* Add tests for RemoveTmpDirs
* Ensure db.Open will still cleanup extra temporary checkpoints

Signed-off-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>
2026-02-17 09:23:54 +01:00
Bryan Boreham
b908cc48a2
Merge pull request #18069 from mishraa-G/optimize-label-api
tsdb:Optimize LabelValues API performance (#14551)
2026-02-16 21:51:40 +00:00
Arve Knudsen
b0718d5c93 tsdb: fix flaky TestBlockRanges by using explicit compaction
Replace polling loops (for range 100 { time.Sleep }) with explicit
db.Compact() calls after disabling background compaction, eliminating
CI flakiness on slow machines. Also fix incorrect overlap assertions
that were checking the wrong direction (LessOrEqual -> GreaterOrEqual).

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2026-02-15 11:44:23 +01:00
Divyansh Mishra
fcb68060cb tsdb: Optimize LabelValues for sparse intersections (Fixes #14551)
Signed-off-by: Divyansh Mishra <divyanshmishra@Divyanshs-MacBook-Air-3.local>
2026-02-14 15:18:54 +05:30
Patryk Prus
d8c24c6bde
tsdb/index: export sentinel error for symbol table size exceeded
Signed-off-by: Patryk Prus <p@trykpr.us>
2026-02-10 12:44:32 -05:00
Ganesh Vernekar
fe5cb190e6
tsdb: Add metrics for stale series compaction (#17957)
Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
2026-02-06 09:05:56 +00:00
Bartlomiej Plotka
c8e7f4e2a6
tests: Unify TestDiskFillingUpAfterDisablingOOO and avoid hiding errors (#18017)
* tests: Unify TestDiskFillingUpAfterDisablingOOO and avoid hiding errors

Signed-off-by: bwplotka <bwplotka@gmail.com>

* addressed comments

Signed-off-by: bwplotka <bwplotka@gmail.com>

---------

Signed-off-by: bwplotka <bwplotka@gmail.com>
2026-02-05 16:11:35 +00:00
Ben Kochie
8d491cc642
tsdb: Migrate multi-errors to errors package (#17768)
Modernize tsdb package by migrating multi-error handling
to the standard library errors package.
* Add a modernized CloseAll helper.

Signed-off-by: SuperQ <superq@gmail.com>
2026-02-04 10:41:57 +01:00
Bartlomiej Plotka
eefa6178fb
fix: fix rare race on empty head.initialized() vs head.initTime() (#17963)
* fix: fix rare race on empty head.initized() vs head.initTime()

Relates to https://github.com/prometheus/prometheus/issues/17941

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Apply suggestions from code review

Co-authored-by: Owen Williams <owen.williams@grafana.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* addressed comments

Signed-off-by: bwplotka <bwplotka@gmail.com>

---------

Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: Owen Williams <owen.williams@grafana.com>
2026-02-02 09:13:02 +00:00
Arve Knudsen
d9db76631d
tsdb: fix flaky TestWaitForPendingReadersInTimeRange tests (#17985)
The tests were flaky because they used hard-coded time.After(550ms)
waits, which had only 50ms margin over WaitForPendingReadersInTimeRange's
500ms poll interval. On slow CI runners, this margin wasn't reliable.

Use synctest for deterministic time control:
- Wrap test logic in synctest.Test() to use fake time
- Use synctest.Wait() to let goroutines reach dormant state
- Use time.Sleep() to advance fake time past the poll interval
- No more timing-dependent assertions

This makes the tests both reliable and ~60x faster (0.05s vs 3s).

Fixes both TestWaitForPendingReadersInTimeRange and
TestWaitForPendingReadersInTimeRange_AppenderV2.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2026-02-01 15:52:26 +00:00
Bartlomiej Plotka
88f6ee4c8e
tests(scrape): add TestScrapeLoopAppend_WithStorage (#17937)
Signed-off-by: bwplotka <bwplotka@gmail.com>
2026-01-30 11:44:07 +00:00
Bartlomiej Plotka
0fc70f3a62
tsdb: kill unused mint,maxt tracking (#17967)
Signed-off-by: bwplotka <bwplotka@gmail.com>
2026-01-29 14:26:40 +00:00
Arve Knudsen
00a7faa2e3
tsdb: fix division by zero in stale series compaction (#17952)
Guard the stale series ratio calculation by checking numSeries > 0
before computing the ratio. This prevents division by zero when
the head has no series.

Fixes #17949

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2026-01-29 08:06:00 +01:00
Bartlomiej Plotka
2597a12080
st: Add a hidden 'st-storage' feature flag for PROM-60 (#17907)
Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2026-01-28 09:05:54 +00:00
Björn Rabenstein
15030d6f19
Merge pull request #17738 from aknuds1/arve/leveled-compactor
enhancement(tsdb): add test for `LeveledCompactor.Plan` stopping after excluding block
2026-01-27 12:38:42 +01:00
Arve Knudsen
c8ff2d739b enhancement(tsdb): add test for LeveledCompactor stopping after excluding block
Add a test for `LeveledCompactor.Plan()` stopping after a block matches the
`BlockExcludeFilter`, as a sub-test
`TestLeveledCompactor/Plan/BlockExcludeFilter stops iteration`.

Also moving `TestLeveledCompactor_plan` to a sub-test
of `TestLeveledCompactor`, for consistency.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2026-01-24 16:07:22 +01:00
Ganesh Vernekar
3e4a094dbb Add stale_series_compaction_threshold config file option
Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
2026-01-23 18:12:34 -08:00
Ganesh Vernekar
4f3de8da29 tsdb: Add unit tests for stale series compaction
Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
2026-01-23 18:07:34 -08:00
Ganesh Vernekar
43dc23afe7 tsdb: Clear stale series from the Head during WAL replay
Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
2026-01-23 18:02:45 -08:00
Ganesh Vernekar
43e69388df tsdb: Add stale series compaction support in the DB
Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
2026-01-23 18:00:08 -08:00
Ganesh Vernekar
9b444b57af tsdb: Add StaleHead and GC for stale series in the Head block
Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
2026-01-23 17:59:41 -08:00
Bartlomiej Plotka
bec70227f1
feat(scrape)[PART5b]: Add AppenderV2 support to scrape.NewManager constructor (#17872)
* feat(scrape)[PART5b]: Add AppenderV2 support to scrape.NewManager optionally to V1

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Update scrape/manager.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* fixes after rebase

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Apply suggestions from code review

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

---------

Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2026-01-23 09:04:05 +00:00
yy
0e77dcfdb9
fix: typo in error message (#17878)
Signed-off-by: Yohei Yamamoto <yhymmt123@gmail.com>
2026-01-20 12:14:02 +01:00
Arve Knudsen
572f247b4d
tsdb: add auto-cleanup to newTestHead and remove redundant cleanup calls (#17890)
Add automatic cleanup to newTestHeadWithOptions so that heads created
with newTestHead are automatically closed when the test ends. This
simplifies test code by removing the need for manual cleanup in most
cases.

Changes:
- Add t.Cleanup in newTestHeadWithOptions immediately after creating
  the head, using _ = h.Close() to handle double-close gracefully
- Remove redundant t.Cleanup, defer, and explicit Close calls from
  tests that use newTestHead
- Add cleanup for heads created with NewHead directly in restart
  patterns (e.g., restartHeadAndVerifySeriesCounts, startHead)

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2026-01-19 12:57:05 +01:00
Arve Knudsen
dd85d7ca97
tsdb: fix memory leaks in buffer pools (#17879)
* tsdb: fix memory leaks in buffer pools

Clear reference fields when returning buffers to pools to avoid
retaining data after the buffer is reused.

Affected pools:
- refSeriesPool: clear Labels
- histogramsPool: clear H pointer
- floatHistogramsPool: clear FH pointer
- metadataPool: clear Unit and Help strings

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Clear slices

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Clear metadata buffer

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Remove tests

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

---------

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2026-01-19 11:14:31 +00:00
Bartlomiej Plotka
c4b0da94db
Merge pull request #17835 from prometheus/bwplotka/a2-storage-support
feat(storage)[PART4b]: add AppenderV2 to the rest of storage.Storage implementations + mock exemplar fix
2026-01-16 13:04:14 +00:00
George Krajcsovits
ef350b2b54
Merge pull request #17840 from prometheus/krajo/st-in-chunks
feat(tsdb): new Append parameter and AtST interface for chunks
2026-01-16 10:32:53 +01:00
Julius Hinze
e49d11c88d
tsdb: fix rare case of OOO exemplar insertion pointing to the exemplar being deleted (#17865)
Signed-off-by: Julius Hinze <julius.hinze@grafana.com>
2026-01-15 08:49:37 +01:00
Julius Hinze
ccb7468b09
tsdb: fix grow/shrink nextIndex calculation (#17863)
Signed-off-by: Julius Hinze <julius.hinze@grafana.com>
2026-01-14 16:44:50 +01:00
George Krajcsovits
06a59346fe
Update tsdb/chunkenc/chunk.go
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>
2026-01-14 16:41:03 +01:00
bwplotka
49c3aea56d feat(storage)[PART4b]: add AppenderV2 to the rest of storage.Storage impl
Signed-off-by: bwplotka <bwplotka@gmail.com>
2026-01-14 13:54:27 +00:00
György Krajcsovits
adf734db7a
update remaining tests
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-01-14 13:15:16 +01:00
György Krajcsovits
a5ac0bff1d
update ooo_head.go but only with TODOs
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-01-14 13:15:15 +01:00
György Krajcsovits
28dca34f4f
auto update head sample use in tests
find . -name "*.go" -type f -exec sed -E -i \
's/([^[:alpha:]]sample\{)([^,{:]+,[^,]+,[^,]+,[^,]+\})/\10, \2/g' {} +

I've omitted tsdb/ooo_head.go from the commit because I'm also adding todo
there.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-01-14 13:15:13 +01:00
György Krajcsovits
f616689f09
tsdb/head.go: add start timestamp to sample type used in tests
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-01-14 13:15:12 +01:00
György Krajcsovits
70bc06718d
feat(tsdb): new AppenderV2 and AtST interface for chunks
No implementation yet. Just to test the shape of the interface.
AtST is implemented for trivial cases, anything else is hard coded
to return 0.

Ref: https://github.com/prometheus/prometheus/issues/17791

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-01-14 13:15:09 +01:00
Ben Kochie
72a23934ad
Refactor various tsdb sub-packages (#17847)
Migrate various tsdb related packages from `tsdb/errors` to the standard
library `errors` package.

Signed-off-by: SuperQ <superq@gmail.com>
2026-01-13 13:38:58 +00:00
Bryan Boreham
da754cd240 Merge remote-tracking branch 'origin/release-3.9' into merge-3.9-into-main 2026-01-07 15:51:44 +00:00
Bryan Boreham
f1719fa1d4
[BUGFIX] Agent: fix crash from invalid type in pool (#17802)
We have separate pools for Appender and AppenderV2 objects, and must not
put another kind of object into them.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2026-01-07 14:01:02 +00:00
Julius Hinze
22463b1e9f
tsdb: add support for OOO exemplars in CircularExemplarStorage (#17469)
* tsdb: add support for OOO exemplars in CircularExemplarStorage

Doubly linked exemplar storage resize.
Split exemplar buffer resize into shrink and grow functions.
Skip duplicate OOO exemplars, re-initialize emptied index after deleting its last exemplar.

Signed-off-by: Julius Hinze <julius.hinze@grafana.com>
2026-01-07 13:25:50 +01:00
Patryk Prus
dcda4840a7
tsdb/index: export sentinel errors for size limit failures (#17773)
* tsdb/index: export sentinel errors for size limit failures

---------

Signed-off-by: Patryk Prus <p@trykpr.us>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2026-01-06 14:07:23 +01:00