1538 Commits

Author SHA1 Message Date
bwplotka
9db326ad64 feat[head]: implement EnableSTStorage for float appending
Signed-off-by: bwplotka <bwplotka@gmail.com>
2026-02-17 10:05:26 +00:00
Owen Williams
b57f5b59b3
tsdb: ST-in-WAL: Counter implementation and benchmarks (#17671)
Initial implementation of https://github.com/prometheus/prometheus/issues/17790.
Only implements ST-per-sample for Counters. Tests and benchmarks updated.

Note: This increases the size of the RefSample object for all users, whether st-per-sample is turned on or not.

Signed-off-by: Owen Williams <owen.williams@grafana.com>
2026-02-12 13:17:50 -05:00
György Krajcsovits
64208c7e62
Add cases with jitter in the start time as well
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-02-11 16:34:44 +01:00
Bartlomiej Plotka
d86068c52e
Take latest from main (#18060)
* Update npm dependencies for v3.10

Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>

* Fix critical npm vulnerabilities with npm audit fix

Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>

* UI: Move HistoryCompleteStrategy into its own file and fix lint

Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>

* [REFACTOR] Relabel: Remove unnecessary Process() function

All uses can be replaced by ProcessBuilder, which is more efficient.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* promql: use Kahan summation for Native Histograms (#15687)

As for float samples, Kahan summation is used for the `sum` and `avg` aggregation and for the respective `_over_time` functions.

Kahan summation is not perfect. This commit also adds tests that even Kahan summation cannot reliably pass.
These tests are commented out.

Note that the behavior might be different on other hardware platforms. We have to keep an eye on test failing on other hardware platforms and adjust them accordingly.

Signed-off-by: Aleksandr Smirnov <5targazer@mail.ru>

* fix(docs): typo in prometheus_agent.md doc

Signed-off-by: Mohammad Abbasi <mohammad.v184@gmail.com>

* AWS SD: ECS Standalone Tasks

The current ECS role in AWS SD assumes that a task is part of a service.
This means that tasks that are started as part of AWS Batch will get
missed and not be discovered. This changed fixes this so that standalone
tasks can be discovered as well.

Signed-off-by: matt-gp <small_minority@hotmail.com>

* AWS SD: Optmise MSK Role

Signed-off-by: matt-gp <small_minority@hotmail.com>

* promtool: support missing promql syntax features (#17926)

Namely promql-duration-expr and promql-extended-range-selectors. This
allows promtool to e.g. check rules files using syntax gated by these
features.

Signed-off-by: Ian Kerins <git@isk.haus>

* Also run CI on release tags

Make sure we also run the main CI workflow `v*` release tags
so `publish_release` job is run.

Signed-off-by: SuperQ <superq@gmail.com>

* chore(deps): bump github.com/hashicorp/consul/api from 1.32.0 to 1.33.2 (#17449)

Bumps [github.com/hashicorp/consul/api](https://github.com/hashicorp/consul) from 1.32.0 to 1.33.2.
- [Release notes](https://github.com/hashicorp/consul/releases)
- [Changelog](https://github.com/hashicorp/consul/blob/main/CHANGELOG.md)
- [Commits](https://github.com/hashicorp/consul/compare/api/v1.32.0...api/v1.33.2)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/consul/api
  dependency-version: 1.33.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>

* AWS SD: Load Region Fallback (#18019)

* AWS SD: Load Region Fallback

---------

Signed-off-by: matt-gp <small_minority@hotmail.com>

* Build riscv64 docker image by default (#17508)

* Allow building riscv64 docker  image

Co-authored by: nijincheng@iscas.ac.cn;

Signed-off-by: ffgan <sudoemt@gmail.com>

* Update Makefile

Co-authored-by: Ben Kochie <superq@gmail.com>
Signed-off-by: ffgan <sudoemt@gmail.com>

---------

Signed-off-by: ffgan <sudoemt@gmail.com>
Co-authored-by: Ben Kochie <superq@gmail.com>

* Increase Renovate speed (#18052)

* Allow Renovate to run all day on the monthly day.
* Increase the max PRs from 10 to 20.
* Incresae the PR open rate from 2 to 5.

Signed-off-by: SuperQ <superq@gmail.com>

* docs(api): clarify metadata vs remote protocols (#17481)

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* tsdb/index: export sentinel error for symbol table size exceeded

Signed-off-by: Patryk Prus <p@trykpr.us>

* Update Go dependencies for v3.10 release

Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>

* Kubernetes SD: Disable WatchListClient in tests

Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>

* Replace deprecated google.CredentialsFromJSON with option.WithAuthCredentialsFile

Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>

* chore(deps): update actions/checkout action to v4.3.1 (#18054)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

* chore(deps): update actions/cache action to v5.0.3 (#18053)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

* chore(deps): update actions/setup-go action to v6.2.0 (#18055)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

---------

Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Signed-off-by: Aleksandr Smirnov <5targazer@mail.ru>
Signed-off-by: Mohammad Abbasi <mohammad.v184@gmail.com>
Signed-off-by: matt-gp <small_minority@hotmail.com>
Signed-off-by: Ian Kerins <git@isk.haus>
Signed-off-by: SuperQ <superq@gmail.com>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Signed-off-by: ffgan <sudoemt@gmail.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: Patryk Prus <p@trykpr.us>
Co-authored-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
Co-authored-by: Bryan Boreham <bjboreham@gmail.com>
Co-authored-by: Sasha <103973965+crush-on-anechka@users.noreply.github.com>
Co-authored-by: Mohammad Abbasi <mohammad.v184@gmail.com>
Co-authored-by: matt-gp <small_minority@hotmail.com>
Co-authored-by: Ian Kerins <git@isk.haus>
Co-authored-by: SuperQ <superq@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Julien <291750+roidelapluie@users.noreply.github.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Co-authored-by: ffgan <sudoemt@gmail.com>
Co-authored-by: Patryk Prus <p@trykpr.us>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Joe Adams <github@joeadams.io>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-02-11 09:20:27 +00:00
Patryk Prus
d8c24c6bde
tsdb/index: export sentinel error for symbol table size exceeded
Signed-off-by: Patryk Prus <p@trykpr.us>
2026-02-10 12:44:32 -05:00
George Krajcsovits
411eb36b13
Merge pull request #17909 from prometheus/krajo/st-in-xorchunk
feat(tsdb/chunkenc): add float chunk format with start timestamp support
2026-02-06 19:28:22 +01:00
György Krajcsovits
b2491c7cf7
fix(chunkenc): bug in initializing appender on existing chunk
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-02-06 15:35:45 +01:00
György Krajcsovits
07c543f02d
test(chunkenc): test that appender can contonue chunks
Test that initializing a chunk appender from an existing chunk
works correctly.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-02-06 15:35:20 +01:00
Ganesh Vernekar
fe5cb190e6
tsdb: Add metrics for stale series compaction (#17957)
Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
2026-02-06 09:05:56 +00:00
György Krajcsovits
de4a24325e
add st equal t to bechmarks
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-02-05 18:52:37 +01:00
György Krajcsovits
65efa8f95a
test the case where st equals the t
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-02-05 18:44:01 +01:00
Bartlomiej Plotka
c8e7f4e2a6
tests: Unify TestDiskFillingUpAfterDisablingOOO and avoid hiding errors (#18017)
* tests: Unify TestDiskFillingUpAfterDisablingOOO and avoid hiding errors

Signed-off-by: bwplotka <bwplotka@gmail.com>

* addressed comments

Signed-off-by: bwplotka <bwplotka@gmail.com>

---------

Signed-off-by: bwplotka <bwplotka@gmail.com>
2026-02-05 16:11:35 +00:00
György Krajcsovits
2917c91727
Merge remote-tracking branch 'origin/main' into krajo/st-in-xorchunk 2026-02-04 21:26:18 +01:00
György Krajcsovits
59bd0a2a30
Reduce size of first st stored a little
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-02-04 21:20:32 +01:00
György Krajcsovits
e777d4077c
verify chunk sample len function
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-02-04 19:48:15 +01:00
Ben Kochie
8d491cc642
tsdb: Migrate multi-errors to errors package (#17768)
Modernize tsdb package by migrating multi-error handling
to the standard library errors package.
* Add a modernized CloseAll helper.

Signed-off-by: SuperQ <superq@gmail.com>
2026-02-04 10:41:57 +01:00
György Krajcsovits
bbc85a3a96
Write ST after T and V so we can write a single bit on the second sample
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-02-03 22:11:04 +01:00
György Krajcsovits
59a6706abb
Switch to delta of difference of st to prev t
from delta of delta of st.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-02-03 20:55:44 +01:00
György Krajcsovits
9fc9033ad8
Add delta st inclusive test case for random vt
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-02-03 08:24:12 +01:00
Bartlomiej Plotka
eefa6178fb
fix: fix rare race on empty head.initialized() vs head.initTime() (#17963)
* fix: fix rare race on empty head.initized() vs head.initTime()

Relates to https://github.com/prometheus/prometheus/issues/17941

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Apply suggestions from code review

Co-authored-by: Owen Williams <owen.williams@grafana.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* addressed comments

Signed-off-by: bwplotka <bwplotka@gmail.com>

---------

Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: Owen Williams <owen.williams@grafana.com>
2026-02-02 09:13:02 +00:00
Arve Knudsen
d9db76631d
tsdb: fix flaky TestWaitForPendingReadersInTimeRange tests (#17985)
The tests were flaky because they used hard-coded time.After(550ms)
waits, which had only 50ms margin over WaitForPendingReadersInTimeRange's
500ms poll interval. On slow CI runners, this margin wasn't reliable.

Use synctest for deterministic time control:
- Wrap test logic in synctest.Test() to use fake time
- Use synctest.Wait() to let goroutines reach dormant state
- Use time.Sleep() to advance fake time past the poll interval
- No more timing-dependent assertions

This makes the tests both reliable and ~60x faster (0.05s vs 3s).

Fixes both TestWaitForPendingReadersInTimeRange and
TestWaitForPendingReadersInTimeRange_AppenderV2.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2026-02-01 15:52:26 +00:00
György Krajcsovits
e7c5105772
Merge remote-tracking branch 'origin/main' into krajo/st-in-xorchunk 2026-01-30 13:46:16 +01:00
Bartlomiej Plotka
88f6ee4c8e
tests(scrape): add TestScrapeLoopAppend_WithStorage (#17937)
Signed-off-by: bwplotka <bwplotka@gmail.com>
2026-01-30 11:44:07 +00:00
György Krajcsovits
1ac3b8d8ec
Small simplification in the doc
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-01-30 11:34:59 +01:00
György Krajcsovits
6133dd5f3e
documentation
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-01-30 11:31:54 +01:00
György Krajcsovits
475dfb5ea9
Drop commented out line
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-01-30 10:41:05 +01:00
György Krajcsovits
65b70bbe77
drop unused code
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-01-30 10:39:50 +01:00
György Krajcsovits
e6a0193a27
make testcases independent of order
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-01-30 09:30:05 +01:00
György Krajcsovits
4ef8186575
add inclusive delta test case
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-01-30 09:23:29 +01:00
György Krajcsovits
a0314d5539
Fix benchmark expectations on ST in interator
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-01-30 09:01:29 +01:00
Bartlomiej Plotka
0fc70f3a62
tsdb: kill unused mint,maxt tracking (#17967)
Signed-off-by: bwplotka <bwplotka@gmail.com>
2026-01-29 14:26:40 +00:00
Arve Knudsen
00a7faa2e3
tsdb: fix division by zero in stale series compaction (#17952)
Guard the stale series ratio calculation by checking numSeries > 0
before computing the ratio. This prevents division by zero when
the head has no series.

Fixes #17949

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2026-01-29 08:06:00 +01:00
Bartlomiej Plotka
2597a12080
st: Add a hidden 'st-storage' feature flag for PROM-60 (#17907)
Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2026-01-28 09:05:54 +00:00
Björn Rabenstein
15030d6f19
Merge pull request #17738 from aknuds1/arve/leveled-compactor
enhancement(tsdb): add test for `LeveledCompactor.Plan` stopping after excluding block
2026-01-27 12:38:42 +01:00
György Krajcsovits
5d0f59d8fe
reduce footprint of the xoroptst chunk iterator object
It was 80 bytes with a lot of padding compared to the 56 bytes of the
original xor chunk iterator. Made it 64 bytes, tightly packed.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-01-26 11:07:52 +01:00
György Krajcsovits
e55f09f460
fix iterator benchmark for chunks not supporting ST
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-01-26 10:27:24 +01:00
Arve Knudsen
c8ff2d739b enhancement(tsdb): add test for LeveledCompactor stopping after excluding block
Add a test for `LeveledCompactor.Plan()` stopping after a block matches the
`BlockExcludeFilter`, as a sub-test
`TestLeveledCompactor/Plan/BlockExcludeFilter stops iteration`.

Also moving `TestLeveledCompactor_plan` to a sub-test
of `TestLeveledCompactor`, for consistency.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2026-01-24 16:07:22 +01:00
Ganesh Vernekar
3e4a094dbb Add stale_series_compaction_threshold config file option
Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
2026-01-23 18:12:34 -08:00
Ganesh Vernekar
4f3de8da29 tsdb: Add unit tests for stale series compaction
Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
2026-01-23 18:07:34 -08:00
Ganesh Vernekar
43dc23afe7 tsdb: Clear stale series from the Head during WAL replay
Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
2026-01-23 18:02:45 -08:00
Ganesh Vernekar
43e69388df tsdb: Add stale series compaction support in the DB
Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
2026-01-23 18:00:08 -08:00
Ganesh Vernekar
9b444b57af tsdb: Add StaleHead and GC for stale series in the Head block
Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
2026-01-23 17:59:41 -08:00
Bartlomiej Plotka
bec70227f1
feat(scrape)[PART5b]: Add AppenderV2 support to scrape.NewManager constructor (#17872)
* feat(scrape)[PART5b]: Add AppenderV2 support to scrape.NewManager optionally to V1

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Update scrape/manager.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* fixes after rebase

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Apply suggestions from code review

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

---------

Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2026-01-23 09:04:05 +00:00
György Krajcsovits
ca530d7f85
fix issue with seeking to last sample again
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-01-22 18:08:41 +01:00
György Krajcsovits
091379dc44
make new format usable in head
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-01-22 14:17:51 +01:00
György Krajcsovits
8c14b3c99c
optimize code path and layout
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-01-22 14:03:53 +01:00
György Krajcsovits
4b8fb76d95
feat(tsdb/chunkenc): add float chunk format with start timestamp support
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2026-01-22 08:57:38 +01:00
yy
0e77dcfdb9
fix: typo in error message (#17878)
Signed-off-by: Yohei Yamamoto <yhymmt123@gmail.com>
2026-01-20 12:14:02 +01:00
Arve Knudsen
572f247b4d
tsdb: add auto-cleanup to newTestHead and remove redundant cleanup calls (#17890)
Add automatic cleanup to newTestHeadWithOptions so that heads created
with newTestHead are automatically closed when the test ends. This
simplifies test code by removing the need for manual cleanup in most
cases.

Changes:
- Add t.Cleanup in newTestHeadWithOptions immediately after creating
  the head, using _ = h.Close() to handle double-close gracefully
- Remove redundant t.Cleanup, defer, and explicit Close calls from
  tests that use newTestHead
- Add cleanup for heads created with NewHead directly in restart
  patterns (e.g., restartHeadAndVerifySeriesCounts, startHead)

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2026-01-19 12:57:05 +01:00
Arve Knudsen
dd85d7ca97
tsdb: fix memory leaks in buffer pools (#17879)
* tsdb: fix memory leaks in buffer pools

Clear reference fields when returning buffers to pools to avoid
retaining data after the buffer is reused.

Affected pools:
- refSeriesPool: clear Labels
- histogramsPool: clear H pointer
- floatHistogramsPool: clear FH pointer
- metadataPool: clear Unit and Help strings

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Clear slices

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Clear metadata buffer

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Remove tests

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

---------

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2026-01-19 11:14:31 +00:00