mirrors/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus.git synced 2025-08-11 00:27:12 +02:00
Code Issues Projects Releases Wiki Activity
997 Commits 286 Branches 479 Tags 642 MiB
c3ffdf1a99
Commit Graph

288 Commits

Author SHA1 Message Date
Fabian Reinartz
76c1b2cdb6 Ensure correct block order on reload 
Due to a regression blocks were no longer ordered by time before
being stored in memory. This made data intermittently become unavailable
for queries.

Signed-off-by: Fabian Reinartz <freinartz@google.com>
 2018-05-28 16:00:36 -04:00
Matt Bostock
def6e5a574 db: Match comment case to function name (#316) 
`validateBlockSequence` is not exported, make the letter case in the comment match the function name.

Signed-off-by: Matt Bostock <matt@mattbostock.com>
 2018-04-13 16:50:15 +01:00
Bartek Plotka
00594b85cd db: Addressed comments. 
Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
 2018-04-05 16:53:24 +01:00
Bartek Plotka
03e94365e1 db: Made overlap String() prettier and more readable. 
Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
 2018-04-05 16:01:16 +01:00
Fabian Reinartz
00e13f519a
Merge pull request #310 from Bplotka/overlap-detection-fix 
db: Fixed validateBlockSequence, exported it and added tests.
 2018-04-05 15:30:22 +02:00
Bartek Plotka
15b5d89222 db: Addressed comments. 
Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
 2018-04-05 14:15:24 +01:00
Bartek Plotka
cc306ef0d5 Added grouping by overlap range. 
Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
 2018-04-05 13:51:33 +01:00
Bartek Plotka
7412e2b44b Added more cases and modified one var name. 
Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
 2018-03-29 12:50:46 +01:00
Bartek Plotka
f07d829946 db: Tiny tuning of algo + added proper print. 
Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
 2018-03-28 23:50:42 +01:00
Bartek Plotka
c8b4a7b839 db: Simplified algorithm. 
Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
 2018-03-28 23:18:24 +01:00
Bartek Plotka
51ce1cc7ff db: Fixed validateBlockSequence. 
Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
 2018-03-28 18:33:41 +01:00
Bartek Plotka
a9b28a6aa0 db: Added tests for validateBlockSequence to confirm a bug. 
(That's why test fails)

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
 2018-03-28 15:50:52 +01:00
Mario Trangoni
e5dabad1d7 fix megacheck issue: simplified return err 2018-03-21 22:39:15 +01:00
Mario Trangoni
09142e4dd1 fix unconvert issues: unnecessary conversion 2018-03-21 22:39:14 +01:00
Matt Bostock
aa517c39bb
Fix typo in db.go 2018-03-02 11:12:32 +00:00
Fabian Reinartz
8cd32d8fd4 Improve usability of snapshots 2018-02-28 12:07:34 +01:00
Fabian Reinartz
a345231a0e Add test for block meta version and comments 2018-02-12 11:46:22 +01:00
Fabian Reinartz
04d5ac5aae Add repair test 2018-02-09 13:43:42 +01:00
Fabian Reinartz
d09283f60a Add repair of bad index version 2018-02-09 13:11:03 +01:00
Fabian Reinartz
5b26bd5f6f
Merge pull request #252 from simonpasquier/add-cuttoff-metrics 
Count the total & failed numbers of block cutoffs
 2018-02-09 12:28:22 +01:00
Goutham Veeramachaneni
b7bea04e62
Fix extra reloads. 
Previously we reloaded every 1 minute irrespective of whether there are
deleted blocks or not.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2018-01-26 19:33:59 +05:30
Simon Pasquier
e9fc37be61 Count the total and failed block cutoffs 2018-01-11 10:22:59 +01:00
Fabian Reinartz
7087f68647 Return new block ULID in compactor 2017-12-21 12:02:05 +01:00
Fabian Reinartz
67f0ca8f0e Move index and chunk encoders to own packages 2017-12-21 11:27:54 +01:00
Goutham Veeramachaneni
30bbbe34f8
Merge pull request #210 from Gouthamve/tomb-clean 
Add a function to cleanup tombstones.
 2017-11-30 15:28:01 +05:30
Goutham Veeramachaneni
f35e9c7fec
Merge branch 'master' into fix-dels 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-11-30 13:30:13 +05:30
Goutham Veeramachaneni
8a5ea9db74
Merge branch 'master' into tomb-clean 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-11-30 13:15:23 +05:30
Goutham Veeramachaneni
e7445d00b0
Add a function to cleanup tombstones. 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-11-27 20:03:37 +05:30
Fabian Reinartz
ad3c4849a9
Merge pull request #212 from prometheus/fabxc 
Cleanup, small refactoring, expose a few things
 2017-11-23 18:41:24 +01:00
Goutham Veeramachaneni
41fd9c66ef
Close the retention blocks before deleting them. 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-11-23 18:57:10 +05:30
Goutham Veeramachaneni
1627a47640
A copy is being assigned newChunk not the original 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-11-22 17:58:06 +05:30
Fabian Reinartz
a031cf7424 Return ULID in Compactor 2017-11-14 15:25:30 +01:00
zhangjun3
65e6b79a7c fix typing 2017-11-13 13:10:23 +08:00
Julius Volz
1dad3370fd Close WAL when closing the DB 
Also, the `wal` field of the `DB` was not used anywhere, so this removes
it.
 2017-11-11 14:56:23 +01:00
Tobias Schmidt
5170198276 Fix typo in prometheus_tsdb_reloads_failures_total description 2017-11-09 14:25:27 +01:00
Fabian Reinartz
d226411a2a db: delete old blocks during reload 
Windows requires blocks to be closed before deleting their directories.
This adds a set of deleteable blocks to reload(), which then deletes
them without causing disruption to querying.
 2017-11-03 20:41:24 +01:00
Your Name
4a4a706234 close blocks which will be remove 2017-11-02 00:30:19 +08:00
Fabian Reinartz
d109149d17 Merge pull request #186 from prometheus/closeallblocks 
Ensure readers are closed on setup failure.
 2017-10-25 09:31:06 +02:00
Fabian Reinartz
f8e88bfdb7 Close previous block queriers on error 
This ensures we close all previously opened queriers if on of the block
querier fails to open.
Also swap in new blocks before closing old ones to avoid the situation
in general. Make read locking of blocks more conservative to avoid
unnecessary retries by clients, e.g. when blocks are getting closed
before we can successfully instantiate querier against them.
 2017-10-23 21:56:12 +02:00
Fabian Reinartz
d17104f1f0 Prefix all metrics with prometheus_* 2017-10-20 12:32:32 +02:00
Fabian Reinartz
fb9da52b11 Add more verbose error handling for closing, reduce locking 
This commit introduces error returns in various places and is explicit
about closing persisted blocks.
{Index,Chunk,Tombstone}Readers are more consistent about their Close()
method. Whenever a reader is retrieved, the corresponding close method
must eventually be called. We use this to track pending readers against
persisted blocks.

Querier's against the DB no longer hold a read lock for their entire
lifecycle. This avoids long running queriers to starve new ones when we
have to acquire a write lock when reloading blocks.
 2017-10-10 12:13:37 +02:00
Fabian Reinartz
27f1b8aac3 Merge pull request #162 from BasPH/fsync-duration 
Instrument WAL fsync
 2017-10-05 08:18:36 +02:00
Bas Harenslak
5e1c258a98 Instrument WAL fsync 2017-10-04 22:17:02 +02:00
Goutham Veeramachaneni
da565f975e Merge pull request #161 from prometheus/fileutil 
Remove dependency on etcd/pkg/fileutil
 2017-10-04 17:08:54 +05:30
Fabian Reinartz
bbe72dccb9 Remove dependency on etcd/pkg/fileutil 2017-10-04 10:23:41 +02:00
Dimitrij Denissenko
83c984d9a3 Use configurable WAL flush inteval 2017-10-03 11:35:38 +01:00
Fabian Reinartz
e19b648377 Merge pull request #155 from Gouthamve/level-log 
Add levels to all log lines.
 2017-10-02 16:34:49 +02:00
Goutham Veeramachaneni
c35d3a65bd
Add levels to all log lines. 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-09-28 12:49:34 +05:30
Thibault Chataigner
0ff8f91ec8 Add db.Head() method 
Signed-off-by: Thibault Chataigner <t.chataigner@criteo.com>
 2017-09-25 16:45:24 +02:00
Thibault Chataigner
5ca18837e9 Make DB blocks list public. 
This enable computing the first timestamp in the DB
only accessing public interfaces.

Signed-off-by: Thibault Chataigner <t.chataigner@criteo.com>
 2017-09-21 11:56:12 +02:00
Goutham Veeramachaneni
8919baef03
Expose NewIndexReader() and cleanups 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-09-13 13:47:20 +05:30
Fabian Reinartz
f904cd385f Do not build a superflous 'all' postings 2017-09-08 18:41:43 +02:00
Fabian Reinartz
81222849bc Filter WAL data in Head, misc fixes 2017-09-06 16:20:37 +02:00
Fabian Reinartz
c36d574290 Replace single head lock with granular locks 
This adds various new locks to replace the single big lock on
the head. All parts now must be COW as they may be held by clients
after initial retrieval.
Series by ID and hashes are now held in a stripe lock to reduce
contention and total holding time during GC. This should reduce
starvation of readers.
 2017-09-05 14:41:39 +02:00
Fabian Reinartz
af2c2f9674 Re-enable head block delet test 2017-09-04 15:07:30 +02:00
Goutham Veeramachaneni
1698c516ad [WIP]: WAL implementation 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-09-04 14:52:40 +02:00
Fabian Reinartz
893b6ec506 Add tests for GC and chunk truncation 2017-09-01 14:38:49 +02:00
Fabian Reinartz
4cc37eecab Refactor and add tests for compactor 2017-09-01 11:50:58 +02:00
Fabian Reinartz
5cf2662074 Refactor WAL into Head and misc improvements 2017-09-01 11:50:58 +02:00
Fabian Reinartz
8209e3ec23 Add various metrics 2017-09-01 11:50:58 +02:00
Fabian Reinartz
3901b6e70b Remove multiple heads 
This changes the structure to a single WAL backed by a single head
block.
Parts of the head block can be compacted. This relieves us from any head
amangement and greatly simplifies any consistency and isolation concerns
by just having a single head.
 2017-09-01 11:50:58 +02:00
Andreas Kohn
4003da0fdd
Log the directory when the WAL cannot be opened 
Fixes #125
 2017-08-22 11:17:07 +02:00
Fabian Reinartz
905af27cf9 Refactor compactor 2017-08-09 11:10:29 +02:00
Fabian Reinartz
66ff7b12e9 Pool Chunk objects during compaction 2017-08-08 17:35:34 +02:00
Fabian Reinartz
fa04455cd7 Properly detect compactable heads 
The fullness of the wrong head was being tracked, causing wrong
lists of compactable heads.
 2017-08-03 13:58:11 +02:00
tianwenjie01
b64054365d Fix DefaultOptions 2017-07-18 17:42:58 +08:00
Fabian Reinartz
74f67e8271 Fix compacting disable/enable 
Enabling and disabling compaction no longer blocks are potentially
causes panics.
 2017-07-14 10:08:05 +02:00
Fabian Reinartz
3065be97d8 Fix and document locking order for DB 2017-07-14 09:00:22 +02:00
Fabian Reinartz
9c4235532e Fix compaction selection after creating new heads 
This fixes the case where between block creations no compaction
plans are ran. We were not compacting anything in these
cases since the on creation the most recent head block always had
a high timestamp of 0.
 2017-07-13 16:15:13 +02:00
Goutham Veeramachaneni
7b9c536883 Merge pull request #103 from Gouthamve/block-ranges 
Compaction implementation for block-ranges
 2017-07-12 17:32:49 +02:00
Goutham Veeramachaneni
9a2ab732b5 Choose ranges by retention and min-block-duration 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-07-12 17:30:34 +02:00
Goutham Veeramachaneni
f1ae239c20 Persist the right MaxTime when snapshotting 
This is because we cut a new block from where the snapshotted block ends
if we restore from backups and highTimestamp would be where we should be
 starting from.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-07-12 13:48:13 +02:00
Goutham Veeramachaneni
ce23adb579 Initial compaction implementation for block-ranges 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-07-07 16:32:18 +02:00
Fabian Reinartz
1e74c155eb Return empty string to signal non-caching 2017-06-26 14:58:00 +02:00
Fabian Reinartz
3410559c1b Compact head block early 
Let older head blocks be compacted once the newest once has samples at
50% of its total range. This allows the memory of the compacted blocks
to be released and garbage collected before a new head block gets
created. Thereby the number of head blocks is 1 or 2 instead of 2 or 3
and memory spikes are reduced.
 2017-06-26 08:52:59 +02:00
Fabian Reinartz
03f95ee096 Fix directory file descriptor leak. 2017-06-12 00:05:04 +02:00
Fabian Reinartz
d4cb579443 Add DB.Dir method 2017-06-08 12:14:13 +02:00
Goutham Veeramachaneni
ff4ccb6eb0
Remove unnecessary error from ToggleCompaction fns 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-06-06 23:45:23 +05:30
Goutham Veeramachaneni
261cd9f393
Incorporate feedback. 
Move back to {Enable, Disable}Compactions.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-06-06 20:23:20 +05:30
Goutham Veeramachaneni
a110a64abd
Add full Snapshot support 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-06-06 18:15:54 +05:30
Goutham Veeramachaneni
a1c8425357
Initial implementation of HeadBlock Snapshots 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-06-05 13:48:31 +05:30
Goutham Veeramachaneni
3a5ae6b1a4 Merge pull request #82 from Gouthamve/deletes-1 
Implement deletions
 2017-05-27 22:06:45 +05:30
Fabian Reinartz
b3b9245ad6 Add and fix metrics 2017-05-26 15:13:03 +02:00
Goutham Veeramachaneni
6febabeb28
Final delete fixes. 
* Make sure no reads happen on the block when delete is in progress.
* Fix bugs in compaction.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-05-26 16:31:45 +05:30
Goutham Veeramachaneni
9bf7aa9af1
Misc. fixes incorporating feedback. 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-05-23 18:13:30 +05:30
Goutham Veeramachaneni
8434019ad9
Merge branch 'master' into deletes-1 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-05-22 12:58:38 +05:30
Goutham Veeramachaneni
0b70333ef6
Add tests for tombstones. 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-05-21 23:20:05 +05:30
Goutham Veeramachaneni
d32eb25662
Synchronise deletes and compactions. 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-05-20 13:21:10 +05:30
Goutham Veeramachaneni
3dbb400bef
Initial implementation of Delete on DB. 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-05-20 00:35:50 +05:30
Fabian Reinartz
debfe9b1e5 Simplify AddFast and avoid errgroup Commits in general 2017-05-18 17:17:56 +02:00
Fabian Reinartz
fb5c5535fc Misc cleanup 2017-05-18 16:38:45 +02:00
Fabian Reinartz
39df7e2bba Switch blocks to ULID directories, drop sequenc numbers 2017-05-18 16:09:30 +02:00
Fabian Reinartz
285bc07030 Switch append refs to string 2017-05-18 10:56:57 +02:00
Fabian Reinartz
8b51b7e2be Make WAL for HeadBlock composeable. 2017-05-13 18:14:18 +02:00
Fabian Reinartz
5534e6c53c Make HeadBlock impl public, make interface private 2017-05-12 16:34:41 +02:00
Fabian Reinartz
44769c1654 Allow disabling lock file 2017-05-09 12:52:47 +02:00
Fabian Reinartz
35b62f001e Change offset table layout, add TOC, ... 2017-04-30 10:18:07 +02:00
Fabian Reinartz
54a7b98cff Fix and test ref handling in DB appender 2017-04-28 15:24:28 +02:00
Brian Brazil
d04bd947ea Merge pull request #51 from prometheus/rollback 
Make rollback actually rollback.
 2017-04-19 11:29:03 +01:00
Brian Brazil
dba222b53c Make rollback actually rollback. 
Also add tests for basic commit/rollback
functionality.

Fixes #39
 2017-04-19 11:28:36 +01:00