Commit Graph

110 Commits

Author SHA1 Message Date
Fabian Reinartz
09cd2021de Merge pull request #75 from Gouthamve/head-gen
E2E test for headBlock
2017-05-05 18:56:53 +02:00
Goutham Veeramachaneni
8096d11e4e
Add bounds check to headBlockAppender
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-05 19:52:11 +05:30
Goutham Veeramachaneni
adaf4d2099
Handle duplicate & out of order values in same txn
Add docs about not erroring out on exact dupes.
Moved tests to require.*

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-03 02:36:40 +05:30
Brian Brazil
72443bddfc Remove dead code. 2017-04-20 13:45:07 +01:00
Brian Brazil
bceb5c1b16 When checking for amended points, do it in terms of bits.
NaN != NaN, so the previous code would incorrectly report
it as changed.

There's also plans to take advantage of the NaN payload,
so look at the entire value.
2017-04-12 16:25:32 +01:00
Fabian Reinartz
778103b450 Add liecence file and headers 2017-04-10 20:59:45 +02:00
Fabian Reinartz
c73a397da2 Adjust maximum samples per chunk. 2017-04-07 10:58:37 +02:00
Fabian Reinartz
10c7c9acbe Adjust import names to new repository organisation 2017-04-04 11:27:26 +02:00
Fabian Reinartz
87d48bf9de Merge branch 'master' of github.com:fabxc/tsdb 2017-03-27 19:07:27 +02:00
Fabian Reinartz
a52980e0a8 Add workaround for deadlocks
This adds a workaround to avoid deadlocks for inconsistent write lock
order across headBlocks.
Things keep working if transactions only append data for the same
timestamp, which is generally the case for Prometheus.

Full behavior should be restored in a subsequent change.
2017-03-27 19:05:34 +02:00
Goutham Veeramachaneni
61f866bb94
Add Sample Back
The compilation and tests are broken as head.go requires sample which
has been moved to another package while moving BufferedSeriesIterator.

Duplication seemed better compared to exposing sample from tsdbutil.
2017-03-26 23:22:58 +05:30
Fabian Reinartz
3be4ef94ce Move BufferedSeriesIterator in own package
This functionality is useful for a lot of clients but not relevant to
the TSDB's core features.
2017-03-24 13:23:32 +01:00
Fabian Reinartz
e478d0e3bc Actually close olds blocks in reloadBlocks
This fixes a bug leaking memory because blocks were not actually closed
as the closing call references the initial, empty slice
2017-03-23 18:27:20 +01:00
Fabian Reinartz
789e8224ff Fix wrong comparison in head block resorting 2017-03-21 12:12:33 +01:00
Fabian Reinartz
55ee4b5b3b Merge branch 'master' of github.com:fabxc/tsdb 2017-03-21 10:11:39 +01:00
Fabian Reinartz
c18e055d7c Fix races and add comments on remaining ones 2017-03-21 10:11:23 +01:00
Fabian Reinartz
e837034360 Merge pull request #14 from Gouthamve/log-update
Update kit/log To New API
2017-03-21 09:56:32 +01:00
Fabian Reinartz
9c93f8f2aa Fix various races
This fixes different race condition encoutnered when running Prometheus.
It reduces the overall performance in the synthetic benchmark a fair bit
but has no indiciations of impacting a real-world setup notably.
2017-03-20 14:45:27 +01:00
Fabian Reinartz
3635569257 Trigger reload correctly on interrupted compaction 2017-03-20 10:41:43 +01:00
Fabian Reinartz
2c999836fb Add Queryable interface to Block
This adds the Queryable interface to the Block interface. Head and
persisted blocks now implement their own Querier() method and thus
isolate customization (e.g. remapPostings) more cleanly.
2017-03-20 10:21:21 +01:00
Fabian Reinartz
11be2cc585 Add composed Block interfaces, remove head generation
This adds more lower-leve interfaces which are used to compose
to different Block interfaces.
The DB only uses interfaces instead of explicit persistedBlock and
headBlock. The headBlock generation property is dropped as the use-case
can be implemented using block sequence numbers.
2017-03-20 09:02:36 +01:00
Goutham Veeramachaneni
df7db4ac07
Update kit/log To New API
NewContext has been removed couple of weeks back.
Ref: https://github.com/go-kit/kit/releases/tag/v0.4.0
2017-03-19 21:03:09 +05:30
Fabian Reinartz
65b846ae5b Remove unreturned locks, detect writes on closed heads 2017-03-17 12:12:50 +01:00
Fabian Reinartz
e0b33a7a28 Remove leftover signaling channels from headBlock 2017-03-17 10:16:55 +01:00
Fabian Reinartz
34efe4e2c8 Drop position mapper from head block
The position mapper was intended to pre-computed "expensive" ordering
of label sets. It was expensive to update and caused a lot of trouble.
Skipping this optimization entirely actually revelead it was pointless
and even harmful from the e2e perspective.
2017-03-15 14:44:29 +01:00
Fabian Reinartz
ad5812d83a Handle WAL corruption by truncating
This adds handling for various corruption scenarios of the WAL.
If corruption is encountered, we truncate the WAL after the last valid
entry transparently and continue appending after the offset.
2017-03-15 11:16:16 +01:00
Fabian Reinartz
a8e8903350 Use ChunkMeta references for clarity
This has been a common source of hard to debug issues. Its a premature
and unbenchmarked optimization and semantically, we want ChunkMetas to
be references in all changed cases.
2017-03-14 15:40:16 +01:00
Fabian Reinartz
e825a0b40c Make mapper updates asynchronous 2017-03-14 10:57:00 +01:00
Fabian Reinartz
cb4dde7659 Fix WAL log recovery bug
This fixes a bug where the last WAL file was closed after consuming it
instead of being left open for further writes.
Reloading of blocks on startup considers loading head blocks now.
2017-03-08 16:53:07 +01:00
Fabian Reinartz
87805fb83f Remove Partitioned* code 2017-03-06 17:34:49 +01:00
Fabian Reinartz
50791a412e Remove leaky Appender abstraction for hashedAppend 2017-03-06 14:27:33 +01:00
Fabian Reinartz
55a9b5428a Add separate head mutex
Introduce a seperate mutex for the head blocks to avoid a race where
a post-compaction reload may run between switching the DB's base mutex
to create a new head block in an appender.
2017-03-04 17:41:01 +01:00
Fabian Reinartz
92120448c2 Properly cleanup compacted dirs, fixes, docs 2017-03-02 14:32:09 +01:00
Fabian Reinartz
2c3e778d90 Compactor interface 2017-03-02 09:17:30 +01:00
Fabian Reinartz
306831f151 Add per-block state ULID 2017-02-27 10:46:15 +01:00
Fabian Reinartz
78780cd2ba Segment chunk file
This adds write path support for segmented chunk data files.
Files of 512MB are pre-allocated and written to. If the file size
is exceeded, the next file is started. On completion, files
are truncated to their final size.
2017-02-23 10:50:22 +01:00
Fabian Reinartz
a3d042b54e Support multiple chunk files in read path 2017-02-18 17:33:20 +01:00
Fabian Reinartz
9c7a88223e Add full encode/decode WAL cycle test 2017-02-14 21:55:50 -08:00
Fabian Reinartz
f1435f2e2c Track appended samples properly in metric 2017-02-08 16:13:16 -08:00
Fabian Reinartz
012cf4ef25 Count writer references on head blocks 2017-02-04 11:53:52 +01:00
Fabian Reinartz
5a1c8eaa0e Fix missing appends after reference lookups 2017-02-02 11:10:17 +01:00
Fabian Reinartz
ac5229e1b4 Correctly write empty blocks and extend appenders by new blocks 2017-02-02 07:58:54 +01:00
Fabian Reinartz
30efe4a58c Support writing to multiple head blocks
This is an initial (and hacky) first pass on allowing
appending to multiple blocks simultaniously to avoid
dropping samples right after cutting a new head block.
It's also required for cases like the PGW, where a scrape may
contain varying timestamps.
2017-02-01 15:57:28 +01:00
Fabian Reinartz
c20cc44b06 Add docs, write sequence number to meta.json 2017-01-29 08:11:47 +01:00
Fabian Reinartz
472c618c39 Drop out-of-bound samples 2017-01-19 15:03:57 +01:00
Fabian Reinartz
d4779b374c Properly track and write meta file 2017-01-19 14:01:38 +01:00
Fabian Reinartz
9ddbd64d00 Move stats into meta.json file, cleanup, docs 2017-01-19 11:22:47 +01:00
Fabian Reinartz
5ceca3c810 Write to WAL before appending to memory storage 2017-01-17 16:33:58 +01:00
Fabian Reinartz
343dd9d94c Fix wrong byte size in WAL base ref 2017-01-17 08:40:31 +01:00
Fabian Reinartz
5fb01d41aa Use new Prometheus text format parser 2017-01-16 21:29:53 +01:00