Fabian Reinartz
0ca755b4ae
Replace single head chunk per series with memSeries
...
This adds a memory series holding several chunk to replace
the single head chunk per series so far.
This is necessary for uniform maximum chunk sizes in cases
where some series have higher frequency samples than others.
2017-01-11 13:02:38 +01:00
Fabian Reinartz
c32a94d409
Unexport HeadBlock, export Block interface
2017-01-10 15:41:57 +01:00
Fabian Reinartz
d86e8a63c7
Report correct number of appended samples
2017-01-10 11:17:37 +01:00
Fabian Reinartz
29883a18fc
Add own Appender() method for DB
2017-01-09 22:54:08 +01:00
Fabian Reinartz
4c4e0c614e
Simplify position mapper updating
2017-01-09 19:24:05 +01:00
Fabian Reinartz
0dffd52238
Use page writer in compaction
2017-01-09 18:47:43 +01:00
Fabian Reinartz
89d8467f5c
Add missing lock
2017-01-09 18:07:45 +01:00
Fabian Reinartz
8c31c6e934
Make concurrent head chunk reads safe, fix misc races
...
This adds a 4 sample buffer to every head chunk. The XOR
compression scheme may edit bytes in place. The minimum size
of a sample is 2 bits. So keeping the last 4 samples in an in-memory
buffer makes it safe to query the preceeding ones while samples
are added
2017-01-09 16:51:39 +01:00
Fabian Reinartz
1943f8d1bb
Fix head block stats races
2017-01-07 18:02:17 +01:00
Fabian Reinartz
6aa922c5a6
Fix races
2017-01-07 16:20:32 +01:00
Fabian Reinartz
300f4e2abf
Use separate lock for series creation
...
This uses the head block's own lock to only lock if new series were
encountered.
In the general append case we just need to hold a
2017-01-06 18:10:50 +01:00
Fabian Reinartz
63e12807da
Don't update head postings mapper on every append
2017-01-06 16:43:18 +01:00
Fabian Reinartz
71efd2e08d
Periodically fsync WAL, make head cut async
2017-01-06 15:18:06 +01:00
Fabian Reinartz
96c2bd249f
Handle compaction trigger and reinitializing in DB
2017-01-06 13:03:23 +01:00
Fabian Reinartz
937cdb579c
Switch to sequential block names
...
This changes block directory names from the int64 timestamp
to sequential numbering.
2017-01-06 10:45:03 +01:00
Fabian Reinartz
9790aa98ac
Add postings wrapper that emits head postings in label set order
...
This adds a position mapper that takes series from a head block
in the order they were appended and creates a mapping representing
them in order of their label sets.
Write-repair of the postings list would cause very expensive writing.
Hence, we keep them as they are and only apply the postition mapping
at the very end, after a postings list has been sufficienctly reduced
through intersections etc.
2017-01-05 16:05:42 +01:00
Fabian Reinartz
5aa7f7cce8
Compact head block into persisted block
2017-01-04 21:11:15 +01:00
Fabian Reinartz
3f72d5d027
Fix last timestamp initialization
...
This initializes the chunkDesc's last timestamp to the minimum
value so initial samples with a timestamp of 0 (e.g. in tests)
are not accidentally dropped.
2017-01-04 14:06:40 +01:00
Fabian Reinartz
ac49f8c15e
Consolidate persistence and compaction
2017-01-03 16:49:37 +01:00
Fabian Reinartz
e7f04d14d5
Lock mmapped files
2017-01-03 10:09:20 +01:00
Fabian Reinartz
91b65b55e7
Run persistence in separate goroutine
2017-01-02 22:24:35 +01:00
Fabian Reinartz
a648ef5252
Convert persister into function
2017-01-02 16:58:47 +01:00
Fabian Reinartz
dbd2b21d2e
Make persistence atomic
2017-01-02 14:41:13 +01:00
Fabian Reinartz
beb842a856
Change block interface
...
This changes the block interface to directly expose index
and series readers rather than the direct querier
2017-01-02 11:12:28 +01:00
Fabian Reinartz
5d7ec06e04
Use ChunkMeta in SeriesWriter
2017-01-01 20:01:17 +01:00
Fabian Reinartz
c00d17e691
Modify IndexReader API to accomodate compaction
...
This changes the IndexReader API to expose plain labels
and chunk meta information instead of a Series interface.
Dropping of irrelevant chunks is moved into the querier.
A LabelIndices method is added to query for existing label
value indices.
2016-12-31 16:24:04 +01:00
Fabian Reinartz
1e1a37b15b
Remove double-reference in chunk hashmap
2016-12-31 10:19:02 +01:00
Fabian Reinartz
675f0886f0
Append to chunks cannot error
2016-12-31 10:10:27 +01:00
Fabian Reinartz
7280533c42
Add basic shard metrics
2016-12-31 09:48:49 +01:00
Fabian Reinartz
201d7687b6
Fix uniqueness of new series
2016-12-26 16:55:32 +01:00
Fabian Reinartz
787199a88e
Fix erroneous value assignments
2016-12-22 20:57:00 +01:00
Fabian Reinartz
8aba95048a
Rename to OpenHeadBlock
2016-12-22 20:00:24 +01:00
Fabian Reinartz
1dde3b6d31
Add WAL decoder+loading and benchmarks
2016-12-22 15:18:33 +01:00
Fabian Reinartz
0b8c77361e
Add initial WAL writing
2016-12-22 12:05:24 +01:00
Fabian Reinartz
2a825f6c28
Consolidate mem index into HeadBlock
2016-12-22 01:12:28 +01:00
Fabian Reinartz
ede733ab6c
Extract labels package
2016-12-21 09:39:01 +01:00
Fabian Reinartz
1b23d62e3f
Properly close files before reopening
2016-12-19 22:37:03 +01:00
Fabian Reinartz
282d9ae6e2
Implement label value queries in all layers.
2016-12-19 12:26:25 +01:00
Fabian Reinartz
bad93d8d57
Extract head serialization into Head method
2016-12-18 14:43:27 +01:00
Fabian Reinartz
b08f82fa4e
Pre-select relevant chunks on series access.
...
This adds interval metadata to indexed chunks. The queried interval
is used to filter chunks when queried from the index to save
unnecessary accesses of the chunks file.
This is especially relevant for series that come and go often and larger
files.
2016-12-16 12:13:17 +01:00
Fabian Reinartz
bd77103a49
Add stats serialization, load querier of all blocks
2016-12-15 16:14:33 +01:00
Fabian Reinartz
d56b281006
Rename Iterator to Postings
2016-12-15 07:05:29 +01:00
Fabian Reinartz
ca89080128
Misc fixes for initial Prometheus integration
2016-12-14 18:38:46 +01:00
Fabian Reinartz
14dbc59f2b
cleanup and switching removal of unsafe calls.
2016-12-10 18:09:57 +01:00
Fabian Reinartz
40a451694f
Refactor persistence into interfaces
2016-12-09 20:45:46 +01:00
Fabian Reinartz
74f8dfd95d
Persist blocks periodically
2016-12-09 13:41:38 +01:00
Fabian Reinartz
0cf8bb9e53
Move sub-indexes into single index structure
2016-12-09 10:41:51 +01:00
Fabian Reinartz
8aa99a3ebd
misc
2016-12-09 10:00:14 +01:00
Fabian Reinartz
2c34a15fe6
Add initial seriailization of block data
2016-12-08 17:43:10 +01:00
Fabian Reinartz
3ef7da33c8
Restructure files
2016-12-08 12:21:03 +01:00
