mirrors/prometheus

mirror of https://github.com/prometheus/prometheus.git synced 2025-12-25 11:21:02 +01:00

History

Mauro Stettler 0df3489275

Write chunks via queue, predicting the refs (#10051 )

* Write chunks via queue, predicting the refs

Our load tests have shown that there is a latency spike in the
remote write handler whenever the head chunks need to be written,
because chunkDiskMapper.WriteChunk() blocks until the chunks are written
to disk.

This adds a queue to the chunk disk mapper which makes the WriteChunk()
method non-blocking unless the queue is full. Reads can still be served
from the queue.

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* address PR feeddback

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* initialize metrics without .Add(0)

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* change isRunningMtx to normal lock

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* do not re-initialize chunkrefmap

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* update metric outside of lock scope

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* add benchmark for adding job to chunk write queue

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* remove unnecessary "success" var

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* gofumpt -extra

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* avoid WithLabelValues call in addJob

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* format comments

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* addressing PR feedback

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* rename cutExpectRef to cutAndExpectRef

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* use head.Init() instead of .initTime()

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* address PR feedback

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* PR feedback

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* update test according to PR feedback

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* replace callbackWg -> awaitCb

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* better test of truncation with empty files

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* replace callbackWg -> awaitCb

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>

2022-01-10 13:36:45 +00:00

..

tsdb/agent: Synchronize appender code with grafana/agent main (#9664 )

2021-11-30 21:14:40 +05:30

improve bstream comments and doc (#9560 )

2021-10-25 18:44:15 +05:30

Write chunks via queue, predicting the refs (#10051 )

2022-01-10 13:36:45 +00:00

clarify HeadChunkID type and usage (#9726 )

2021-11-17 18:35:10 +05:30

Format Go source files using 'gofumpt -w -s -extra'

2021-11-02 19:52:34 +01:00

Address golint failures from revive

2021-10-23 00:53:11 +02:00

tsdb: Add windows arm64 support.

2021-11-09 11:07:27 +01:00

Add new Go build tags.

2021-08-27 10:24:14 +02:00

Tidy postingsWithIndexHeap (#10123 )

2022-01-06 16:03:44 +05:30

Add basic initial developer docs for TSDB (#9451 )

2021-11-17 15:51:27 +05:30

Move packages out of deprecated pkg directory

2021-11-09 08:03:10 +01:00

tsdb: Delete blocks atomically; Remove tmp blocks on start; Added test. (#7772 )

2020-08-11 06:56:08 +01:00

TSDB: demistify SeriesRefs and ChunkRefs (#9536 )

2021-11-06 15:40:04 +05:30

storage: fix bug #10027 in iterators' Seek method

2021-12-16 12:07:35 +01:00

fix: panic when checkpoint directory is empty (#9687 )

2021-11-17 16:39:04 +05:30

.gitignore

Moving tsdb into its own subdirectory

2019-08-13 13:58:49 +05:30

block_test.go

Move packages out of deprecated pkg directory

2021-11-09 08:03:10 +01:00

block.go

Move packages out of deprecated pkg directory

2021-11-09 08:03:10 +01:00

blockwriter_test.go

Move packages out of deprecated pkg directory

2021-11-09 08:03:10 +01:00

blockwriter.go

Move packages out of deprecated pkg directory

2021-11-09 08:03:10 +01:00

CHANGELOG.md

Rename default branch to main

2021-02-22 20:28:02 +01:00

compact_test.go

CompactBlockMetas should produce correct mint/maxt for overlapping blocks. (#10108 )

2022-01-05 15:10:00 +05:30

compact.go

CompactBlockMetas should produce correct mint/maxt for overlapping blocks. (#10108 )

2022-01-05 15:10:00 +05:30

db_test.go

Write chunks via queue, predicting the refs (#10051 )

2022-01-10 13:36:45 +00:00

db.go

Write chunks via queue, predicting the refs (#10051 )

2022-01-10 13:36:45 +00:00

example_test.go

Add basic initial developer docs for TSDB (#9451 )

2021-11-17 15:51:27 +05:30

exemplar_test.go

tsdb: use simpler map key to improve exemplar ingest performance (#10111 )

2022-01-06 15:58:58 +05:30

exemplar.go

tsdb: use simpler map key to improve exemplar ingest performance (#10111 )

2022-01-06 15:58:58 +05:30

head_append.go

Write chunks via queue, predicting the refs (#10051 )

2022-01-10 13:36:45 +00:00

head_bench_test.go

Move packages out of deprecated pkg directory

2021-11-09 08:03:10 +01:00

head_read.go

Label values with matchers by intersecting postings (#9907 )

2021-12-28 15:59:03 +01:00

head_test.go

Write chunks via queue, predicting the refs (#10051 )

2022-01-10 13:36:45 +00:00

head_wal.go

Fix queries after a failed snapshot replay (#9980 )

2021-12-09 15:15:27 +05:30

head.go

Write chunks via queue, predicting the refs (#10051 )

2022-01-10 13:36:45 +00:00

isolation_test.go

Add the ability to disable tsdb isolation (#9270 )

2021-11-19 15:41:32 +05:30

isolation.go

Add the ability to disable tsdb isolation (#9270 )

2021-11-19 15:41:32 +05:30

mocks_test.go

Move packages out of deprecated pkg directory

2021-11-09 08:03:10 +01:00

querier_bench_test.go

Label values with matchers by intersecting postings (#9907 )

2021-12-28 15:59:03 +01:00

querier_test.go

Move packages out of deprecated pkg directory

2021-11-09 08:03:10 +01:00

querier.go

Label values with matchers by intersecting postings (#9907 )

2021-12-28 15:59:03 +01:00

README.md

Add basic initial developer docs for TSDB (#9451 )

2021-11-17 15:51:27 +05:30

repair_test.go

Move packages out of deprecated pkg directory

2021-11-09 08:03:10 +01:00

repair.go

Switched to go-kit/log

2021-06-11 12:28:36 -04:00

tsdbblockutil.go

TSDB: demistify SeriesRefs and ChunkRefs (#9536 )

2021-11-06 15:40:04 +05:30

wal_test.go

Move packages out of deprecated pkg directory

2021-11-09 08:03:10 +01:00

wal.go

fix potential goroutine leaks

2021-12-17 18:35:30 -05:00

README.md

TSDB

This directory contains the Prometheus TSDB (Time Series DataBase) library, which handles storage and querying of all Prometheus v2 data.

Due to an issue with versioning, the "latest" docs shown on Godoc are outdated. Instead you may use the docs for v2.31.1

Documentation

External resources

A writeup of the original design can be found here.
Video: Storing 16 Bytes at Scale from PromCon 2017.
Compression is based on the Gorilla TSDB white paper.

A series of blog posts explaining different components of TSDB: