While preparing PR #16701, we identified an inconsistency in the chunk
format documentation. The `varint` encoding can require up to 10 bytes
for a 64-bit integer, such as when timestamps are encoded. However, the
chunk length field is a 32-bit integer, which requires at most 5 bytes
in `varint` encoding.
This is reflected in the code, where a maximum of 5 bytes are read when
parsing the chunk length.
50ba25f273/tsdb/chunks/chunks.go (L709-L711)50ba25f273/tsdb/chunks/chunks.go (L47-L48)
Co-authored-by: Istvan Zoltan Ballok <istvan.zoltan.ballok@sap.com>
Signed-off-by: Victor Herrero Otal <victor.herrero.otal@sap.com>
This change updates `DirSize` to ignore `os.ErrNotExist` errors,
since they are expected during normal WAL cleanup. All other errors
continue to propagate.
Fixes: #17005
Signed-off-by: Sujal Shah <sujalshah28092004@gmail.com>
This is intended to make `intersectPostings` easier to follow.
Instead of cryptic `arr` and `cur`, name the members `postings` and
`current`.
Instead of updating `cur` to intermediate values encountered during
operations, introduce a local variable `target` meaning the ref we might
expect to find next, and only update `current` when an intersection is
found.
Name the function which implements seeking `Seek` instead of `doNext`.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Lets take the given example:
P1: [2, 5, 9, 18, 21]
P2: [3, 7, 14, 19, 21]
P3: [1, 21]
Currently, we would only advance through P1 and P2 until discovering
an intersection and then checking P3. In essence, the traversal order
was: 2, 3, 5, 7, 9, 14, 18, 19, 21 (intersection found).
With the proposed change, P3 is also examined even if P1 and P2
haven't found an intersection yet. This adjustment allows for the
possibility of skipping some iterations.
Post-change, the traversal order becomes: 2, 3, 21 (3 iterations instead of 9).
Signed-off-by: alanprot <alanprot@gmail.com>
* test(chunkenc): appending histograms with empty buckets and gaps
Append such native histograms that have empty buckets and gaps
between the indexes of those buckets.
There is a special case for appending counter native histograms to a chunk in TSDB: if we append a histogram that is missing some buckets that are already in chunk, then usually that's a counter reset. However if the missing bucket is empty, meaning its value is 0, then we don't consider it missing.
For this case to trigger , we need to write empty buckets into the chunk. Normally native histograms are compacted when we emit them , so this is very rare and compact make sure that there are no multiple continuous empty buckets with gaps between them.
The code that I've added in #14513 did not take into account that you can bypass compact and write histograms with many empty buckets, with gaps between them. These are still valid, so the code has to account for them.
Main fix in the expandIntSpansAndBuckets and expandFloatSpansAndBuckets function. I've also refactored them for clarity. Consequently needed to fix insert and adjustForInserts to also allow gaps between inserts.
I've added some new test cases (data driven test would be nice here, too many cases). And removed the deprecated old function that didn't deal with empty buckets at all.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>
Co-authored-by: Björn Rabenstein <beorn@grafana.com>
* Unregister metrics emitted by `remote.WriteStorage` when closed
Signed-off-by: Charles Korn <charles.korn@grafana.com>
* Address PR feedback: add test
Signed-off-by: Charles Korn <charles.korn@grafana.com>
---------
Signed-off-by: Charles Korn <charles.korn@grafana.com>
* tsdb: fix issue where a new segment file is created for every chunk
Signed-off-by: Charles Korn <charles.korn@grafana.com>
* Address PR feedback
Signed-off-by: Charles Korn <charles.korn@grafana.com>
---------
Signed-off-by: Charles Korn <charles.korn@grafana.com>
The custom values are the "le" bucket boundaries of native histograms
with custom buckets. They are never modified. It is ok to not copy them
when iterating a chunk, just reference them.
If we will ever have a function that modifies the custom values, like
'trim' for example. That function will have to make a copy on write.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* tsdb/errors.MultiError: implement Unwrap
the multierror was hiding some errors in Mimir. I also added unit tests because I had them handy from a similar change I and yuri did in XXX and some time ago
---------
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
