We have always validated that none of the bucket is negative. We
should do the same for the count of observations and the zero bucket.
Note that this was always implied in the protobuf exposition format
because a count or a zero bucket population is ignored if it is not
positive.
Signed-off-by: beorn7 <beorn@grafana.com>
Histogram.Validate and FloatHistogram.Validate now return error on
unsupported schemas.
Scrape and remote-write handler reduces the schema to the maximum allowed
if it is above the maximum, but below theoretical maximum of 52.
For scrape the maximum is a configuration option, for remote-write it is 8.
Note: OTLP endpont already does the reduction, without checking that it is
below 52 as the spec does not specify a maximum.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* OTLP writer writes directly to appender
Do not convert to Remote-Write 1.0 protocol. Convert to TSDB Appender interface instead.
For downstream projects that still convert OTLP to something else (e.g. Mimir using
its own RW 1.0+2.0 compatible protocol), introduce a compatibility layer between
OTLP decoding and TSDB Appender. This is the CombinedAppender that hides the
implementation. Name is subject to change.
---------
Signed-off-by: David Ashpole <dashpole@google.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>
Co-authored-by: David Ashpole <dashpole@google.com>
Co-authored-by: Jesus Vazquez <jesusvazquez@users.noreply.github.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Modify storage/remote.writeHandler.ServeHTTP to return after responding
with an error due to receiving an unrecognized protobuf message type.
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
* PROM-39: Add type and unit labels to OTLP endpoint
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* Extract label addition into helper function
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* Wire feature flag and web handler configuration
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* Apply suggestions from code review
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* Use lowercase for units too
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* Use otlptranslator.UnitNamer to build units
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* Address copilot's comment
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* Verify label presence before adding them
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* Overwrite type/unit labels when already set
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* sed/addTypeAndUnitLabels/enableTypeAndUnitLabels/
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* Reduce duplicated code
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
---------
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
* Support including scope metadata as metric labels
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* Ensure Scope Name, Version and Schema URL aren't overriden by attributes
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
---------
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
The last permutation of the translation options does underscore translation but does not add suffixes.
This translation option already exists in Mimir as otel_metric_suffixes_enabled, indicating external demand for this strategy.
There is an accompanying update to prometheus-docs to explain the use of this mode: https://github.com/prometheus/docs/pull/2688
Signed-off-by: Owen Williams <owen.williams@grafana.com>
Reverts #16730 and #16760
This is being done because we've noticed a problem in the spec that could
lead to name collisions if attributes name, version or schema_url are added
to the scope. They would collide with the already reserved labels
otel_scope_name, otel_scope_version and otel_scope_schema_url.
Since this new configuration option never made it into a release, we can
safely remove it from the 3.5 release. We'll sort this out for the 3.6 release
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
* OTLP receiver: Generate target_info samples between the earliest and latest samples per resource
Modify the OTLP receiver to generate target_info samples between the earliest
and latest samples per resource instead of only one for the latest timestamp.
The samples are spaced lookback delta/2 apart.
---------
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Add support for promoting all OTel resource attributes via `promote_all_resource_attributes`,
except for those ignored using 'ignore_resource_attributes'.
---------
Signed-off-by: Antonio Jimenez <antonjim@thousandEyes.com>
Signed-off-by: Antonio Jimenez <123171955+antonjim-te@users.noreply.github.com>
* feat: Support 'NoTranslation' mode in OTLP endpoint
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
---------
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
Global and Data Source configurations can specify legacy mode, but Prometheus now requires that the overall validation mode be set to UTF-8
Signed-off-by: Owen Williams <owen.williams@grafana.com>
What
Adds support for OTLP delta temporality to the OTLP endpoint.
This is done by calling the deltatocumulative processor from the OpenTelemetry collector during OTLP conversion.
Why
Delta conversion is a naturally stateful process, which requires careful request routing when operated inside a collector.
Prometheus is already stateful and doing the conversion in-server reduces the operational burden on the ingest architecture by only having one stateful component.
How
deltatocumulative is a OTel collector component that works as follows:
* pmetric.Metrics come from a receiver or in this case from the HTTP client
* It operates as an in-place update loop:
* for each sample, if not delta, leave unmodified
* if delta, do:
* state += sample, where state is the in-memory sum of all previous samples
* sample = state, sample value is now cumulative
* this is supported for sums (counters), gauges, histograms (old histograms) and exponential histograms (native histograms)
If a series receives no new samples for 5m, its state is removed from memory
Performance
Delta performance is a stateful operation and the OTel code is not highly optimized yet, e.g. it locks the entire processor for each request. Nonetheless, care has been taken to mitigate those effects:
delta conversion is behind a feature flag. If disabled, no conversion code is ever invoked
if enabled, conversion is not invoked if request not actually contains delta samples. This leads to no measureable performance difference between default-cumulative to convert-cumulative (only cumulative, feature on/off)
Signed-off-by: sh0rez <me@shorez.de>
For: #14355
This commit updates Prometheus to adopt stdlib's log/slog package in
favor of go-kit/log. As part of converting to use slog, several other
related changes are required to get prometheus working, including:
- removed unused logging util func `RateLimit()`
- forward ported the util/logging/Deduper logging by implementing a small custom slog.Handler that does the deduping before chaining log calls to the underlying real slog.Logger
- move some of the json file logging functionality to use prom/common package functionality
- refactored some of the new json file logging for scraping
- changes to promql.QueryLogger interface to swap out logging methods for relevant slog sugar wrappers
- updated lots of tests that used/replicated custom logging functionality, attempting to keep the logical goal of the tests consistent after the transition
- added a healthy amount of `if logger == nil { $makeLogger }` type conditional checks amongst various functions where none were provided -- old code that used the go-kit/log.Logger interface had several places where there were nil references when trying to use functions like `With()` to add keyvals on the new *slog.Logger type
Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>
fix(utf8): ensure correct validation when legacy mode turned on
This depends on the included update of the prometheus/common dependency.
---------
Signed-off-by: Owen Williams <owen.williams@grafana.com>
