From 0032ce06449781d0fc0bb5a4731e87e59aa3efae Mon Sep 17 00:00:00 2001 From: Patryk Prus Date: Sun, 26 Feb 2023 21:05:27 -0500 Subject: [PATCH 001/231] Render background in images to play nicely with dark mode Signed-off-by: Patryk Prus --- README.md | 2 +- documentation/images/architecture.svg | 4 +++- documentation/images/internal_architecture.svg | 4 +++- 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 9becf71aa1..8b89bb01e5 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ The features that distinguish Prometheus from other metrics and monitoring syste ## Architecture overview -![Architecture overview](https://cdn.jsdelivr.net/gh/prometheus/prometheus@c34257d069c630685da35bcef084632ffd5d6209/documentation/images/architecture.svg) +![Architecture overview](documentation/images/architecture.svg) ## Install diff --git a/documentation/images/architecture.svg b/documentation/images/architecture.svg index df93e13cb2..4e1e85995d 100644 --- a/documentation/images/architecture.svg +++ b/documentation/images/architecture.svg @@ -1,2 +1,4 @@ + + -
pull metrics


[Not supported by viewer]
HDD / SSD
[Not supported by viewer]
Pushgateway
Pushgateway
Short-lived jobs
Short-lived jobs
Jobs / Exporters
Jobs / Exporters
Storage
Storage
Retrieval
Retrieval
PromQL
PromQL
Prometheus Server
[Not supported by viewer]
Node
<b>Node</b>
Service Discovery

Service Discovery<div><br></div>
             find 
                 targets
[Not supported by viewer]
Prometheus Server
Prometheus Server
Alertmanager
Alertmanager
push alerts         


[Not supported by viewer]
Web UI
Web UI
Grafana
Grafana
API clients
API clients
PagerDuty
PagerDuty
Email
Email
  • DNS
  • Kubernetes
  • Consul
  • ...
  • Custom integration
[Not supported by viewer]
              notify
[Not supported by viewer]
...
... \ No newline at end of file +
pull metrics


pull metrics...
HDD / SSD
HDD / SSD
Pushgateway
Pushgateway
Short-lived jobs
Short-lived jobs
Jobs / Exporters
Jobs / Exporters
Storage
Storage
Retrieval
Retrieval
PromQL
PromQL
Prometheus Server
Prometheus Server
Node
Node
Service Discovery

Service Discovery
             find 
                 targets
find...
Prometheus Server
Prometheus Server
Alertmanager
Alertmanager
push alerts         


push alerts...
Web UI
Web UI
Grafana
Grafana
API clients
API clients
PagerDuty
PagerDuty
Email
Email
  • DNS
  • Kubernetes
  • Consul
  • ...
  • Custom integration
DNSKubernetesConsul...Custom in...
              notify
              notify
...
...Text is not SVG - cannot display \ No newline at end of file diff --git a/documentation/images/internal_architecture.svg b/documentation/images/internal_architecture.svg index 5948186a7d..1242548ddb 100644 --- a/documentation/images/internal_architecture.svg +++ b/documentation/images/internal_architecture.svg @@ -1,2 +1,4 @@ + + -
Fanout Storage
Fanout Storage
read/write
series data 
[Not supported by viewer]
Local
Storage
Local<br>Storage
disk
disk
write series data
write series data
Remote Storage
Remote Storage
send
alerts
[Not supported by viewer]
append rule results
append rule results
Rule Manager
Rule Manager
Notifier
Notifier
discover Alertmanager targets
discover Alertmanager targets
Notifier Discovery
Notifier Discovery
update
Alertmanager
targets
[Not supported by viewer]
Alertmanagers
Alertmanagers
send
alerts
[Not supported by viewer]
read/write
series data
[Not supported by viewer]
scrape
metrics
[Not supported by viewer]
Scrape Manager
Scrape Manager
append 
samples 
[Not supported by viewer]
update scrape targets
[Not supported by viewer]
discover scrape targets
discover scrape targets
Scrape Discovery
Scrape Discovery
query
query
PromQL
[Not supported by viewer]
read series data
read series data
Remote Read Endpoints
Remote Read Endpoints<br>
Remote Write Endpoints
Remote Write Endpoints
Targets
Targets
Service Discovery
Service Discovery
Web API/UI
Web API/UI
query
query
PromQL
[Not supported by viewer]
Reload Handler
Reload Handler
Termination
Handler
Termination<br>Handler<br>
(most other
components)
[Not supported by viewer]
reload
reload
terminate
terminate<br>
view/control
view/control
view/control
view/control
Web Clients
Web Clients
Prometheus Server
<b>Prometheus Server</b> \ No newline at end of file +
Fanout Storage
Fanout Storage
read/write
series data 
read/write...
Local
Storage
Local...
disk
disk
write series data
write series data
Remote Storage
Remote Storage
send
alerts
send...
append rule results
append rule results
Rule Manager
Rule Manager
Notifier
Notifier
discover Alertmanager targets
discover Alertmanager targets
Notifier Discovery
Notifier Disco...
update
Alertmanager
targets
update...
Alertmanagers
Alertmanagers
send
alerts
send...
read/write
series data
read/write...
scrape
metrics
scrape...
Scrape Manager
Scrape Manager
append 
samples 
append...
update scrape targets
update scrape targets
discover scrape targets
discover scrape targets
Scrape Discovery
Scrape Discove...
query
query
PromQL
PromQL
read series data
read series data
Remote Read Endpoints
Remote Read En...
Remote Write Endpoints
Remote Write E...
Targets
Targets
Service Discovery
Service Discov...
Web API/UI
Web API/UI
query
query
PromQL
PromQL
Reload Handler
Reload Handler
Termination
Handler
Termination...
(most other
components)
(most other...
reload
reload
terminate
termin...
view/control
view/c...
view/control
view/control
Web Clients
Web Clients
Prometheus Server
Prometheus ServerText is not SVG - cannot display \ No newline at end of file From 504a16d135e28c8e4bb88ac9275a55382bc52959 Mon Sep 17 00:00:00 2001 From: SuperQ Date: Thu, 9 Mar 2023 14:41:24 +0100 Subject: [PATCH 002/231] Update Go version Update build/test to use Go 1.20. Signed-off-by: SuperQ --- .github/workflows/ci.yml | 6 +++--- .github/workflows/codeql-analysis.yml | 2 +- .promu.yml | 2 +- go.mod | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d6d79b6fc3..ceb374c8c5 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -10,7 +10,7 @@ jobs: # Whenever the Go version is updated here, .promu.yml # should also be updated. container: - image: quay.io/prometheus/golang-builder:1.19-base + image: quay.io/prometheus/golang-builder:1.20-base steps: - uses: actions/checkout@v3 - uses: prometheus/promci@v0.0.2 @@ -31,7 +31,7 @@ jobs: # Whenever the Go version is updated here, .promu.yml # should also be updated. container: - image: quay.io/prometheus/golang-builder:1.19-base + image: quay.io/prometheus/golang-builder:1.20-base steps: - uses: actions/checkout@v3 @@ -54,7 +54,7 @@ jobs: - uses: actions/checkout@v3 - uses: actions/setup-go@v3 with: - go-version: '>=1.19 <1.20' + go-version: '>=1.20 <1.21' - run: | $TestTargets = go list ./... | Where-Object { $_ -NotMatch "(github.com/prometheus/prometheus/discovery.*|github.com/prometheus/prometheus/config|github.com/prometheus/prometheus/web)"} go test $TestTargets -vet=off -v diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/codeql-analysis.yml index 01075f0c22..43c9778575 100644 --- a/.github/workflows/codeql-analysis.yml +++ b/.github/workflows/codeql-analysis.yml @@ -23,7 +23,7 @@ jobs: uses: actions/checkout@v3 - uses: actions/setup-go@v3 with: - go-version: '>=1.19 <1.20' + go-version: '>=1.20 <1.21' - name: Initialize CodeQL uses: github/codeql-action/init@v2 diff --git a/.promu.yml b/.promu.yml index 233295f852..ef69c35c8e 100644 --- a/.promu.yml +++ b/.promu.yml @@ -1,7 +1,7 @@ go: # Whenever the Go version is updated here, # .circle/config.yml should also be updated. - version: 1.19 + version: 1.20 repository: path: github.com/prometheus/prometheus build: diff --git a/go.mod b/go.mod index 76b6cb0625..02c4980aee 100644 --- a/go.mod +++ b/go.mod @@ -1,6 +1,6 @@ module github.com/prometheus/prometheus -go 1.18 +go 1.19 require ( github.com/Azure/azure-sdk-for-go v65.0.0+incompatible From c16b6a0185feea8b65911fe370808c823b90a93d Mon Sep 17 00:00:00 2001 From: Justin Lei Date: Tue, 7 Mar 2023 12:21:55 -0800 Subject: [PATCH 003/231] Handle native histograms in remote read Signed-off-by: Justin Lei --- prompb/types.pb.go | 152 ++++++++++++++-------------- prompb/types.proto | 7 +- promql/test.go | 31 ++++-- storage/remote/codec.go | 36 +++++-- storage/remote/read_handler_test.go | 73 +++++++++++-- storage/remote/read_test.go | 4 +- 6 files changed, 201 insertions(+), 102 deletions(-) diff --git a/prompb/types.pb.go b/prompb/types.pb.go index e78e48809a..125f868e97 100644 --- a/prompb/types.pb.go +++ b/prompb/types.pb.go @@ -134,21 +134,24 @@ func (LabelMatcher_Type) EnumDescriptor() ([]byte, []int) { type Chunk_Encoding int32 const ( - Chunk_UNKNOWN Chunk_Encoding = 0 - Chunk_XOR Chunk_Encoding = 1 - Chunk_HISTOGRAM Chunk_Encoding = 2 + Chunk_UNKNOWN Chunk_Encoding = 0 + Chunk_XOR Chunk_Encoding = 1 + Chunk_HISTOGRAM Chunk_Encoding = 2 + Chunk_FLOAT_HISTOGRAM Chunk_Encoding = 3 ) var Chunk_Encoding_name = map[int32]string{ 0: "UNKNOWN", 1: "XOR", 2: "HISTOGRAM", + 3: "FLOAT_HISTOGRAM", } var Chunk_Encoding_value = map[string]int32{ - "UNKNOWN": 0, - "XOR": 1, - "HISTOGRAM": 2, + "UNKNOWN": 0, + "XOR": 1, + "HISTOGRAM": 2, + "FLOAT_HISTOGRAM": 3, } func (x Chunk_Encoding) String() string { @@ -1143,75 +1146,76 @@ func init() { func init() { proto.RegisterFile("types.proto", fileDescriptor_d938547f84707355) } var fileDescriptor_d938547f84707355 = []byte{ - // 1081 bytes of a gzipped FileDescriptorProto + // 1092 bytes of a gzipped FileDescriptorProto 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x9c, 0x56, 0xdb, 0x6e, 0xdb, 0x46, - 0x13, 0x36, 0x49, 0x89, 0x12, 0x47, 0x87, 0xd0, 0x0b, 0x27, 0x3f, 0xff, 0xa0, 0x71, 0x54, 0x02, - 0x69, 0x85, 0xa2, 0x90, 0x91, 0xb4, 0x17, 0x0d, 0x1a, 0x14, 0xb0, 0x5d, 0xf9, 0x80, 0x46, 0x12, - 0xb2, 0x92, 0xd1, 0xa6, 0x37, 0xc2, 0x5a, 0x5a, 0x4b, 0x44, 0x78, 0x2a, 0x77, 0x15, 0x58, 0x7d, - 0x8f, 0xde, 0xf5, 0x25, 0x7a, 0xdf, 0x07, 0x08, 0xd0, 0x9b, 0x3e, 0x41, 0x51, 0xf8, 0xaa, 0x8f, - 0x51, 0xec, 0x90, 0x14, 0xa9, 0x38, 0x05, 0x9a, 0xde, 0xed, 0x7c, 0xf3, 0xcd, 0xec, 0xc7, 0xdd, - 0x99, 0x59, 0x42, 0x43, 0xae, 0x63, 0x2e, 0x7a, 0x71, 0x12, 0xc9, 0x88, 0x40, 0x9c, 0x44, 0x01, - 0x97, 0x4b, 0xbe, 0x12, 0xf7, 0xf7, 0x16, 0xd1, 0x22, 0x42, 0xf8, 0x40, 0xad, 0x52, 0x86, 0xfb, - 0xb3, 0x0e, 0xed, 0x01, 0x97, 0x89, 0x37, 0x1b, 0x70, 0xc9, 0xe6, 0x4c, 0x32, 0xf2, 0x14, 0x2a, - 0x2a, 0x87, 0xa3, 0x75, 0xb4, 0x6e, 0xfb, 0xc9, 0xa3, 0x5e, 0x91, 0xa3, 0xb7, 0xcd, 0xcc, 0xcc, - 0xc9, 0x3a, 0xe6, 0x14, 0x43, 0xc8, 0xa7, 0x40, 0x02, 0xc4, 0xa6, 0x57, 0x2c, 0xf0, 0xfc, 0xf5, - 0x34, 0x64, 0x01, 0x77, 0xf4, 0x8e, 0xd6, 0xb5, 0xa8, 0x9d, 0x7a, 0x4e, 0xd0, 0x31, 0x64, 0x01, - 0x27, 0x04, 0x2a, 0x4b, 0xee, 0xc7, 0x4e, 0x05, 0xfd, 0xb8, 0x56, 0xd8, 0x2a, 0xf4, 0xa4, 0x53, - 0x4d, 0x31, 0xb5, 0x76, 0xd7, 0x00, 0xc5, 0x4e, 0xa4, 0x01, 0xb5, 0x8b, 0xe1, 0x37, 0xc3, 0xd1, - 0xb7, 0x43, 0x7b, 0x47, 0x19, 0xc7, 0xa3, 0x8b, 0xe1, 0xa4, 0x4f, 0x6d, 0x8d, 0x58, 0x50, 0x3d, - 0x3d, 0xbc, 0x38, 0xed, 0xdb, 0x3a, 0x69, 0x81, 0x75, 0x76, 0x3e, 0x9e, 0x8c, 0x4e, 0xe9, 0xe1, - 0xc0, 0x36, 0x08, 0x81, 0x36, 0x7a, 0x0a, 0xac, 0xa2, 0x42, 0xc7, 0x17, 0x83, 0xc1, 0x21, 0x7d, - 0x69, 0x57, 0x49, 0x1d, 0x2a, 0xe7, 0xc3, 0x93, 0x91, 0x6d, 0x92, 0x26, 0xd4, 0xc7, 0x93, 0xc3, - 0x49, 0x7f, 0xdc, 0x9f, 0xd8, 0x35, 0xf7, 0x19, 0x98, 0x63, 0x16, 0xc4, 0x3e, 0x27, 0x7b, 0x50, - 0x7d, 0xcd, 0xfc, 0x55, 0x7a, 0x2c, 0x1a, 0x4d, 0x0d, 0xf2, 0x01, 0x58, 0xd2, 0x0b, 0xb8, 0x90, - 0x2c, 0x88, 0xf1, 0x3b, 0x0d, 0x5a, 0x00, 0x6e, 0x04, 0xf5, 0xfe, 0x35, 0x0f, 0x62, 0x9f, 0x25, - 0xe4, 0x00, 0x4c, 0x9f, 0x5d, 0x72, 0x5f, 0x38, 0x5a, 0xc7, 0xe8, 0x36, 0x9e, 0xec, 0x96, 0xcf, - 0xf5, 0xb9, 0xf2, 0x1c, 0x55, 0xde, 0xfc, 0xf1, 0x70, 0x87, 0x66, 0xb4, 0x62, 0x43, 0xfd, 0x1f, - 0x37, 0x34, 0xde, 0xde, 0xf0, 0xb7, 0x2a, 0x58, 0x67, 0x9e, 0x90, 0xd1, 0x22, 0x61, 0x01, 0x79, - 0x00, 0xd6, 0x2c, 0x5a, 0x85, 0x72, 0xea, 0x85, 0x12, 0x65, 0x57, 0xce, 0x76, 0x68, 0x1d, 0xa1, - 0xf3, 0x50, 0x92, 0x0f, 0xa1, 0x91, 0xba, 0xaf, 0xfc, 0x88, 0xc9, 0x74, 0x9b, 0xb3, 0x1d, 0x0a, - 0x08, 0x9e, 0x28, 0x8c, 0xd8, 0x60, 0x88, 0x55, 0x80, 0xfb, 0x68, 0x54, 0x2d, 0xc9, 0x3d, 0x30, - 0xc5, 0x6c, 0xc9, 0x03, 0x86, 0xb7, 0xb6, 0x4b, 0x33, 0x8b, 0x3c, 0x82, 0xf6, 0x8f, 0x3c, 0x89, - 0xa6, 0x72, 0x99, 0x70, 0xb1, 0x8c, 0xfc, 0x39, 0xde, 0xa0, 0x46, 0x5b, 0x0a, 0x9d, 0xe4, 0x20, - 0xf9, 0x28, 0xa3, 0x15, 0xba, 0x4c, 0xd4, 0xa5, 0xd1, 0xa6, 0xc2, 0x8f, 0x73, 0x6d, 0x9f, 0x80, - 0x5d, 0xe2, 0xa5, 0x02, 0x6b, 0x28, 0x50, 0xa3, 0xed, 0x0d, 0x33, 0x15, 0x79, 0x0c, 0xed, 0x90, - 0x2f, 0x98, 0xf4, 0x5e, 0xf3, 0xa9, 0x88, 0x59, 0x28, 0x9c, 0x3a, 0x9e, 0xf0, 0xbd, 0xf2, 0x09, - 0x1f, 0xad, 0x66, 0xaf, 0xb8, 0x1c, 0xc7, 0x2c, 0xcc, 0x8e, 0xb9, 0x95, 0xc7, 0x28, 0x4c, 0x90, - 0x8f, 0xe1, 0xce, 0x26, 0xc9, 0x9c, 0xfb, 0x92, 0x09, 0xc7, 0xea, 0x18, 0x5d, 0x42, 0x37, 0xb9, - 0xbf, 0x46, 0x74, 0x8b, 0x88, 0xea, 0x84, 0x03, 0x1d, 0xa3, 0xab, 0x15, 0x44, 0x94, 0x26, 0x94, - 0xac, 0x38, 0x12, 0x5e, 0x49, 0x56, 0xe3, 0xdf, 0xc8, 0xca, 0x63, 0x36, 0xb2, 0x36, 0x49, 0x32, - 0x59, 0xcd, 0x54, 0x56, 0x0e, 0x17, 0xb2, 0x36, 0xc4, 0x4c, 0x56, 0x2b, 0x95, 0x95, 0xc3, 0x99, - 0xac, 0xaf, 0x00, 0x12, 0x2e, 0xb8, 0x9c, 0x2e, 0xd5, 0xe9, 0xb7, 0xb1, 0xc7, 0x1f, 0x96, 0x25, - 0x6d, 0xea, 0xa7, 0x47, 0x15, 0xef, 0xcc, 0x0b, 0x25, 0xb5, 0x92, 0x7c, 0xb9, 0x5d, 0x80, 0x77, - 0xde, 0x2e, 0xc0, 0xcf, 0xc1, 0xda, 0x44, 0x6d, 0x77, 0x6a, 0x0d, 0x8c, 0x97, 0xfd, 0xb1, 0xad, - 0x11, 0x13, 0xf4, 0xe1, 0xc8, 0xd6, 0x8b, 0x6e, 0x35, 0x8e, 0x6a, 0x50, 0x45, 0xcd, 0x47, 0x4d, - 0x80, 0xe2, 0xda, 0xdd, 0x67, 0x00, 0xc5, 0xf9, 0xa8, 0xca, 0x8b, 0xae, 0xae, 0x04, 0x4f, 0x4b, - 0x79, 0x97, 0x66, 0x96, 0xc2, 0x7d, 0x1e, 0x2e, 0xe4, 0x12, 0x2b, 0xb8, 0x45, 0x33, 0xcb, 0xfd, - 0x4b, 0x03, 0x98, 0x78, 0x01, 0x1f, 0xf3, 0xc4, 0xe3, 0xe2, 0xfd, 0xfb, 0xef, 0x09, 0xd4, 0x04, - 0xb6, 0xbe, 0x70, 0x74, 0x8c, 0x20, 0xe5, 0x88, 0x74, 0x2a, 0x64, 0x21, 0x39, 0x91, 0x7c, 0x01, - 0x16, 0xcf, 0x1a, 0x5e, 0x38, 0x06, 0x46, 0xed, 0x95, 0xa3, 0xf2, 0x69, 0x90, 0xc5, 0x15, 0x64, - 0xf2, 0x25, 0xc0, 0x32, 0x3f, 0x78, 0xe1, 0x54, 0x30, 0xf4, 0xee, 0x3b, 0xaf, 0x25, 0x8b, 0x2d, - 0xd1, 0xdd, 0xc7, 0x50, 0xc5, 0x2f, 0x50, 0xd3, 0x13, 0x27, 0xae, 0x96, 0x4e, 0x4f, 0xb5, 0xde, - 0x9e, 0x23, 0x56, 0x36, 0x47, 0xdc, 0xa7, 0x60, 0x3e, 0x4f, 0xbf, 0xf3, 0x7d, 0x0f, 0xc6, 0xfd, - 0x49, 0x83, 0x26, 0xe2, 0x03, 0x26, 0x67, 0x4b, 0x9e, 0x90, 0xc7, 0x5b, 0x0f, 0xc6, 0x83, 0x5b, - 0xf1, 0x19, 0xaf, 0x57, 0x7a, 0x28, 0x72, 0xa1, 0xfa, 0xbb, 0x84, 0x1a, 0x65, 0xa1, 0x5d, 0xa8, - 0xe0, 0xd8, 0x37, 0x41, 0xef, 0xbf, 0x48, 0xeb, 0x68, 0xd8, 0x7f, 0x91, 0xd6, 0x11, 0x55, 0xa3, - 0x5e, 0x01, 0xb4, 0x6f, 0x1b, 0xee, 0x2f, 0x9a, 0x2a, 0x3e, 0x36, 0x57, 0xb5, 0x27, 0xc8, 0xff, - 0xa0, 0x26, 0x24, 0x8f, 0xa7, 0x81, 0x40, 0x5d, 0x06, 0x35, 0x95, 0x39, 0x10, 0x6a, 0xeb, 0xab, - 0x55, 0x38, 0xcb, 0xb7, 0x56, 0x6b, 0xf2, 0x7f, 0xa8, 0x0b, 0xc9, 0x12, 0xa9, 0xd8, 0xe9, 0x50, - 0xad, 0xa1, 0x3d, 0x10, 0xe4, 0x2e, 0x98, 0x3c, 0x9c, 0x4f, 0xf1, 0x52, 0x94, 0xa3, 0xca, 0xc3, - 0xf9, 0x40, 0x90, 0xfb, 0x50, 0x5f, 0x24, 0xd1, 0x2a, 0xf6, 0xc2, 0x85, 0x53, 0xed, 0x18, 0x5d, - 0x8b, 0x6e, 0x6c, 0xd2, 0x06, 0xfd, 0x72, 0x8d, 0x83, 0xad, 0x4e, 0xf5, 0xcb, 0xb5, 0xca, 0x9e, - 0xb0, 0x70, 0xc1, 0x55, 0x92, 0x5a, 0x9a, 0x1d, 0xed, 0x81, 0x70, 0x7f, 0xd5, 0xa0, 0x7a, 0xbc, - 0x5c, 0x85, 0xaf, 0xc8, 0x3e, 0x34, 0x02, 0x2f, 0x9c, 0xaa, 0x56, 0x2a, 0x34, 0x5b, 0x81, 0x17, - 0xaa, 0x1a, 0x1e, 0x08, 0xf4, 0xb3, 0xeb, 0x8d, 0x3f, 0x7b, 0x6b, 0x02, 0x76, 0x9d, 0xf9, 0x7b, - 0xd9, 0x25, 0x18, 0x78, 0x09, 0xf7, 0xcb, 0x97, 0x80, 0x1b, 0xf4, 0xfa, 0xe1, 0x2c, 0x9a, 0x7b, - 0xe1, 0xa2, 0xb8, 0x01, 0xf5, 0x86, 0xe3, 0x57, 0x35, 0x29, 0xae, 0xdd, 0x03, 0xa8, 0xe7, 0xac, - 0x5b, 0xcd, 0xfb, 0xdd, 0x48, 0x3d, 0xb1, 0x5b, 0xef, 0xaa, 0xee, 0xfe, 0x00, 0x2d, 0x4c, 0xce, - 0xe7, 0xff, 0xb5, 0xcb, 0x0e, 0xc0, 0x9c, 0xa9, 0x0c, 0x79, 0x93, 0xed, 0xde, 0x12, 0x9e, 0x07, - 0xa4, 0xb4, 0xa3, 0xbd, 0x37, 0x37, 0xfb, 0xda, 0xef, 0x37, 0xfb, 0xda, 0x9f, 0x37, 0xfb, 0xda, - 0xf7, 0xa6, 0x62, 0xc7, 0x97, 0x97, 0x26, 0xfe, 0xcd, 0x7c, 0xf6, 0x77, 0x00, 0x00, 0x00, 0xff, - 0xff, 0x53, 0x09, 0xe5, 0x37, 0xfe, 0x08, 0x00, 0x00, + 0x13, 0x36, 0x49, 0x89, 0x12, 0x47, 0x87, 0xd0, 0xfb, 0x3b, 0xf9, 0x59, 0xa3, 0x71, 0x54, 0x02, + 0x69, 0x85, 0xa2, 0x90, 0x11, 0xb7, 0x17, 0x0d, 0x1a, 0x14, 0xb0, 0x1d, 0xf9, 0x80, 0x5a, 0x12, + 0xb2, 0x92, 0xd1, 0xa6, 0x37, 0xc2, 0x5a, 0x5a, 0x4b, 0x44, 0xc4, 0x43, 0xb9, 0xab, 0xc0, 0xea, + 0x7b, 0xf4, 0xae, 0x2f, 0xd1, 0xb7, 0x08, 0xd0, 0x9b, 0xf6, 0x05, 0x8a, 0xc2, 0x57, 0x7d, 0x8c, + 0x62, 0x87, 0xa4, 0x48, 0xc5, 0x29, 0xd0, 0xf4, 0x6e, 0xe7, 0x9b, 0x6f, 0x76, 0x3e, 0xee, 0xce, + 0xcc, 0x12, 0x6a, 0x72, 0x15, 0x71, 0xd1, 0x89, 0xe2, 0x50, 0x86, 0x04, 0xa2, 0x38, 0xf4, 0xb9, + 0x9c, 0xf3, 0xa5, 0xd8, 0xdd, 0x99, 0x85, 0xb3, 0x10, 0xe1, 0x7d, 0xb5, 0x4a, 0x18, 0xee, 0xcf, + 0x3a, 0x34, 0x7b, 0x5c, 0xc6, 0xde, 0xa4, 0xc7, 0x25, 0x9b, 0x32, 0xc9, 0xc8, 0x53, 0x28, 0xa9, + 0x3d, 0x1c, 0xad, 0xa5, 0xb5, 0x9b, 0x07, 0x8f, 0x3b, 0xf9, 0x1e, 0x9d, 0x4d, 0x66, 0x6a, 0x8e, + 0x56, 0x11, 0xa7, 0x18, 0x42, 0x3e, 0x03, 0xe2, 0x23, 0x36, 0xbe, 0x66, 0xbe, 0xb7, 0x58, 0x8d, + 0x03, 0xe6, 0x73, 0x47, 0x6f, 0x69, 0x6d, 0x8b, 0xda, 0x89, 0xe7, 0x04, 0x1d, 0x7d, 0xe6, 0x73, + 0x42, 0xa0, 0x34, 0xe7, 0x8b, 0xc8, 0x29, 0xa1, 0x1f, 0xd7, 0x0a, 0x5b, 0x06, 0x9e, 0x74, 0xca, + 0x09, 0xa6, 0xd6, 0xee, 0x0a, 0x20, 0xcf, 0x44, 0x6a, 0x50, 0xb9, 0xec, 0x7f, 0xd3, 0x1f, 0x7c, + 0xdb, 0xb7, 0xb7, 0x94, 0x71, 0x3c, 0xb8, 0xec, 0x8f, 0xba, 0xd4, 0xd6, 0x88, 0x05, 0xe5, 0xd3, + 0xc3, 0xcb, 0xd3, 0xae, 0xad, 0x93, 0x06, 0x58, 0x67, 0xe7, 0xc3, 0xd1, 0xe0, 0x94, 0x1e, 0xf6, + 0x6c, 0x83, 0x10, 0x68, 0xa2, 0x27, 0xc7, 0x4a, 0x2a, 0x74, 0x78, 0xd9, 0xeb, 0x1d, 0xd2, 0x97, + 0x76, 0x99, 0x54, 0xa1, 0x74, 0xde, 0x3f, 0x19, 0xd8, 0x26, 0xa9, 0x43, 0x75, 0x38, 0x3a, 0x1c, + 0x75, 0x87, 0xdd, 0x91, 0x5d, 0x71, 0x9f, 0x81, 0x39, 0x64, 0x7e, 0xb4, 0xe0, 0x64, 0x07, 0xca, + 0xaf, 0xd9, 0x62, 0x99, 0x1c, 0x8b, 0x46, 0x13, 0x83, 0x7c, 0x08, 0x96, 0xf4, 0x7c, 0x2e, 0x24, + 0xf3, 0x23, 0xfc, 0x4e, 0x83, 0xe6, 0x80, 0x1b, 0x42, 0xb5, 0x7b, 0xc3, 0xfd, 0x68, 0xc1, 0x62, + 0xb2, 0x0f, 0xe6, 0x82, 0x5d, 0xf1, 0x85, 0x70, 0xb4, 0x96, 0xd1, 0xae, 0x1d, 0x6c, 0x17, 0xcf, + 0xf5, 0x42, 0x79, 0x8e, 0x4a, 0x6f, 0xfe, 0x78, 0xb4, 0x45, 0x53, 0x5a, 0x9e, 0x50, 0xff, 0xc7, + 0x84, 0xc6, 0xdb, 0x09, 0x7f, 0x2d, 0x83, 0x75, 0xe6, 0x09, 0x19, 0xce, 0x62, 0xe6, 0x93, 0x87, + 0x60, 0x4d, 0xc2, 0x65, 0x20, 0xc7, 0x5e, 0x20, 0x51, 0x76, 0xe9, 0x6c, 0x8b, 0x56, 0x11, 0x3a, + 0x0f, 0x24, 0xf9, 0x08, 0x6a, 0x89, 0xfb, 0x7a, 0x11, 0x32, 0x99, 0xa4, 0x39, 0xdb, 0xa2, 0x80, + 0xe0, 0x89, 0xc2, 0x88, 0x0d, 0x86, 0x58, 0xfa, 0x98, 0x47, 0xa3, 0x6a, 0x49, 0x1e, 0x80, 0x29, + 0x26, 0x73, 0xee, 0x33, 0xbc, 0xb5, 0x6d, 0x9a, 0x5a, 0xe4, 0x31, 0x34, 0x7f, 0xe4, 0x71, 0x38, + 0x96, 0xf3, 0x98, 0x8b, 0x79, 0xb8, 0x98, 0xe2, 0x0d, 0x6a, 0xb4, 0xa1, 0xd0, 0x51, 0x06, 0x92, + 0x8f, 0x53, 0x5a, 0xae, 0xcb, 0x44, 0x5d, 0x1a, 0xad, 0x2b, 0xfc, 0x38, 0xd3, 0xf6, 0x29, 0xd8, + 0x05, 0x5e, 0x22, 0xb0, 0x82, 0x02, 0x35, 0xda, 0x5c, 0x33, 0x13, 0x91, 0xc7, 0xd0, 0x0c, 0xf8, + 0x8c, 0x49, 0xef, 0x35, 0x1f, 0x8b, 0x88, 0x05, 0xc2, 0xa9, 0xe2, 0x09, 0x3f, 0x28, 0x9e, 0xf0, + 0xd1, 0x72, 0xf2, 0x8a, 0xcb, 0x61, 0xc4, 0x82, 0xf4, 0x98, 0x1b, 0x59, 0x8c, 0xc2, 0x04, 0xf9, + 0x04, 0xee, 0xad, 0x37, 0x99, 0xf2, 0x85, 0x64, 0xc2, 0xb1, 0x5a, 0x46, 0x9b, 0xd0, 0xf5, 0xde, + 0xcf, 0x11, 0xdd, 0x20, 0xa2, 0x3a, 0xe1, 0x40, 0xcb, 0x68, 0x6b, 0x39, 0x11, 0xa5, 0x09, 0x25, + 0x2b, 0x0a, 0x85, 0x57, 0x90, 0x55, 0xfb, 0x37, 0xb2, 0xb2, 0x98, 0xb5, 0xac, 0xf5, 0x26, 0xa9, + 0xac, 0x7a, 0x22, 0x2b, 0x83, 0x73, 0x59, 0x6b, 0x62, 0x2a, 0xab, 0x91, 0xc8, 0xca, 0xe0, 0x54, + 0xd6, 0xd7, 0x00, 0x31, 0x17, 0x5c, 0x8e, 0xe7, 0xea, 0xf4, 0x9b, 0xd8, 0xe3, 0x8f, 0x8a, 0x92, + 0xd6, 0xf5, 0xd3, 0xa1, 0x8a, 0x77, 0xe6, 0x05, 0x92, 0x5a, 0x71, 0xb6, 0xdc, 0x2c, 0xc0, 0x7b, + 0x6f, 0x17, 0xe0, 0x17, 0x60, 0xad, 0xa3, 0x36, 0x3b, 0xb5, 0x02, 0xc6, 0xcb, 0xee, 0xd0, 0xd6, + 0x88, 0x09, 0x7a, 0x7f, 0x60, 0xeb, 0x79, 0xb7, 0x1a, 0x47, 0x15, 0x28, 0xa3, 0xe6, 0xa3, 0x3a, + 0x40, 0x7e, 0xed, 0xee, 0x33, 0x80, 0xfc, 0x7c, 0x54, 0xe5, 0x85, 0xd7, 0xd7, 0x82, 0x27, 0xa5, + 0xbc, 0x4d, 0x53, 0x4b, 0xe1, 0x0b, 0x1e, 0xcc, 0xe4, 0x1c, 0x2b, 0xb8, 0x41, 0x53, 0xcb, 0xfd, + 0x4b, 0x03, 0x18, 0x79, 0x3e, 0x1f, 0xf2, 0xd8, 0xe3, 0xe2, 0xfd, 0xfb, 0xef, 0x00, 0x2a, 0x02, + 0x5b, 0x5f, 0x38, 0x3a, 0x46, 0x90, 0x62, 0x44, 0x32, 0x15, 0xd2, 0x90, 0x8c, 0x48, 0xbe, 0x04, + 0x8b, 0xa7, 0x0d, 0x2f, 0x1c, 0x03, 0xa3, 0x76, 0x8a, 0x51, 0xd9, 0x34, 0x48, 0xe3, 0x72, 0x32, + 0xf9, 0x0a, 0x60, 0x9e, 0x1d, 0xbc, 0x70, 0x4a, 0x18, 0x7a, 0xff, 0x9d, 0xd7, 0x92, 0xc6, 0x16, + 0xe8, 0xee, 0x13, 0x28, 0xe3, 0x17, 0xa8, 0xe9, 0x89, 0x13, 0x57, 0x4b, 0xa6, 0xa7, 0x5a, 0x6f, + 0xce, 0x11, 0x2b, 0x9d, 0x23, 0xee, 0x53, 0x30, 0x2f, 0x92, 0xef, 0x7c, 0xdf, 0x83, 0x71, 0x7f, + 0xd2, 0xa0, 0x8e, 0x78, 0x8f, 0xc9, 0xc9, 0x9c, 0xc7, 0xe4, 0xc9, 0xc6, 0x83, 0xf1, 0xf0, 0x4e, + 0x7c, 0xca, 0xeb, 0x14, 0x1e, 0x8a, 0x4c, 0xa8, 0xfe, 0x2e, 0xa1, 0x46, 0x51, 0x68, 0x1b, 0x4a, + 0x38, 0xf6, 0x4d, 0xd0, 0xbb, 0x2f, 0x92, 0x3a, 0xea, 0x77, 0x5f, 0x24, 0x75, 0x44, 0xd5, 0xa8, + 0x57, 0x00, 0xed, 0xda, 0x86, 0xfb, 0x8b, 0xa6, 0x8a, 0x8f, 0x4d, 0x55, 0xed, 0x09, 0xf2, 0x7f, + 0xa8, 0x08, 0xc9, 0xa3, 0xb1, 0x2f, 0x50, 0x97, 0x41, 0x4d, 0x65, 0xf6, 0x84, 0x4a, 0x7d, 0xbd, + 0x0c, 0x26, 0x59, 0x6a, 0xb5, 0x26, 0x1f, 0x40, 0x55, 0x48, 0x16, 0x4b, 0xc5, 0x4e, 0x86, 0x6a, + 0x05, 0xed, 0x9e, 0x20, 0xf7, 0xc1, 0xe4, 0xc1, 0x74, 0x8c, 0x97, 0xa2, 0x1c, 0x65, 0x1e, 0x4c, + 0x7b, 0x82, 0xec, 0x42, 0x75, 0x16, 0x87, 0xcb, 0xc8, 0x0b, 0x66, 0x4e, 0xb9, 0x65, 0xb4, 0x2d, + 0xba, 0xb6, 0x49, 0x13, 0xf4, 0xab, 0x15, 0x0e, 0xb6, 0x2a, 0xd5, 0xaf, 0x56, 0x6a, 0xf7, 0x98, + 0x05, 0x33, 0xae, 0x36, 0xa9, 0x24, 0xbb, 0xa3, 0xdd, 0x13, 0xee, 0xef, 0x1a, 0x94, 0x8f, 0xe7, + 0xcb, 0xe0, 0x15, 0xd9, 0x83, 0x9a, 0xef, 0x05, 0x63, 0xd5, 0x4a, 0xb9, 0x66, 0xcb, 0xf7, 0x02, + 0x55, 0xc3, 0x3d, 0x81, 0x7e, 0x76, 0xb3, 0xf6, 0xa7, 0x6f, 0x8d, 0xcf, 0x6e, 0x52, 0x7f, 0x27, + 0xbd, 0x04, 0x03, 0x2f, 0x61, 0xb7, 0x78, 0x09, 0x98, 0xa0, 0xd3, 0x0d, 0x26, 0xe1, 0xd4, 0x0b, + 0x66, 0xf9, 0x0d, 0xa8, 0x37, 0x1c, 0xbf, 0xaa, 0x4e, 0x71, 0xed, 0x3e, 0x87, 0x6a, 0xc6, 0xba, + 0xd3, 0xbc, 0xdf, 0x0d, 0xd4, 0x13, 0xbb, 0xf1, 0xae, 0xea, 0xe4, 0x7f, 0x70, 0xef, 0xe4, 0x62, + 0x70, 0x38, 0x1a, 0x17, 0x1e, 0x5b, 0xf7, 0x07, 0x68, 0x60, 0x46, 0x3e, 0xfd, 0xaf, 0xad, 0xb7, + 0x0f, 0xe6, 0x44, 0xed, 0x90, 0x75, 0xde, 0xf6, 0x9d, 0xaf, 0xc9, 0x02, 0x12, 0xda, 0xd1, 0xce, + 0x9b, 0xdb, 0x3d, 0xed, 0xb7, 0xdb, 0x3d, 0xed, 0xcf, 0xdb, 0x3d, 0xed, 0x7b, 0x53, 0xb1, 0xa3, + 0xab, 0x2b, 0x13, 0x7f, 0x71, 0x3e, 0xff, 0x3b, 0x00, 0x00, 0xff, 0xff, 0xfb, 0x5f, 0xf2, 0x4d, + 0x13, 0x09, 0x00, 0x00, } func (m *MetricMetadata) Marshal() (dAtA []byte, err error) { diff --git a/prompb/types.proto b/prompb/types.proto index 57216b81d9..aa322515c3 100644 --- a/prompb/types.proto +++ b/prompb/types.proto @@ -169,9 +169,10 @@ message Chunk { // We require this to match chunkenc.Encoding. enum Encoding { - UNKNOWN = 0; - XOR = 1; - HISTOGRAM = 2; + UNKNOWN = 0; + XOR = 1; + HISTOGRAM = 2; + FLOAT_HISTOGRAM = 3; } Encoding type = 3; bytes data = 4; diff --git a/promql/test.go b/promql/test.go index 78cc1e9fbb..f47269aeca 100644 --- a/promql/test.go +++ b/promql/test.go @@ -33,6 +33,7 @@ import ( "github.com/prometheus/prometheus/promql/parser" "github.com/prometheus/prometheus/storage" "github.com/prometheus/prometheus/tsdb" + "github.com/prometheus/prometheus/tsdb/tsdbutil" "github.com/prometheus/prometheus/util/teststorage" "github.com/prometheus/prometheus/util/testutil" ) @@ -302,14 +303,26 @@ func (cmd loadCmd) String() string { func (cmd *loadCmd) set(m labels.Labels, vals ...parser.SequenceValue) { h := m.Hash() + metricName := m.Get("__name__") + isHistogram := strings.Contains(metricName, "_histogram_") + samples := make([]Point, 0, len(vals)) ts := testStartTime for _, v := range vals { if !v.Omitted { - samples = append(samples, Point{ - T: ts.UnixNano() / int64(time.Millisecond/time.Nanosecond), - V: v.Value, - }) + t := ts.UnixNano() / int64(time.Millisecond/time.Nanosecond) + + if isHistogram { + samples = append(samples, Point{ + T: t, + H: tsdbutil.GenerateTestFloatHistogram(int(v.Value)), + }) + } else { + samples = append(samples, Point{ + T: t, + V: v.Value, + }) + } } ts = ts.Add(cmd.gap) } @@ -323,8 +336,14 @@ func (cmd *loadCmd) append(a storage.Appender) error { m := cmd.metrics[h] for _, s := range smpls { - if _, err := a.Append(0, m, s.T, s.V); err != nil { - return err + if s.H == nil { + if _, err := a.Append(0, m, s.T, s.V); err != nil { + return err + } + } else { + if _, err := a.AppendHistogram(0, m, s.T, nil, s.H); err != nil { + return err + } } } } diff --git a/storage/remote/codec.go b/storage/remote/codec.go index e3ef58c351..ba802bccb3 100644 --- a/storage/remote/codec.go +++ b/storage/remote/codec.go @@ -120,10 +120,13 @@ func ToQueryResult(ss storage.SeriesSet, sampleLimit int) (*prompb.QueryResult, for ss.Next() { series := ss.At() iter = series.Iterator(iter) - samples := []prompb.Sample{} - for iter.Next() == chunkenc.ValFloat { - // TODO(beorn7): Add Histogram support. + var ( + samples []prompb.Sample + histograms []prompb.Histogram + ) + + for valType := iter.Next(); valType != chunkenc.ValNone; valType = iter.Next() { numSamples++ if sampleLimit > 0 && numSamples > sampleLimit { return nil, ss.Warnings(), HTTPError{ @@ -131,19 +134,32 @@ func ToQueryResult(ss storage.SeriesSet, sampleLimit int) (*prompb.QueryResult, status: http.StatusBadRequest, } } - ts, val := iter.At() - samples = append(samples, prompb.Sample{ - Timestamp: ts, - Value: val, - }) + + switch valType { + case chunkenc.ValFloat: + ts, val := iter.At() + samples = append(samples, prompb.Sample{ + Timestamp: ts, + Value: val, + }) + case chunkenc.ValHistogram: + ts, h := iter.AtHistogram() + histograms = append(histograms, HistogramToHistogramProto(ts, h)) + case chunkenc.ValFloatHistogram: + ts, fh := iter.AtFloatHistogram() + histograms = append(histograms, FloatHistogramToHistogramProto(ts, fh)) + default: + return nil, ss.Warnings(), fmt.Errorf("unrecognized value type: %s", valType) + } } if err := iter.Err(); err != nil { return nil, ss.Warnings(), err } resp.Timeseries = append(resp.Timeseries, &prompb.TimeSeries{ - Labels: labelsToLabelsProto(series.Labels(), nil), - Samples: samples, + Labels: labelsToLabelsProto(series.Labels(), nil), + Samples: samples, + Histograms: histograms, }) } return resp, ss.Warnings(), ss.Err() diff --git a/storage/remote/read_handler_test.go b/storage/remote/read_handler_test.go index b5b717455c..2ac9cc32b4 100644 --- a/storage/remote/read_handler_test.go +++ b/storage/remote/read_handler_test.go @@ -30,12 +30,14 @@ import ( "github.com/prometheus/prometheus/prompb" "github.com/prometheus/prometheus/promql" "github.com/prometheus/prometheus/storage" + "github.com/prometheus/prometheus/tsdb/tsdbutil" ) func TestSampledReadEndpoint(t *testing.T) { suite, err := promql.NewTest(t, ` load 1m test_metric1{foo="bar",baz="qux"} 1 + test_histogram_metric1{foo="bar",baz="qux"} 1 `) require.NoError(t, err) @@ -60,10 +62,16 @@ func TestSampledReadEndpoint(t *testing.T) { matcher2, err := labels.NewMatcher(labels.MatchEqual, "d", "e") require.NoError(t, err) - query, err := ToQuery(0, 1, []*labels.Matcher{matcher1, matcher2}, &storage.SelectHints{Step: 0, Func: "avg"}) + matcher3, err := labels.NewMatcher(labels.MatchEqual, "__name__", "test_histogram_metric1") require.NoError(t, err) - req := &prompb.ReadRequest{Queries: []*prompb.Query{query}} + query1, err := ToQuery(0, 1, []*labels.Matcher{matcher1, matcher2}, &storage.SelectHints{Step: 0, Func: "avg"}) + require.NoError(t, err) + + query2, err := ToQuery(0, 1, []*labels.Matcher{matcher3, matcher2}, &storage.SelectHints{Step: 0, Func: "avg"}) + require.NoError(t, err) + + req := &prompb.ReadRequest{Queries: []*prompb.Query{query1, query2}} data, err := proto.Marshal(req) require.NoError(t, err) @@ -90,7 +98,7 @@ func TestSampledReadEndpoint(t *testing.T) { err = proto.Unmarshal(uncompressed, &resp) require.NoError(t, err) - require.Equal(t, 1, len(resp.Results), "Expected 1 result.") + require.Equal(t, 2, len(resp.Results), "Expected 2 results.") require.Equal(t, &prompb.QueryResult{ Timeseries: []*prompb.TimeSeries{ @@ -106,6 +114,23 @@ func TestSampledReadEndpoint(t *testing.T) { }, }, }, resp.Results[0]) + + require.Equal(t, &prompb.QueryResult{ + Timeseries: []*prompb.TimeSeries{ + { + Labels: []prompb.Label{ + {Name: "__name__", Value: "test_histogram_metric1"}, + {Name: "b", Value: "c"}, + {Name: "baz", Value: "qux"}, + {Name: "d", Value: "e"}, + {Name: "foo", Value: "bar"}, + }, + Histograms: []prompb.Histogram{ + FloatHistogramToHistogramProto(0, tsdbutil.GenerateTestFloatHistogram(1)), + }, + }, + }, + }, resp.Results[1]) } func BenchmarkStreamReadEndpoint(b *testing.B) { @@ -179,11 +204,13 @@ func TestStreamReadEndpoint(t *testing.T) { // First with 120 samples. We expect 1 frame with 1 chunk. // Second with 121 samples, We expect 1 frame with 2 chunks. // Third with 241 samples. We expect 1 frame with 2 chunks, and 1 frame with 1 chunk for the same series due to bytes limit. + // Fourth with 120 histogram samples. We expect 1 frame with 1 chunk. suite, err := promql.NewTest(t, ` load 1m test_metric1{foo="bar1",baz="qux"} 0+100x119 - test_metric1{foo="bar2",baz="qux"} 0+100x120 - test_metric1{foo="bar3",baz="qux"} 0+100x240 + test_metric1{foo="bar2",baz="qux"} 0+100x120 + test_metric1{foo="bar3",baz="qux"} 0+100x240 + test_histogram_metric1{foo="bar4",baz="qux"} 0+1x119 `) require.NoError(t, err) @@ -214,6 +241,9 @@ func TestStreamReadEndpoint(t *testing.T) { matcher3, err := labels.NewMatcher(labels.MatchEqual, "foo", "bar1") require.NoError(t, err) + matcher4, err := labels.NewMatcher(labels.MatchEqual, "__name__", "test_histogram_metric1") + require.NoError(t, err) + query1, err := ToQuery(0, 14400001, []*labels.Matcher{matcher1, matcher2}, &storage.SelectHints{ Step: 1, Func: "avg", @@ -230,8 +260,16 @@ func TestStreamReadEndpoint(t *testing.T) { }) require.NoError(t, err) + query3, err := ToQuery(0, 14400001, []*labels.Matcher{matcher4}, &storage.SelectHints{ + Step: 1, + Func: "avg", + Start: 0, + End: 14400001, + }) + require.NoError(t, err) + req := &prompb.ReadRequest{ - Queries: []*prompb.Query{query1, query2}, + Queries: []*prompb.Query{query1, query2, query3}, AcceptedResponseTypes: []prompb.ReadRequest_ResponseType{prompb.ReadRequest_STREAMED_XOR_CHUNKS}, } data, err := proto.Marshal(req) @@ -261,7 +299,7 @@ func TestStreamReadEndpoint(t *testing.T) { results = append(results, res) } - require.Equal(t, 5, len(results), "Expected 5 results.") + require.Equal(t, 6, len(results), "Expected 6 results.") require.Equal(t, []*prompb.ChunkedReadResponse{ { @@ -378,5 +416,26 @@ func TestStreamReadEndpoint(t *testing.T) { }, QueryIndex: 1, }, + { + ChunkedSeries: []*prompb.ChunkedSeries{ + { + Labels: []prompb.Label{ + {Name: "__name__", Value: "test_histogram_metric1"}, + {Name: "b", Value: "c"}, + {Name: "baz", Value: "qux"}, + {Name: "d", Value: "e"}, + {Name: "foo", Value: "bar4"}, + }, + Chunks: []prompb.Chunk{ + { + Type: prompb.Chunk_FLOAT_HISTOGRAM, + MaxTimeMs: 7140000, + Data: []byte("\x00x\x00\xff?PbM\xd2\xf1\xa9\xfc\x8c\xa4\x94e$\xa2@$\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00\x00\x00\x00@2ffffff?\xf0\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00\x00\x00\x00?\xf0\x00\x00\x00\x00\x00\x00?\xf0\x00\x00\x00\x00\x00\x00?\xf0\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00\x00\x00\x00?\xf0\x00\x00\x00\x00\x00\x00?\xf0\x00\x00\x00\x00\x00\x00\xf8\xea`\xd6%\xec\a\xa4?\x84\xbf\xff\xb0\x1e\x12\xff\xfe\x12\xff\xfe\x12\xff\xfe\xc0xK\xff\xf8K\xff\xe95\x85\xec\xd2\x7f\xff\xff\xff\xff\xff\xf6\x03\xd6\x17\xb0\x1e\xc0{\x01\xeb\v\xd8\x0f`6\x91\xfd\xed\a\xaf\\\xff\xff\xff\xff\xff\xff\xeb\v\xda\x0fX^\xb0\xbda{A\xeb\v\xd6\x16\x82l\v\x8a\xcc\xcc\xcc\xcc\xccʹ\x1e\xc0\xbd\xa0\xf6\x83\xda\x0f`^\xd0{A\xa1\x932fffffg`\\\xec\v\xd8\x17\xb0.v\x05\xec\vA5\t\xfa\x87\xef:\x84\xf9\x99\xd4'̵\x8d\xde\xe0{\xb2\x9f\xff\xff\xff\xff\xff\xf5\t\xfb\x81\xea\x13\xf5\t\xfa\x84\xfd\xc0\xf5\t\xfa\x84\xf4&н\xb9\xedUUUUU]\xc0\xf6\x85\xee\a\xb8\x1e\xe0{B\xf7\x03\xdc\r\x193\xb333333\xda\x17;B\xf6\x85\xed\v\x9d\xa1{BЛ\x03\xfb2\xe4\xcc\xcc\xcc\xcc\xcc\xe7`~fv\a\xe6S\x91ݕ\xaa\xaa\xaa\xaa\xaa\xab\xb0?\x1d\x81\xfd\x81\xfd\x81\xf8\xec\x0f\xec\x0f\xa1'\xb7<\xff\xff\xff\xff\xff\xff\x19\xc61\x9cb\x8c\x8e\xbd{\xff\xff\xff\xff\xff\xff8\xces\x8c\xe6\x84\xd6'\xc1Y\x99\x99\x99\x99\x99\x8e\xb1>1\x8e\xb1>1j#\xefx8d\xcc\xcc\xcc\xcc\xcc\xda\xc4\xfd\xe0\xf5\x89\xfa\xc4\xfdb~\xf0z\xc4\xfdbz\x04\xdc\x17\a\xaa\xaa\xaa\xaa\xaa\xabx=\xc1{\xc1\xef\a\xbc\x1e\xe0\xbd\xe0\xf7\x83A\x93\x1c\xff\xff\xff\xff\xff\xffp\\\xee\v\xdc\x17\xb8.w\x05\xee\v@\x9bC\xf0Z\xaa\xaa\xaa\xaa\xaa\xa7h~fv\x87\xe6P\xe4al\xcc\xcc\xcc\xcc\xcc\xed\x0f\xc7h\x7fh\x7fh~;C\xfbC\xe8\x12sə\x99\x99\x99\x99\xa38\xc63\x8cPd`\xb5UUUUUN3\x9c\xe39\xa0M\x82|3\xff\xff\xff\xff\xff\xf8\xec\x13\xe3\x18\xec\x13\xe3\x14y\f\x1e\xaa\xaa\xaa\xaa\xaa\xad\x82|;\x04\xfd\x82~\xc1>\x1d\x82~\xc1=\x02G\x1c\x99\x99\x99\x99\x99\x9a\x18\xe1\x86\x18\xe1\x85\x06C\x05ffffff8c\x8e8c\x8d\x02O\v\xaa\xaa\xaa\xaa\xaa\xaa\x19\xe1\x86\x19\xe1\x85\x0eC\xa3\x8f\xf1UUUUUY\xe1\x9ey\xe1\x9et\t\x1c\x01j\xaa\xaa\xaa\xaa\xab\fp\xc3\fp\u0083!\x80{33333#\x868\xe3\x868\xd0&\x91\xff\xc0\x12fffffp\xe9\x1f\xfc0ä\x7f\xf0\xc2\xd6G\xdf\x00p\x1d\xaa\xaa\xaa\xaa\xaa\xae\x91\xff\xf0\a\xa4\x7f\xfaG\xff\xa4\x7f\xfc\x01\xe9\x1f\xfe\x91\xff\xa0M\xe1p\x04\xff\xff\xff\xff\xff\xff\x00{\xc2\xf8\x03\xe0\x0f\x80=\xe1|\x01\xf0\x06\x83&\x01uUUUUU\xde\x17;\xc2\xf7\x85\xef\v\x9d\xe1{\xc2\xd0&\xe0\xfc\x0fY\x99\x99\x99\x99\x99;\x83\xf33\xb8?2\x87#\x00I\x99\x99\x99\x99\x99\xee\x0f\xc7p\x7fp\x7fp~;\x83\xfb\x83\xe8\x12p\x0f\xaa\xaa\xaa\xaa\xaa\xacg\x18\xc6q\x8a\f\x8c\x01?\xff\xff\xff\xff\xff8\xces\x8c\xe6\x816\x89\xf0\x1d\xaa\xaa\xaa\xaa\xaa\xacv\x89\xf1\x8cv\x89\xf1\x8a<\x86\x01l\xcc\xcc\xcc\xcc\xcc\xda'ôO\xda'\xed\x13\xe1\xda'\xed\x13\xd0$p\x04\x99\x99\x99\x99\x99\x9c1\xc3\f1\xc3\n\f\x86\x0f\xb5UUUUU\x8e\x18\xe3\x8e\x18\xe3@\x93\xc0\x13\xff\xff\xff\xff\xff\xf0\xcf\f0\xcf\f(r\x18\a\xd5UUUUVxg\x9exg\x9d\x02G\x00I\x99\x99\x99\x99\x99\xc3\x1c0\xc3\x1c0\xa0\xc8`:\xcc\xcc\xcc\xcc\xcc\xc8\xe1\x8e8\xe1\x8e4\t\xb0_\xc0.\xaa\xaa\xaa\xaa\xaa\xb0\xec\x17\xf0\xc3\x0e\xc1\x7f\f)\xf2\f\x01?\xff\xff\xff\xff\xff\xb0_\xc1\xd8/\xf6\v\xfd\x82\xfe\x0e\xc1\x7f\xb0_\xa0Hp=\xaa\xaa\xaa\xaa\xaa\xac\x18p`\xc1\x87\x06\n\f\x83\x00I\x99\x99\x99\x99\x99Ã\x0e\x1c80\xe1\xa0H\xf0\x0ffffffd\x18\xf0`\xc1\x8f\x06\n\x1c\x83\x00Z\xaa\xaa\xaa\xaa\xaaǃ\x1e|\xf83\xe7\xa0Hp\x03\xd5UUUUT\x18p`\xc1\x87\x06\n\f\x83\x00g\xff\xff\xff\xff\xffÃ\x0e\x1c80\xe1\xa0H\xf0\x02\xd5UUUUT\x18\xf0`\xc1\x8f\x06\n\x1c\x83\x00\xdb33333G\x83\x1e \xf8\x83\xe0\x17\xc4\x1f\x10h\x03&\x00I\x99\x99\x99\x99\x99\xe0\x17<\x02\xf8\x05\xf0\v\x9e\x01|\x02\xd0\x02o\x0f\xc07UUUUUS\xbc?3;\xc3\xf3(\a#\x00g\xff\xff\xff\xff\xff\xef\x0f\xc7x\x7fx\x7fx~;\xc3\xfb\xc3\xe8\x01'\x00-UUUUUFq\x8cg\x18\xa0\f\x8c\x0f\xec\xcc\xcc\xcc\xcc\xcd8\xces\x8c\xe6\x80\x13p\x9f\x00$\xcc\xcc\xcc\xcc\xcc\xc7p\x9f\x18\xc7p\x9f\x18\xa0<\x86\x00ڪ\xaa\xaa\xaa\xaa\xdc'øO\xdc'\xee\x13\xe1\xdc'\xee\x13\xd0\x02G\x00'\xff\xff\xff\xff\xff\xc3\x1c0\xc3\x1c0\xa0\f\x86\x01\xba\xaa\xaa\xaa\xaa\xaa\x8e\x18\xe3\x8e\x18\xe3@\t<\x01\xac\xcc\xcc\xcc\xcc\xcd\f\xf0\xc3\f\xf0\u0080r\x18\x01&fffffxg\x9exg\x9d\x00$p\x1f\xd5UUUUT1\xc3\f1\xc3\n\x00\xc8`\x04\xff\xff\xff\xff\xff\xf8\xe1\x8e8\xe1\x8e4\x00\x9bE\xfc\x01\xb5UUUUU\x0e\xd1\x7f\f0\xed\x17\xf0\u0081\xf2\f\x03l\xcc\xcc\xcc\xccʹ_\xc1\xda/\xf6\x8b\xfd\xa2\xfe\x0e\xd1\x7f\xb4_\xa0\x04\x87\x00$\xcc\xcc\xcc\xcc\xcc\xc1\x87\x06\f\x18p`\xa0\f\x83\x00mUUUUUC\x83\x0e\x1c80\xe1\xa0\x04\x8f\x00'\xff\xff\xff\xff\xff\xc1\x8f\x06\f\x18\xf0`\xa0\x1c\x83\a\xfdUUUUUG\x83\x1e|\xf83\xe7\xa0\x04\x87\x00mUUUUUA\x87\x06\f\x18p`\xa0\f\x83\x00$\xcc\xcc\xcc\xcc\xccÃ\x0e\x1c80\xe1\xa0\x04\x8f\x01\xfb33333A\x8f\x06\f\x18\xf0`\xa0\x1c\x83\x00-UUUUUG\x83\x1e Date: Thu, 9 Mar 2023 11:03:30 -0800 Subject: [PATCH 004/231] Remove hacky promql.Test native histogram thing Signed-off-by: Justin Lei --- promql/test.go | 31 ++++++----------------------- storage/remote/read_handler_test.go | 27 +++++++++++++++++-------- 2 files changed, 25 insertions(+), 33 deletions(-) diff --git a/promql/test.go b/promql/test.go index f47269aeca..78cc1e9fbb 100644 --- a/promql/test.go +++ b/promql/test.go @@ -33,7 +33,6 @@ import ( "github.com/prometheus/prometheus/promql/parser" "github.com/prometheus/prometheus/storage" "github.com/prometheus/prometheus/tsdb" - "github.com/prometheus/prometheus/tsdb/tsdbutil" "github.com/prometheus/prometheus/util/teststorage" "github.com/prometheus/prometheus/util/testutil" ) @@ -303,26 +302,14 @@ func (cmd loadCmd) String() string { func (cmd *loadCmd) set(m labels.Labels, vals ...parser.SequenceValue) { h := m.Hash() - metricName := m.Get("__name__") - isHistogram := strings.Contains(metricName, "_histogram_") - samples := make([]Point, 0, len(vals)) ts := testStartTime for _, v := range vals { if !v.Omitted { - t := ts.UnixNano() / int64(time.Millisecond/time.Nanosecond) - - if isHistogram { - samples = append(samples, Point{ - T: t, - H: tsdbutil.GenerateTestFloatHistogram(int(v.Value)), - }) - } else { - samples = append(samples, Point{ - T: t, - V: v.Value, - }) - } + samples = append(samples, Point{ + T: ts.UnixNano() / int64(time.Millisecond/time.Nanosecond), + V: v.Value, + }) } ts = ts.Add(cmd.gap) } @@ -336,14 +323,8 @@ func (cmd *loadCmd) append(a storage.Appender) error { m := cmd.metrics[h] for _, s := range smpls { - if s.H == nil { - if _, err := a.Append(0, m, s.T, s.V); err != nil { - return err - } - } else { - if _, err := a.AppendHistogram(0, m, s.T, nil, s.H); err != nil { - return err - } + if _, err := a.Append(0, m, s.T, s.V); err != nil { + return err } } } diff --git a/storage/remote/read_handler_test.go b/storage/remote/read_handler_test.go index 2ac9cc32b4..261c28e215 100644 --- a/storage/remote/read_handler_test.go +++ b/storage/remote/read_handler_test.go @@ -15,11 +15,13 @@ package remote import ( "bytes" + "context" "errors" "io" "net/http" "net/http/httptest" "testing" + "time" "github.com/gogo/protobuf/proto" "github.com/golang/snappy" @@ -37,12 +39,12 @@ func TestSampledReadEndpoint(t *testing.T) { suite, err := promql.NewTest(t, ` load 1m test_metric1{foo="bar",baz="qux"} 1 - test_histogram_metric1{foo="bar",baz="qux"} 1 `) require.NoError(t, err) - defer suite.Close() + addNativeHistogramsToTestSuite(t, suite, 1) + err = suite.Run() require.NoError(t, err) @@ -123,10 +125,9 @@ func TestSampledReadEndpoint(t *testing.T) { {Name: "b", Value: "c"}, {Name: "baz", Value: "qux"}, {Name: "d", Value: "e"}, - {Name: "foo", Value: "bar"}, }, Histograms: []prompb.Histogram{ - FloatHistogramToHistogramProto(0, tsdbutil.GenerateTestFloatHistogram(1)), + FloatHistogramToHistogramProto(0, tsdbutil.GenerateTestFloatHistogram(0)), }, }, }, @@ -139,7 +140,7 @@ func BenchmarkStreamReadEndpoint(b *testing.B) { test_metric1{foo="bar1",baz="qux"} 0+100x119 test_metric1{foo="bar2",baz="qux"} 0+100x120 test_metric1{foo="bar3",baz="qux"} 0+100x240 -`) + `) require.NoError(b, err) defer suite.Close() @@ -210,12 +211,12 @@ func TestStreamReadEndpoint(t *testing.T) { test_metric1{foo="bar1",baz="qux"} 0+100x119 test_metric1{foo="bar2",baz="qux"} 0+100x120 test_metric1{foo="bar3",baz="qux"} 0+100x240 - test_histogram_metric1{foo="bar4",baz="qux"} 0+1x119 `) require.NoError(t, err) - defer suite.Close() + addNativeHistogramsToTestSuite(t, suite, 120) + require.NoError(t, suite.Run()) api := NewReadHandler(nil, nil, suite.Storage(), func() config.Config { @@ -424,7 +425,6 @@ func TestStreamReadEndpoint(t *testing.T) { {Name: "b", Value: "c"}, {Name: "baz", Value: "qux"}, {Name: "d", Value: "e"}, - {Name: "foo", Value: "bar4"}, }, Chunks: []prompb.Chunk{ { @@ -439,3 +439,14 @@ func TestStreamReadEndpoint(t *testing.T) { }, }, results) } + +func addNativeHistogramsToTestSuite(t *testing.T, pqlTest *promql.Test, n int) { + lbls := labels.FromStrings("__name__", "test_histogram_metric1", "baz", "qux") + + app := pqlTest.Storage().Appender(context.TODO()) + for i, fh := range tsdbutil.GenerateTestFloatHistograms(n) { + _, err := app.AppendHistogram(0, lbls, int64(i)*int64(60*time.Second/time.Millisecond), nil, fh) + require.NoError(t, err) + } + require.NoError(t, app.Commit()) +} From c9d06f2826bef4606e4a8c4edea259f012079f10 Mon Sep 17 00:00:00 2001 From: Ganesh Vernekar Date: Mon, 13 Mar 2023 11:15:45 +0530 Subject: [PATCH 005/231] tsdb: Replay m-map chunk only when required M-map chunks replayed on startup are discarded if there was no WAL and no snapshot loaded, because there is no series created in the Head that it can map to. So only load m-map chunks from disk if there is either a snapshot loaded or there is WAL on disk. Signed-off-by: Ganesh Vernekar --- tsdb/head.go | 44 ++++++++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 16 deletions(-) diff --git a/tsdb/head.go b/tsdb/head.go index b28f5aca5e..f3e0a7f6c9 100644 --- a/tsdb/head.go +++ b/tsdb/head.go @@ -590,6 +590,7 @@ func (h *Head) Init(minValidTime int64) error { snapIdx, snapOffset := -1, 0 refSeries := make(map[chunks.HeadSeriesRef]*memSeries) + snapshotLoaded := false if h.opts.EnableMemorySnapshotOnShutdown { level.Info(h.logger).Log("msg", "Chunk snapshot is enabled, replaying from the snapshot") // If there are any WAL files, there should be at least one WAL file with an index that is current or newer @@ -619,6 +620,7 @@ func (h *Head) Init(minValidTime int64) error { var err error snapIdx, snapOffset, refSeries, err = h.loadChunkSnapshot() if err == nil { + snapshotLoaded = true level.Info(h.logger).Log("msg", "Chunk snapshot loading time", "duration", time.Since(start).String()) } if err != nil { @@ -636,26 +638,36 @@ func (h *Head) Init(minValidTime int64) error { } mmapChunkReplayStart := time.Now() - mmappedChunks, oooMmappedChunks, lastMmapRef, err := h.loadMmappedChunks(refSeries) - if err != nil { - // TODO(codesome): clear out all m-map chunks here for refSeries. - level.Error(h.logger).Log("msg", "Loading on-disk chunks failed", "err", err) - if _, ok := errors.Cause(err).(*chunks.CorruptionErr); ok { - h.metrics.mmapChunkCorruptionTotal.Inc() - } - - // Discard snapshot data since we need to replay the WAL for the missed m-map chunks data. - snapIdx, snapOffset = -1, 0 - - // If this fails, data will be recovered from WAL. - // Hence we wont lose any data (given WAL is not corrupt). - mmappedChunks, oooMmappedChunks, lastMmapRef, err = h.removeCorruptedMmappedChunks(err) + var ( + mmappedChunks map[chunks.HeadSeriesRef][]*mmappedChunk + oooMmappedChunks map[chunks.HeadSeriesRef][]*mmappedChunk + lastMmapRef chunks.ChunkDiskMapperRef + err error + ) + if snapshotLoaded || h.wal != nil { + // If snapshot was not loaded and if there is no WAL, then m-map chunks will be discarded + // anyway. So we only load m-map chunks when it won't be discarded. + mmappedChunks, oooMmappedChunks, lastMmapRef, err = h.loadMmappedChunks(refSeries) if err != nil { - return err + // TODO(codesome): clear out all m-map chunks here for refSeries. + level.Error(h.logger).Log("msg", "Loading on-disk chunks failed", "err", err) + if _, ok := errors.Cause(err).(*chunks.CorruptionErr); ok { + h.metrics.mmapChunkCorruptionTotal.Inc() + } + + // Discard snapshot data since we need to replay the WAL for the missed m-map chunks data. + snapIdx, snapOffset = -1, 0 + + // If this fails, data will be recovered from WAL. + // Hence we wont lose any data (given WAL is not corrupt). + mmappedChunks, oooMmappedChunks, lastMmapRef, err = h.removeCorruptedMmappedChunks(err) + if err != nil { + return err + } } + level.Info(h.logger).Log("msg", "On-disk memory mappable chunks replay completed", "duration", time.Since(mmapChunkReplayStart).String()) } - level.Info(h.logger).Log("msg", "On-disk memory mappable chunks replay completed", "duration", time.Since(mmapChunkReplayStart).String()) if h.wal == nil { level.Info(h.logger).Log("msg", "WAL not found") return nil From 1c3f1216b303aabbaac8f3cf86e186398d717c37 Mon Sep 17 00:00:00 2001 From: Ganesh Vernekar Date: Mon, 13 Mar 2023 12:23:57 +0530 Subject: [PATCH 006/231] tsdb: Test querying after missing wbl with snapshots enabled If the snapshot was enabled with some ooo mmap chunks on disk, and wbl was removed between restarts, then we should still be able to query the ooo mmap chunks after a restart. This test shows that we are not able to query those ooo mmap chunks after a restart under this situation. Signed-off-by: Ganesh Vernekar --- tsdb/db_test.go | 109 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 109 insertions(+) diff --git a/tsdb/db_test.go b/tsdb/db_test.go index cc65069e43..926af273da 100644 --- a/tsdb/db_test.go +++ b/tsdb/db_test.go @@ -4433,6 +4433,115 @@ func TestOOOCompactionWithDisabledWriteLog(t *testing.T) { verifySamples(db.Blocks()[1], 250, 350) } +// TestOOOQueryAfterRestartWithSnapshotAndRemovedWBL tests the scenario where the WBL goes +// missing after a restart while snapshot was enabled, but the query still returns the right +// data from the mmap chunks. +func TestOOOQueryAfterRestartWithSnapshotAndRemovedWBL(t *testing.T) { + dir := t.TempDir() + + opts := DefaultOptions() + opts.OutOfOrderCapMax = 10 + opts.OutOfOrderTimeWindow = 300 * time.Minute.Milliseconds() + opts.EnableMemorySnapshotOnShutdown = true + + db, err := Open(dir, nil, nil, opts, nil) + require.NoError(t, err) + db.DisableCompactions() // We want to manually call it. + t.Cleanup(func() { + require.NoError(t, db.Close()) + }) + + series1 := labels.FromStrings("foo", "bar1") + series2 := labels.FromStrings("foo", "bar2") + + addSamples := func(fromMins, toMins int64) { + app := db.Appender(context.Background()) + for min := fromMins; min <= toMins; min++ { + ts := min * time.Minute.Milliseconds() + _, err := app.Append(0, series1, ts, float64(ts)) + require.NoError(t, err) + _, err = app.Append(0, series2, ts, float64(2*ts)) + require.NoError(t, err) + } + require.NoError(t, app.Commit()) + } + + // Add an in-order samples. + addSamples(250, 350) + + // Add ooo samples that will result into a single block. + addSamples(90, 110) // The sample 110 will not be in m-map chunks. + + // Checking that there are some ooo m-map chunks. + for _, lbls := range []labels.Labels{series1, series2} { + ms, created, err := db.head.getOrCreate(lbls.Hash(), lbls) + require.NoError(t, err) + require.False(t, created) + require.Equal(t, 2, len(ms.ooo.oooMmappedChunks)) + require.NotNil(t, ms.ooo.oooHeadChunk) + } + + // Restart DB. + require.NoError(t, db.Close()) + + // For some reason wbl goes missing. + require.NoError(t, os.RemoveAll(path.Join(dir, "wbl"))) + + db, err = Open(dir, nil, nil, opts, nil) + require.NoError(t, err) + db.DisableCompactions() // We want to manually call it. + + // Check ooo m-map chunks again. + for _, lbls := range []labels.Labels{series1, series2} { + ms, created, err := db.head.getOrCreate(lbls.Hash(), lbls) + require.NoError(t, err) + require.False(t, created) + require.Equal(t, 2, len(ms.ooo.oooMmappedChunks)) + require.Equal(t, 109*time.Minute.Milliseconds(), ms.ooo.oooMmappedChunks[1].maxTime) + require.Nil(t, ms.ooo.oooHeadChunk) // Because of missing wbl. + } + + verifySamples := func(fromMins, toMins int64) { + series1Samples := make([]tsdbutil.Sample, 0, toMins-fromMins+1) + series2Samples := make([]tsdbutil.Sample, 0, toMins-fromMins+1) + for min := fromMins; min <= toMins; min++ { + ts := min * time.Minute.Milliseconds() + series1Samples = append(series1Samples, sample{ts, float64(ts), nil, nil}) + series2Samples = append(series2Samples, sample{ts, float64(2 * ts), nil, nil}) + } + expRes := map[string][]tsdbutil.Sample{ + series1.String(): series1Samples, + series2.String(): series2Samples, + } + + q, err := db.Querier(context.Background(), fromMins*time.Minute.Milliseconds(), toMins*time.Minute.Milliseconds()) + require.NoError(t, err) + + actRes := query(t, q, labels.MustNewMatcher(labels.MatchRegexp, "foo", "bar.*")) + require.Equal(t, expRes, actRes) + } + + // Checking for expected ooo data from mmap chunks. + verifySamples(90, 109) + + // Compaction should also work fine. + require.Equal(t, len(db.Blocks()), 0) + require.NoError(t, db.CompactOOOHead()) + require.Equal(t, len(db.Blocks()), 1) // One block from OOO data. + require.Equal(t, int64(0), db.Blocks()[0].MinTime()) + require.Equal(t, 120*time.Minute.Milliseconds(), db.Blocks()[0].MaxTime()) + + // Checking that ooo chunk is empty in Head. + for _, lbls := range []labels.Labels{series1, series2} { + ms, created, err := db.head.getOrCreate(lbls.Hash(), lbls) + require.NoError(t, err) + require.False(t, created) + require.Nil(t, ms.ooo) + } + + verifySamples(90, 109) +} + func Test_Querier_OOOQuery(t *testing.T) { opts := DefaultOptions() opts.OutOfOrderCapMax = 30 From 2af44f955871a18bd043f807441888aedd31dff7 Mon Sep 17 00:00:00 2001 From: Ganesh Vernekar Date: Mon, 13 Mar 2023 12:27:46 +0530 Subject: [PATCH 007/231] tsdb: Update OOO min/max time properly after replaying m-map chunks Without this fix, if snapshots were enabled, and wbl goes missing between restarts, then TSDB does not recognize that there are ooo mmap chunks on disk and we cannot query them until those chunks are compacted into blocks. Signed-off-by: Ganesh Vernekar --- tsdb/head.go | 1 + 1 file changed, 1 insertion(+) diff --git a/tsdb/head.go b/tsdb/head.go index f3e0a7f6c9..41933d7a04 100644 --- a/tsdb/head.go +++ b/tsdb/head.go @@ -836,6 +836,7 @@ func (h *Head) loadMmappedChunks(refSeries map[chunks.HeadSeriesRef]*memSeries) numSamples: numSamples, }) + h.updateMinOOOMaxOOOTime(mint, maxt) return nil } From dd94ebb87bbc14e103ccca0cb8283246d40c70fb Mon Sep 17 00:00:00 2001 From: Trevor Whitney Date: Wed, 15 Feb 2023 05:49:51 -0700 Subject: [PATCH 008/231] promql: set CounterResetHint after rate and sum Signed-off-by: Trevor Whitney --- model/histogram/float_histogram.go | 11 +++++++++++ promql/engine_test.go | 6 ++---- promql/functions.go | 3 +++ 3 files changed, 16 insertions(+), 4 deletions(-) diff --git a/model/histogram/float_histogram.go b/model/histogram/float_histogram.go index 256679a8c9..6b823f7d38 100644 --- a/model/histogram/float_histogram.go +++ b/model/histogram/float_histogram.go @@ -192,6 +192,8 @@ func (h *FloatHistogram) Scale(factor float64) *FloatHistogram { // // This method returns a pointer to the receiving histogram for convenience. func (h *FloatHistogram) Add(other *FloatHistogram) *FloatHistogram { + // TODO(trevorwhitney): If other.CounterResetHint != h.CounterResetHint then + // we should return some warning. otherZeroCount := h.reconcileZeroBuckets(other) h.ZeroCount += otherZeroCount h.Count += other.Count @@ -438,6 +440,15 @@ func (h *FloatHistogram) Compact(maxEmptyBuckets int) *FloatHistogram { // information can be read directly from there rather than be detected each time // again. func (h *FloatHistogram) DetectReset(previous *FloatHistogram) bool { + if h.CounterResetHint == CounterReset { + return true + } + if h.CounterResetHint == NotCounterReset { + return false + } + // In all other cases of CounterResetHint, we go on as we would otherwise. + // Even in the GaugeHistogram case, we pretend this is a counter histogram + // for consistency. if h.Count < previous.Count { return true } diff --git a/promql/engine_test.go b/promql/engine_test.go index e2e209849b..d1c4570ea2 100644 --- a/promql/engine_test.go +++ b/promql/engine_test.go @@ -3155,8 +3155,7 @@ func TestNativeHistogramRate(t *testing.T) { require.Len(t, vector, 1) actualHistogram := vector[0].H expectedHistogram := &histogram.FloatHistogram{ - // TODO(beorn7): This should be GaugeType. Change it once supported by PromQL. - CounterResetHint: histogram.NotCounterReset, + CounterResetHint: histogram.GaugeType, Schema: 1, ZeroThreshold: 0.001, ZeroCount: 1. / 15., @@ -3200,8 +3199,7 @@ func TestNativeFloatHistogramRate(t *testing.T) { require.Len(t, vector, 1) actualHistogram := vector[0].H expectedHistogram := &histogram.FloatHistogram{ - // TODO(beorn7): This should be GaugeType. Change it once supported by PromQL. - CounterResetHint: histogram.NotCounterReset, + CounterResetHint: histogram.GaugeType, Schema: 1, ZeroThreshold: 0.001, ZeroCount: 1. / 15., diff --git a/promql/functions.go b/promql/functions.go index c5922002b0..3da38ea0f3 100644 --- a/promql/functions.go +++ b/promql/functions.go @@ -187,6 +187,7 @@ func histogramRate(points []Point, isCounter bool) *histogram.FloatHistogram { if curr == nil { return nil // Range contains a mix of histograms and floats. } + // TODO(trevorwhitney): Check if isCounter is consistent with curr.CounterResetHint. if !isCounter { continue } @@ -208,6 +209,8 @@ func histogramRate(points []Point, isCounter bool) *histogram.FloatHistogram { prev = curr } } + + h.CounterResetHint = histogram.GaugeType return h.Compact(0) } From c3e0a83725c77a868310f13a003b324994e93bb7 Mon Sep 17 00:00:00 2001 From: Trevor Whitney Date: Wed, 15 Feb 2023 05:59:02 -0700 Subject: [PATCH 009/231] rules: no longer force CounterResetHint to Gauge Signed-off-by: Trevor Whitney --- rules/manager.go | 4 ---- rules/manager_test.go | 2 -- 2 files changed, 6 deletions(-) diff --git a/rules/manager.go b/rules/manager.go index 6f6ce2cfe4..f8dcf081fe 100644 --- a/rules/manager.go +++ b/rules/manager.go @@ -31,7 +31,6 @@ import ( "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/codes" - "github.com/prometheus/prometheus/model/histogram" "github.com/prometheus/prometheus/model/labels" "github.com/prometheus/prometheus/model/rulefmt" "github.com/prometheus/prometheus/model/timestamp" @@ -671,9 +670,6 @@ func (g *Group) Eval(ctx context.Context, ts time.Time) { for _, s := range vector { if s.H != nil { - // We assume that all native histogram results are gauge histograms. - // TODO(codesome): once PromQL can give the counter reset info, remove this assumption. - s.H.CounterResetHint = histogram.GaugeType _, err = app.AppendHistogram(0, s.Metric, s.T, nil, s.H) } else { _, err = app.Append(0, s.Metric, s.T, s.V) diff --git a/rules/manager_test.go b/rules/manager_test.go index d287c25ce0..aed289c5b9 100644 --- a/rules/manager_test.go +++ b/rules/manager_test.go @@ -30,7 +30,6 @@ import ( "go.uber.org/goleak" "gopkg.in/yaml.v2" - "github.com/prometheus/prometheus/model/histogram" "github.com/prometheus/prometheus/model/labels" "github.com/prometheus/prometheus/model/rulefmt" "github.com/prometheus/prometheus/model/timestamp" @@ -1393,7 +1392,6 @@ func TestNativeHistogramsInRecordingRules(t *testing.T) { for _, h := range hists[1:] { expHist = expHist.Add(h.ToFloat()) } - expHist.CounterResetHint = histogram.GaugeType it := s.Iterator(nil) require.Equal(t, chunkenc.ValFloatHistogram, it.Next()) From b4e324f637281c2fc75893c9f33899697ec38e6a Mon Sep 17 00:00:00 2001 From: Trevor Whitney Date: Wed, 8 Mar 2023 14:58:40 -0700 Subject: [PATCH 010/231] Handle valid cases of mismatched hints when adding Signed-off-by: Trevor Whitney --- model/histogram/float_histogram.go | 43 +++++++++++++++++++++++------- 1 file changed, 33 insertions(+), 10 deletions(-) diff --git a/model/histogram/float_histogram.go b/model/histogram/float_histogram.go index 6b823f7d38..e96b5682b7 100644 --- a/model/histogram/float_histogram.go +++ b/model/histogram/float_histogram.go @@ -192,8 +192,25 @@ func (h *FloatHistogram) Scale(factor float64) *FloatHistogram { // // This method returns a pointer to the receiving histogram for convenience. func (h *FloatHistogram) Add(other *FloatHistogram) *FloatHistogram { - // TODO(trevorwhitney): If other.CounterResetHint != h.CounterResetHint then - // we should return some warning. + if other.CounterResetHint != h.CounterResetHint { + // The outcome of adding an increment to a guage histogram will always be a GaugeType + if other.CounterResetHint == GaugeType && h.CounterResetHint != GaugeType { + h.CounterResetHint = GaugeType + } + + // This could be legitime if the caller knows what they are doing, but the resulting hint + // must be UnknownCounterReset. + if other.CounterResetHint == UnknownCounterReset && h.CounterResetHint != GaugeType { + h.CounterResetHint = UnknownCounterReset + } + + // TODO(trevorwhitney): this leaves CounterReset and NotCounterReset. If we have mismatch of + // these hints, that cannot be right, and we should raise a warning when possible. + // if other.CounterResetHint == CounterReset && h.CounterResetHint == NotCounterReset || + // other.CounterResetHint == NotCounterReset && h.CounterResetHint == CounterReset { + // } + } + otherZeroCount := h.reconcileZeroBuckets(other) h.ZeroCount += otherZeroCount h.Count += other.Count @@ -416,6 +433,10 @@ func (h *FloatHistogram) Compact(maxEmptyBuckets int) *FloatHistogram { // of observations, but NOT the sum of observations) is smaller in the receiving // histogram compared to the previous histogram. Otherwise, it returns false. // +// This method will shortcut to true if a CounterReset is detected, and shortcut +// to false if NotCounterReset is detected. Otherwise it will do the work to detect +// a reset. +// // Special behavior in case the Schema or the ZeroThreshold are not the same in // both histograms: // @@ -434,11 +455,6 @@ func (h *FloatHistogram) Compact(maxEmptyBuckets int) *FloatHistogram { // - Upon a decrease of the Schema, the buckets of the previous histogram are // merged so that they match the new, lower-resolution schema (again without // mutating the provided previous histogram). -// -// Note that this kind of reset detection is quite expensive. Ideally, resets -// are detected at ingest time and stored in the TSDB, so that the reset -// information can be read directly from there rather than be detected each time -// again. func (h *FloatHistogram) DetectReset(previous *FloatHistogram) bool { if h.CounterResetHint == CounterReset { return true @@ -446,9 +462,16 @@ func (h *FloatHistogram) DetectReset(previous *FloatHistogram) bool { if h.CounterResetHint == NotCounterReset { return false } - // In all other cases of CounterResetHint, we go on as we would otherwise. - // Even in the GaugeHistogram case, we pretend this is a counter histogram - // for consistency. + // In all other cases of CounterResetHint (UnknownCounterReset and GaugeType), + // we go on as we would otherwise, for reasons explained below. + // + // If the CounterResetHint is UnknownCounterReset, we do not know yet if this histogram comes + // with a counter reset. Therefore, we have to do all the detailed work to find out if there + // is a counter reset or not. + // We do the same if the CounterResetHint is GaugeType, which should not happen, but PromQL still + // allows the user to apply functions to gauge histograms that are only meant for counter histograms. + // In this case, we treat the gauge histograms as a counter histograms + // (and we plan to return a warning about it to the user). if h.Count < previous.Count { return true } From e3513d1dd222154bafcc355281d17109e0a62b01 Mon Sep 17 00:00:00 2001 From: Trevor Whitney Date: Mon, 13 Mar 2023 14:31:49 -0600 Subject: [PATCH 011/231] Change nested ifs to a switch Signed-off-by: Trevor Whitney --- model/histogram/float_histogram.go | 45 +++++++++++++++++------------- 1 file changed, 25 insertions(+), 20 deletions(-) diff --git a/model/histogram/float_histogram.go b/model/histogram/float_histogram.go index e96b5682b7..cd73083bbd 100644 --- a/model/histogram/float_histogram.go +++ b/model/histogram/float_histogram.go @@ -192,23 +192,28 @@ func (h *FloatHistogram) Scale(factor float64) *FloatHistogram { // // This method returns a pointer to the receiving histogram for convenience. func (h *FloatHistogram) Add(other *FloatHistogram) *FloatHistogram { - if other.CounterResetHint != h.CounterResetHint { - // The outcome of adding an increment to a guage histogram will always be a GaugeType - if other.CounterResetHint == GaugeType && h.CounterResetHint != GaugeType { - h.CounterResetHint = GaugeType - } - - // This could be legitime if the caller knows what they are doing, but the resulting hint - // must be UnknownCounterReset. - if other.CounterResetHint == UnknownCounterReset && h.CounterResetHint != GaugeType { - h.CounterResetHint = UnknownCounterReset - } - - // TODO(trevorwhitney): this leaves CounterReset and NotCounterReset. If we have mismatch of - // these hints, that cannot be right, and we should raise a warning when possible. - // if other.CounterResetHint == CounterReset && h.CounterResetHint == NotCounterReset || - // other.CounterResetHint == NotCounterReset && h.CounterResetHint == CounterReset { - // } + switch { + case other.CounterResetHint == h.CounterResetHint: + // Adding apples to apples, all good. No need to change anything. + case h.CounterResetHint == GaugeType: + // Adding something else to a gauge. That's probably OK. Outcome is a gauge. + // Nothing to do since the receiver is already marked as gauge. + case other.CounterResetHint == GaugeType: + // Similar to before, but this time the receiver is "something else" and we have to change it to gauge. + h.CounterResetHint = GaugeType + case h.CounterResetHint == UnknownCounterReset: + // With the receiver's CounterResetHint being "unknown", this could still be legitimate + // if the caller knows what they are doing. Outcome is then again "unknown". + // No need to do anything since the receiver's CounterResetHint is already "unknown". + case other.CounterResetHint == UnknownCounterReset: + // Similar to before, but now we have to set the receiver's CounterResetHint to "unknown". + h.CounterResetHint = UnknownCounterReset + default: + // All other cases shouldn't actually happen. + // They are a direct collision of CounterReset and NotCounterReset. + // Conservatively set the CounterResetHint to "unknown" and isse a warning. + h.CounterResetHint = UnknownCounterReset + // TODO(trevorwhitney): Actually issue the warning as soon as the plumbing for it is in place } otherZeroCount := h.reconcileZeroBuckets(other) @@ -433,9 +438,9 @@ func (h *FloatHistogram) Compact(maxEmptyBuckets int) *FloatHistogram { // of observations, but NOT the sum of observations) is smaller in the receiving // histogram compared to the previous histogram. Otherwise, it returns false. // -// This method will shortcut to true if a CounterReset is detected, and shortcut -// to false if NotCounterReset is detected. Otherwise it will do the work to detect -// a reset. +// This method will shortcut to true if a CounterReset is detected, and shortcut +// to false if NotCounterReset is detected. Otherwise it will do the work to detect +// a reset. // // Special behavior in case the Schema or the ZeroThreshold are not the same in // both histograms: From 0c0c2af7f5566f1e7002f26ea51549afb793d323 Mon Sep 17 00:00:00 2001 From: Ganesh Vernekar Date: Sun, 19 Feb 2023 23:04:51 +0530 Subject: [PATCH 012/231] Do not re-encode head chunk in ChunkQuerier Signed-off-by: Ganesh Vernekar --- tsdb/db_test.go | 73 +++++++++++++++++++++++++++++++++++++++++++++++ tsdb/head_read.go | 51 ++++++++++++++++++++++++++------- tsdb/querier.go | 46 +++++++++++++++-------------- tsdb/test.txt | 1 + 4 files changed, 139 insertions(+), 32 deletions(-) diff --git a/tsdb/db_test.go b/tsdb/db_test.go index 9e5623bea7..26da7122ad 100644 --- a/tsdb/db_test.go +++ b/tsdb/db_test.go @@ -6334,3 +6334,76 @@ func compareSeries(t require.TestingT, expected, actual map[string][]tsdbutil.Sa } } } + +// TestChunkQuerierReadWriteRace looks for any possible race between appending +// samples and reading chunks because the head chunk that is being appended to +// can be read in parallel and we should be able to make a copy of the chunk without +// worrying about the parallel write. +func TestChunkQuerierReadWriteRace(t *testing.T) { + db := openTestDB(t, nil, nil) + defer func() { + require.NoError(t, db.Close()) + }() + + lbls := labels.FromStrings("foo", "bar") + + writer := func() error { + <-time.After(5 * time.Millisecond) // Initial pause while readers start. + ts := 0 + for i := 0; i < 500; i++ { + app := db.Appender(context.Background()) + for j := 0; j < 10; j++ { + ts++ + _, err := app.Append(0, lbls, int64(ts), float64(ts*100)) + if err != nil { + return err + } + } + err := app.Commit() + if err != nil { + return err + } + <-time.After(time.Millisecond) + } + return nil + } + + reader := func() { + querier, err := db.ChunkQuerier(context.Background(), math.MinInt64, math.MaxInt64) + require.NoError(t, err) + defer func(q storage.ChunkQuerier) { + require.NoError(t, q.Close()) + }(querier) + ss := querier.Select(false, nil, labels.MustNewMatcher(labels.MatchEqual, "foo", "bar")) + for ss.Next() { + cs := ss.At() + it := cs.Iterator(nil) + for it.Next() { + m := it.At() + b := m.Chunk.Bytes() + bb := make([]byte, len(b)) + copy(bb, b) // This copying of chunk bytes detects any race. + } + } + require.NoError(t, ss.Err()) + } + + ch := make(chan struct{}) + var writerErr error + go func() { + defer close(ch) + writerErr = writer() + }() + +Outer: + for { + reader() + select { + case <-ch: + break Outer + default: + } + } + + require.NoError(t, writerErr) +} diff --git a/tsdb/head_read.go b/tsdb/head_read.go index efcafcf6c5..b54e53aa07 100644 --- a/tsdb/head_read.go +++ b/tsdb/head_read.go @@ -274,22 +274,36 @@ func (h *headChunkReader) Close() error { // Chunk returns the chunk for the reference number. func (h *headChunkReader) Chunk(meta chunks.Meta) (chunkenc.Chunk, error) { + chk, _, err := h.chunk(meta, false) + return chk, err +} + +// ChunkWithCopy returns the chunk for the reference number. +// If the chunk is the in-memory chunk, then it makes a copy and returns the copied chunk. +func (h *headChunkReader) ChunkWithCopy(meta chunks.Meta) (chunkenc.Chunk, int64, error) { + return h.chunk(meta, true) +} + +// chunk returns the chunk for the reference number. +// If copyLastChunk is true, then it makes a copy of the head chunk if asked for it. +// Also returns max time of the chunk. +func (h *headChunkReader) chunk(meta chunks.Meta, copyLastChunk bool) (chunkenc.Chunk, int64, error) { sid, cid := chunks.HeadChunkRef(meta.Ref).Unpack() s := h.head.series.getByID(sid) // This means that the series has been garbage collected. if s == nil { - return nil, storage.ErrNotFound + return nil, 0, storage.ErrNotFound } s.Lock() - c, garbageCollect, err := s.chunk(cid, h.head.chunkDiskMapper, &h.head.memChunkPool) + c, headChunk, err := s.chunk(cid, h.head.chunkDiskMapper, &h.head.memChunkPool) if err != nil { s.Unlock() - return nil, err + return nil, 0, err } defer func() { - if garbageCollect { + if !headChunk { // Set this to nil so that Go GC can collect it after it has been used. c.chunk = nil h.head.memChunkPool.Put(c) @@ -299,22 +313,36 @@ func (h *headChunkReader) Chunk(meta chunks.Meta) (chunkenc.Chunk, error) { // This means that the chunk is outside the specified range. if !c.OverlapsClosedInterval(h.mint, h.maxt) { s.Unlock() - return nil, storage.ErrNotFound + return nil, 0, storage.ErrNotFound + } + + chk, maxTime := c.chunk, c.maxTime + if headChunk && copyLastChunk { + // The caller may ask to copy the head chunk in order to take the + // bytes of the chunk without causing the race between read and append. + b := s.headChunk.chunk.Bytes() + newB := make([]byte, len(b)) + copy(newB, b) // TODO(codesome): Use bytes.Clone() when we upgrade to Go 1.20. + // TODO(codesome): Put back in the pool (non-trivial). + chk, err = h.head.opts.ChunkPool.Get(s.headChunk.chunk.Encoding(), newB) + if err != nil { + return nil, 0, err + } } s.Unlock() return &safeChunk{ - Chunk: c.chunk, + Chunk: chk, s: s, cid: cid, isoState: h.isoState, - }, nil + }, maxTime, nil } // chunk returns the chunk for the HeadChunkID from memory or by m-mapping it from the disk. -// If garbageCollect is true, it means that the returned *memChunk +// If headChunk is true, it means that the returned *memChunk // (and not the chunkenc.Chunk inside it) can be garbage collected after its usage. -func (s *memSeries) chunk(id chunks.HeadChunkID, chunkDiskMapper *chunks.ChunkDiskMapper, memChunkPool *sync.Pool) (chunk *memChunk, garbageCollect bool, err error) { +func (s *memSeries) chunk(id chunks.HeadChunkID, chunkDiskMapper *chunks.ChunkDiskMapper, memChunkPool *sync.Pool) (chunk *memChunk, headChunk bool, err error) { // ix represents the index of chunk in the s.mmappedChunks slice. The chunk id's are // incremented by 1 when new chunk is created, hence (id - firstChunkID) gives the slice index. // The max index for the s.mmappedChunks slice can be len(s.mmappedChunks)-1, hence if the ix @@ -323,11 +351,12 @@ func (s *memSeries) chunk(id chunks.HeadChunkID, chunkDiskMapper *chunks.ChunkDi if ix < 0 || ix > len(s.mmappedChunks) { return nil, false, storage.ErrNotFound } + if ix == len(s.mmappedChunks) { if s.headChunk == nil { return nil, false, errors.New("invalid head chunk") } - return s.headChunk, false, nil + return s.headChunk, true, nil } chk, err := chunkDiskMapper.Chunk(s.mmappedChunks[ix].ref) if err != nil { @@ -340,7 +369,7 @@ func (s *memSeries) chunk(id chunks.HeadChunkID, chunkDiskMapper *chunks.ChunkDi mc.chunk = chk mc.minTime = s.mmappedChunks[ix].minTime mc.maxTime = s.mmappedChunks[ix].maxTime - return mc, true, nil + return mc, false, nil } // oooMergedChunk returns the requested chunk based on the given chunks.Meta diff --git a/tsdb/querier.go b/tsdb/querier.go index 061d5b3941..b4513218e9 100644 --- a/tsdb/querier.go +++ b/tsdb/querier.go @@ -584,7 +584,11 @@ func (p *populateWithDelGenericSeriesIterator) reset(blockID ulid.ULID, cr Chunk p.currChkMeta = chunks.Meta{} } -func (p *populateWithDelGenericSeriesIterator) next() bool { +// If copyHeadChunk is true, then the head chunk (i.e. the in-memory chunk of the TSDB) +// is deep copied to avoid races between reads and copying chunk bytes. +// However, if the deletion intervals overlaps with the head chunk, then the head chunk is +// not copied irrespective of copyHeadChunk because it will be re-encoded later anyway. +func (p *populateWithDelGenericSeriesIterator) next(copyHeadChunk bool) bool { if p.err != nil || p.i >= len(p.chks)-1 { return false } @@ -592,12 +596,6 @@ func (p *populateWithDelGenericSeriesIterator) next() bool { p.i++ p.currChkMeta = p.chks[p.i] - p.currChkMeta.Chunk, p.err = p.chunks.Chunk(p.currChkMeta) - if p.err != nil { - p.err = errors.Wrapf(p.err, "cannot populate chunk %d from block %s", p.currChkMeta.Ref, p.blockID.String()) - return false - } - p.bufIter.Intervals = p.bufIter.Intervals[:0] for _, interval := range p.intervals { if p.currChkMeta.OverlapsClosedInterval(interval.Mint, interval.Maxt) { @@ -605,22 +603,28 @@ func (p *populateWithDelGenericSeriesIterator) next() bool { } } - // Re-encode head chunks that are still open (being appended to) or - // outside the compacted MaxTime range. - // The chunk.Bytes() method is not safe for open chunks hence the re-encoding. - // This happens when snapshotting the head block or just fetching chunks from TSDB. - // - // TODO(codesome): think how to avoid the typecasting to verify when it is head block. - _, isSafeChunk := p.currChkMeta.Chunk.(*safeChunk) - if len(p.bufIter.Intervals) == 0 && !(isSafeChunk && p.currChkMeta.MaxTime == math.MaxInt64) { - // If there is no overlap with deletion intervals AND it's NOT - // an "open" head chunk, we can take chunk as it is. + hcr, ok := p.chunks.(*headChunkReader) + if ok && copyHeadChunk && len(p.bufIter.Intervals) == 0 { + // ChunkWithCopy will copy the head chunk. + var maxt int64 + p.currChkMeta.Chunk, maxt, p.err = hcr.ChunkWithCopy(p.currChkMeta) + // For the in-memory head chunk the index reader sets maxt as MaxInt64. We fix it here. + p.currChkMeta.MaxTime = maxt + } else { + p.currChkMeta.Chunk, p.err = p.chunks.Chunk(p.currChkMeta) + } + if p.err != nil { + p.err = errors.Wrapf(p.err, "cannot populate chunk %d from block %s", p.currChkMeta.Ref, p.blockID.String()) + return false + } + + if len(p.bufIter.Intervals) == 0 { + // If there is no overlap with deletion intervals, we can take chunk as it is. p.currDelIter = nil return true } - // We don't want the full chunk, or it's potentially still opened, take - // just a part of it. + // We don't want the full chunk, take just a part of it. p.bufIter.Iter = p.currChkMeta.Chunk.Iterator(p.bufIter.Iter) p.currDelIter = &p.bufIter return true @@ -677,7 +681,7 @@ func (p *populateWithDelSeriesIterator) Next() chunkenc.ValueType { } } - for p.next() { + for p.next(false) { if p.currDelIter != nil { p.curr = p.currDelIter } else { @@ -742,7 +746,7 @@ func (p *populateWithDelChunkSeriesIterator) reset(blockID ulid.ULID, cr ChunkRe } func (p *populateWithDelChunkSeriesIterator) Next() bool { - if !p.next() { + if !p.next(true) { return false } p.curr = p.currChkMeta diff --git a/tsdb/test.txt b/tsdb/test.txt index e69de29bb2..a66a6fb729 100644 --- a/tsdb/test.txt +++ b/tsdb/test.txt @@ -0,0 +1 @@ +make: Nothing to be done for `test'. From 45b025898faeec98569b5900c5adc556f9adcbcf Mon Sep 17 00:00:00 2001 From: Ganesh Vernekar Date: Mon, 20 Feb 2023 13:58:06 +0530 Subject: [PATCH 013/231] Add BenchmarkHeadChunkQuerier and BenchmarkHeadQuerier Signed-off-by: Ganesh Vernekar --- tsdb/querier_test.go | 90 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 90 insertions(+) diff --git a/tsdb/querier_test.go b/tsdb/querier_test.go index e6e9f143f1..ae9ddad015 100644 --- a/tsdb/querier_test.go +++ b/tsdb/querier_test.go @@ -2246,3 +2246,93 @@ func TestBlockBaseSeriesSet(t *testing.T) { require.NoError(t, bcs.Err()) } } + +func BenchmarkHeadChunkQuerier(b *testing.B) { + db := openTestDB(b, nil, nil) + defer func() { + require.NoError(b, db.Close()) + }() + + // 3h of data. + numTimeseries := 100 + app := db.Appender(context.Background()) + for i := 0; i < 120*6; i++ { + for j := 0; j < numTimeseries; j++ { + lbls := labels.FromStrings("foo", fmt.Sprintf("bar%d", j)) + if i%10 == 0 { + require.NoError(b, app.Commit()) + app = db.Appender(context.Background()) + } + _, err := app.Append(0, lbls, int64(i*15)*time.Second.Milliseconds(), float64(i*100)) + require.NoError(b, err) + } + } + require.NoError(b, app.Commit()) + + querier, err := db.ChunkQuerier(context.Background(), math.MinInt64, math.MaxInt64) + require.NoError(b, err) + defer func(q storage.ChunkQuerier) { + require.NoError(b, q.Close()) + }(querier) + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + ss := querier.Select(false, nil, labels.MustNewMatcher(labels.MatchRegexp, "foo", "bar.*")) + total := 0 + for ss.Next() { + cs := ss.At() + it := cs.Iterator(nil) + for it.Next() { + m := it.At() + total += m.Chunk.NumSamples() + } + } + _ = total + require.NoError(b, ss.Err()) + } +} + +func BenchmarkHeadQuerier(b *testing.B) { + db := openTestDB(b, nil, nil) + defer func() { + require.NoError(b, db.Close()) + }() + + // 3h of data. + numTimeseries := 100 + app := db.Appender(context.Background()) + for i := 0; i < 120*6; i++ { + for j := 0; j < numTimeseries; j++ { + lbls := labels.FromStrings("foo", fmt.Sprintf("bar%d", j)) + if i%10 == 0 { + require.NoError(b, app.Commit()) + app = db.Appender(context.Background()) + } + _, err := app.Append(0, lbls, int64(i*15)*time.Second.Milliseconds(), float64(i*100)) + require.NoError(b, err) + } + } + require.NoError(b, app.Commit()) + + querier, err := db.Querier(context.Background(), math.MinInt64, math.MaxInt64) + require.NoError(b, err) + defer func(q storage.Querier) { + require.NoError(b, q.Close()) + }(querier) + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + ss := querier.Select(false, nil, labels.MustNewMatcher(labels.MatchRegexp, "foo", "bar.*")) + total := int64(0) + for ss.Next() { + cs := ss.At() + it := cs.Iterator(nil) + for it.Next() != chunkenc.ValNone { + ts, _ := it.At() + total += ts + } + } + _ = total + require.NoError(b, ss.Err()) + } +} From 0a3f203c636c5de3a32b3ef35350b041de5afcb0 Mon Sep 17 00:00:00 2001 From: Ganesh Vernekar Date: Tue, 21 Feb 2023 14:02:59 +0530 Subject: [PATCH 014/231] Update tests to not assume the chunk implementation Signed-off-by: Ganesh Vernekar --- tsdb/db_test.go | 26 ++++++++++++++++++++++---- tsdb/querier_test.go | 14 +++++++++++++- 2 files changed, 35 insertions(+), 5 deletions(-) diff --git a/tsdb/db_test.go b/tsdb/db_test.go index 26da7122ad..04db4560c2 100644 --- a/tsdb/db_test.go +++ b/tsdb/db_test.go @@ -130,7 +130,25 @@ func query(t testing.TB, q storage.Querier, matchers ...*labels.Matcher) map[str return result } -// queryChunks runs a matcher query against the querier and fully expands its data. +// queryAndExpandChunks runs a matcher query against the querier and fully expands its data into samples. +func queryAndExpandChunks(t testing.TB, q storage.ChunkQuerier, matchers ...*labels.Matcher) map[string][][]tsdbutil.Sample { + s := queryChunks(t, q, matchers...) + + res := make(map[string][][]tsdbutil.Sample) + for k, v := range s { + var samples [][]tsdbutil.Sample + for _, chk := range v { + sam, err := storage.ExpandSamples(chk.Chunk.Iterator(nil), nil) + require.NoError(t, err) + samples = append(samples, sam) + } + res[k] = samples + } + + return res +} + +// queryChunks runs a matcher query against the querier and expands its data. func queryChunks(t testing.TB, q storage.ChunkQuerier, matchers ...*labels.Matcher) map[string][]chunks.Meta { ss := q.Select(false, nil, matchers...) defer func() { @@ -2367,7 +2385,7 @@ func TestDBReadOnly(t *testing.T) { logger = log.NewLogfmtLogger(log.NewSyncWriter(os.Stderr)) expBlocks []*Block expSeries map[string][]tsdbutil.Sample - expChunks map[string][]chunks.Meta + expChunks map[string][][]tsdbutil.Sample expDBHash []byte matchAll = labels.MustNewMatcher(labels.MatchEqual, "", "") err error @@ -2418,7 +2436,7 @@ func TestDBReadOnly(t *testing.T) { expSeries = query(t, q, matchAll) cq, err := dbWritable.ChunkQuerier(context.TODO(), math.MinInt64, math.MaxInt64) require.NoError(t, err) - expChunks = queryChunks(t, cq, matchAll) + expChunks = queryAndExpandChunks(t, cq, matchAll) require.NoError(t, dbWritable.Close()) // Close here to allow getting the dir hash for windows. expDBHash = testutil.DirHash(t, dbWritable.Dir()) @@ -2452,7 +2470,7 @@ func TestDBReadOnly(t *testing.T) { t.Run("chunk querier", func(t *testing.T) { cq, err := dbReadOnly.ChunkQuerier(context.TODO(), math.MinInt64, math.MaxInt64) require.NoError(t, err) - readOnlySeries := queryChunks(t, cq, matchAll) + readOnlySeries := queryAndExpandChunks(t, cq, matchAll) readOnlyDBHash := testutil.DirHash(t, dbDir) require.Equal(t, len(expChunks), len(readOnlySeries), "total series mismatch") diff --git a/tsdb/querier_test.go b/tsdb/querier_test.go index ae9ddad015..cf9867a4f3 100644 --- a/tsdb/querier_test.go +++ b/tsdb/querier_test.go @@ -235,7 +235,19 @@ func testBlockQuerier(t *testing.T, c blockQuerierTestCase, ir IndexReader, cr C chksRes, errRes := storage.ExpandChunks(sres.Iterator(nil)) rmChunkRefs(chksRes) require.Equal(t, errExp, errRes) - require.Equal(t, chksExp, chksRes) + + require.Equal(t, len(chksExp), len(chksRes)) + var exp, act [][]tsdbutil.Sample + for i := range chksExp { + samples, err := storage.ExpandSamples(chksExp[i].Chunk.Iterator(nil), nil) + require.NoError(t, err) + exp = append(exp, samples) + samples, err = storage.ExpandSamples(chksRes[i].Chunk.Iterator(nil), nil) + require.NoError(t, err) + act = append(act, samples) + } + + require.Equal(t, exp, act) } require.NoError(t, res.Err()) }) From d01c51fad09784aa53bdf5b16fc39bbd335d58e2 Mon Sep 17 00:00:00 2001 From: g3offrey <11151445+g3offrey@users.noreply.github.com> Date: Wed, 15 Mar 2023 15:57:15 +0100 Subject: [PATCH 015/231] docs: update ansible installation link Signed-off-by: g3offrey <11151445+g3offrey@users.noreply.github.com> --- docs/installation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/installation.md b/docs/installation.md index 592d67b28e..05df14a46e 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -87,7 +87,7 @@ the following third-party contributions: ### Ansible -* [Cloud Alchemy/ansible-prometheus](https://github.com/cloudalchemy/ansible-prometheus) +* [prometheus-community/ansible](https://github.com/prometheus-community/ansible) ### Chef From beb7d3b80f479aafa452828ff7d62ec78389a774 Mon Sep 17 00:00:00 2001 From: Oleg Zaytsev Date: Thu, 16 Mar 2023 09:36:19 +0100 Subject: [PATCH 016/231] remote.Client: store urlString During remote write, we call url.String() twice: - to add the Endpoint() to the span - to actually know where whe should send the request This value does not change over time, and it's not really that lightweight to calculate. I wrote this simple benchmark: func BenchmarkURLString(b *testing.B) { u, err := url.Parse("https://remote.write.com/api/v1") require.NoError(b, err) b.Run("string", func(b *testing.B) { count := 0 for i := 0; i < b.N; i++ { count += len(u.String()) } }) } And the results are ~200ns/op, 80B/op, 3 allocs/op. Yes, we're going to go to the network here, which is a huge amount of resources compared to this, but still, on agents that send 500 requests per second, that is 1500 wasteful allocations per second. Signed-off-by: Oleg Zaytsev --- storage/remote/client.go | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/storage/remote/client.go b/storage/remote/client.go index 92666cd1d1..1625c9918d 100644 --- a/storage/remote/client.go +++ b/storage/remote/client.go @@ -80,7 +80,7 @@ func init() { // Client allows reading and writing from/to a remote HTTP endpoint. type Client struct { remoteName string // Used to differentiate clients in metrics. - url *config_util.URL + urlString string // url.String() Client *http.Client timeout time.Duration @@ -122,7 +122,7 @@ func NewReadClient(name string, conf *ClientConfig) (ReadClient, error) { return &Client{ remoteName: name, - url: conf.URL, + urlString: conf.URL.String(), Client: httpClient, timeout: time.Duration(conf.Timeout), readQueries: remoteReadQueries.WithLabelValues(name, conf.URL.String()), @@ -154,7 +154,7 @@ func NewWriteClient(name string, conf *ClientConfig) (WriteClient, error) { return &Client{ remoteName: name, - url: conf.URL, + urlString: conf.URL.String(), Client: httpClient, retryOnRateLimit: conf.RetryOnRateLimit, timeout: time.Duration(conf.Timeout), @@ -187,7 +187,7 @@ type RecoverableError struct { // Store sends a batch of samples to the HTTP endpoint, the request is the proto marshalled // and encoded bytes from codec.go. func (c *Client) Store(ctx context.Context, req []byte) error { - httpReq, err := http.NewRequest("POST", c.url.String(), bytes.NewReader(req)) + httpReq, err := http.NewRequest("POST", c.urlString, bytes.NewReader(req)) if err != nil { // Errors from NewRequest are from unparsable URLs, so are not // recoverable. @@ -255,7 +255,7 @@ func (c Client) Name() string { // Endpoint is the remote read or write endpoint. func (c Client) Endpoint() string { - return c.url.String() + return c.urlString } // Read reads from a remote endpoint. @@ -276,7 +276,7 @@ func (c *Client) Read(ctx context.Context, query *prompb.Query) (*prompb.QueryRe } compressed := snappy.Encode(nil, data) - httpReq, err := http.NewRequest("POST", c.url.String(), bytes.NewReader(compressed)) + httpReq, err := http.NewRequest("POST", c.urlString, bytes.NewReader(compressed)) if err != nil { return nil, fmt.Errorf("unable to create request: %w", err) } @@ -310,7 +310,7 @@ func (c *Client) Read(ctx context.Context, query *prompb.Query) (*prompb.QueryRe } if httpResp.StatusCode/100 != 2 { - return nil, fmt.Errorf("remote server %s returned HTTP status %s: %s", c.url.String(), httpResp.Status, strings.TrimSpace(string(compressed))) + return nil, fmt.Errorf("remote server %s returned HTTP status %s: %s", c.urlString, httpResp.Status, strings.TrimSpace(string(compressed))) } uncompressed, err := snappy.Decode(nil, compressed) From 8f6d5dcd4516ec9ef99b079a6f8b786b91534995 Mon Sep 17 00:00:00 2001 From: Abhijit Mukherjee Date: Thu, 16 Mar 2023 15:53:47 +0530 Subject: [PATCH 017/231] Fix: getting rid of EncOOOXOR chunk encoding (#12111) Signed-off-by: mabhi --- tsdb/chunkenc/chunk.go | 19 +++-------- tsdb/chunkenc/xor.go | 9 ----- tsdb/chunks/chunk_write_queue.go | 5 +-- tsdb/chunks/chunk_write_queue_test.go | 10 +++--- tsdb/chunks/head_chunks.go | 49 +++++++++++++++++++++------ tsdb/chunks/head_chunks_test.go | 25 ++++++++------ tsdb/head.go | 3 +- tsdb/head_append.go | 5 ++- tsdb/head_test.go | 4 +-- 9 files changed, 70 insertions(+), 59 deletions(-) diff --git a/tsdb/chunkenc/chunk.go b/tsdb/chunkenc/chunk.go index b7d240123b..c550cbc78e 100644 --- a/tsdb/chunkenc/chunk.go +++ b/tsdb/chunkenc/chunk.go @@ -47,20 +47,9 @@ func (e Encoding) String() string { return "" } -// Chunk encodings for out-of-order chunks. -// These encodings must be only used by the Head block for its internal bookkeeping. -const ( - OutOfOrderMask = 0b10000000 - EncOOOXOR = EncXOR | OutOfOrderMask -) - -func IsOutOfOrderChunk(e Encoding) bool { - return (e & OutOfOrderMask) != 0 -} - // IsValidEncoding returns true for supported encodings. func IsValidEncoding(e Encoding) bool { - return e == EncXOR || e == EncOOOXOR || e == EncHistogram || e == EncFloatHistogram + return e == EncXOR || e == EncHistogram || e == EncFloatHistogram } // Chunk holds a sequence of sample pairs that can be iterated over and appended to. @@ -262,7 +251,7 @@ func NewPool() Pool { func (p *pool) Get(e Encoding, b []byte) (Chunk, error) { switch e { - case EncXOR, EncOOOXOR: + case EncXOR: c := p.xor.Get().(*XORChunk) c.b.stream = b c.b.count = 0 @@ -283,7 +272,7 @@ func (p *pool) Get(e Encoding, b []byte) (Chunk, error) { func (p *pool) Put(c Chunk) error { switch c.Encoding() { - case EncXOR, EncOOOXOR: + case EncXOR: xc, ok := c.(*XORChunk) // This may happen often with wrapped chunks. Nothing we can really do about // it but returning an error would cause a lot of allocations again. Thus, @@ -327,7 +316,7 @@ func (p *pool) Put(c Chunk) error { // bytes. func FromData(e Encoding, d []byte) (Chunk, error) { switch e { - case EncXOR, EncOOOXOR: + case EncXOR: return &XORChunk{b: bstream{count: 0, stream: d}}, nil case EncHistogram: return &HistogramChunk{b: bstream{count: 0, stream: d}}, nil diff --git a/tsdb/chunkenc/xor.go b/tsdb/chunkenc/xor.go index 62e90cbaae..2fa2f613cb 100644 --- a/tsdb/chunkenc/xor.go +++ b/tsdb/chunkenc/xor.go @@ -506,12 +506,3 @@ func xorRead(br *bstreamReader, value *float64, leading, trailing *uint8) error *value = math.Float64frombits(vbits) return nil } - -// OOOXORChunk holds a XORChunk and overrides the Encoding() method. -type OOOXORChunk struct { - *XORChunk -} - -func (c *OOOXORChunk) Encoding() Encoding { - return EncOOOXOR -} diff --git a/tsdb/chunks/chunk_write_queue.go b/tsdb/chunks/chunk_write_queue.go index ab34eb06c7..6d2dc743b0 100644 --- a/tsdb/chunks/chunk_write_queue.go +++ b/tsdb/chunks/chunk_write_queue.go @@ -42,6 +42,7 @@ type chunkWriteJob struct { maxt int64 chk chunkenc.Chunk ref ChunkDiskMapperRef + isOOO bool callback func(error) } @@ -76,7 +77,7 @@ type chunkWriteQueue struct { } // writeChunkF is a function which writes chunks, it is dynamic to allow mocking in tests. -type writeChunkF func(HeadSeriesRef, int64, int64, chunkenc.Chunk, ChunkDiskMapperRef, bool) error +type writeChunkF func(HeadSeriesRef, int64, int64, chunkenc.Chunk, ChunkDiskMapperRef, bool, bool) error func newChunkWriteQueue(reg prometheus.Registerer, size int, writeChunk writeChunkF) *chunkWriteQueue { counters := prometheus.NewCounterVec( @@ -133,7 +134,7 @@ func (c *chunkWriteQueue) start() { } func (c *chunkWriteQueue) processJob(job chunkWriteJob) { - err := c.writeChunk(job.seriesRef, job.mint, job.maxt, job.chk, job.ref, job.cutFile) + err := c.writeChunk(job.seriesRef, job.mint, job.maxt, job.chk, job.ref, job.isOOO, job.cutFile) if job.callback != nil { job.callback(err) } diff --git a/tsdb/chunks/chunk_write_queue_test.go b/tsdb/chunks/chunk_write_queue_test.go index a55896a6d6..c908d47f5b 100644 --- a/tsdb/chunks/chunk_write_queue_test.go +++ b/tsdb/chunks/chunk_write_queue_test.go @@ -31,7 +31,7 @@ func TestChunkWriteQueue_GettingChunkFromQueue(t *testing.T) { blockWriterWg.Add(1) // blockingChunkWriter blocks until blockWriterWg is done. - blockingChunkWriter := func(_ HeadSeriesRef, _, _ int64, _ chunkenc.Chunk, _ ChunkDiskMapperRef, _ bool) error { + blockingChunkWriter := func(_ HeadSeriesRef, _, _ int64, _ chunkenc.Chunk, _ ChunkDiskMapperRef, _, _ bool) error { blockWriterWg.Wait() return nil } @@ -63,7 +63,7 @@ func TestChunkWriteQueue_WritingThroughQueue(t *testing.T) { gotCutFile bool ) - blockingChunkWriter := func(seriesRef HeadSeriesRef, mint, maxt int64, chunk chunkenc.Chunk, ref ChunkDiskMapperRef, cutFile bool) error { + blockingChunkWriter := func(seriesRef HeadSeriesRef, mint, maxt int64, chunk chunkenc.Chunk, ref ChunkDiskMapperRef, isOOO, cutFile bool) error { gotSeriesRef = seriesRef gotMint = mint gotMaxt = maxt @@ -101,7 +101,7 @@ func TestChunkWriteQueue_WrappingAroundSizeLimit(t *testing.T) { unblockChunkWriterCh := make(chan struct{}, sizeLimit) // blockingChunkWriter blocks until the unblockChunkWriterCh channel returns a value. - blockingChunkWriter := func(seriesRef HeadSeriesRef, mint, maxt int64, chunk chunkenc.Chunk, ref ChunkDiskMapperRef, cutFile bool) error { + blockingChunkWriter := func(seriesRef HeadSeriesRef, mint, maxt int64, chunk chunkenc.Chunk, ref ChunkDiskMapperRef, isOOO, cutFile bool) error { <-unblockChunkWriterCh return nil } @@ -184,7 +184,7 @@ func TestChunkWriteQueue_WrappingAroundSizeLimit(t *testing.T) { func TestChunkWriteQueue_HandlerErrorViaCallback(t *testing.T) { testError := errors.New("test error") - chunkWriter := func(_ HeadSeriesRef, _, _ int64, _ chunkenc.Chunk, _ ChunkDiskMapperRef, _ bool) error { + chunkWriter := func(_ HeadSeriesRef, _, _ int64, _ chunkenc.Chunk, _ ChunkDiskMapperRef, _, _ bool) error { return testError } @@ -212,7 +212,7 @@ func BenchmarkChunkWriteQueue_addJob(b *testing.B) { for _, concurrentWrites := range []int{1, 10, 100, 1000} { b.Run(fmt.Sprintf("%d concurrent writes", concurrentWrites), func(b *testing.B) { issueReadSignal := make(chan struct{}) - q := newChunkWriteQueue(nil, 1000, func(ref HeadSeriesRef, i, i2 int64, chunk chunkenc.Chunk, ref2 ChunkDiskMapperRef, b bool) error { + q := newChunkWriteQueue(nil, 1000, func(ref HeadSeriesRef, i, i2 int64, chunk chunkenc.Chunk, ref2 ChunkDiskMapperRef, ooo, b bool) error { if withReads { select { case issueReadSignal <- struct{}{}: diff --git a/tsdb/chunks/head_chunks.go b/tsdb/chunks/head_chunks.go index a0bd735b8b..a7ff90475e 100644 --- a/tsdb/chunks/head_chunks.go +++ b/tsdb/chunks/head_chunks.go @@ -273,6 +273,26 @@ func NewChunkDiskMapper(reg prometheus.Registerer, dir string, pool chunkenc.Poo return m, m.openMMapFiles() } +// Chunk encodings for out-of-order chunks. +// These encodings must be only used by the Head block for its internal bookkeeping. +const ( + OutOfOrderMask = uint8(0b10000000) +) + +func (cdm *ChunkDiskMapper) ApplyOutOfOrderMask(sourceEncoding chunkenc.Encoding) chunkenc.Encoding { + enc := uint8(sourceEncoding) | OutOfOrderMask + return chunkenc.Encoding(enc) +} + +func (cdm *ChunkDiskMapper) IsOutOfOrderChunk(e chunkenc.Encoding) bool { + return (uint8(e) & OutOfOrderMask) != 0 +} + +func (cdm *ChunkDiskMapper) RemoveMasks(sourceEncoding chunkenc.Encoding) chunkenc.Encoding { + restored := uint8(sourceEncoding) & (^OutOfOrderMask) + return chunkenc.Encoding(restored) +} + // openMMapFiles opens all files within dir for mmapping. func (cdm *ChunkDiskMapper) openMMapFiles() (returnErr error) { cdm.mmappedChunkFiles = map[int]*mmappedChunkFile{} @@ -403,17 +423,17 @@ func repairLastChunkFile(files map[int]string) (_ map[int]string, returnErr erro // WriteChunk writes the chunk to the disk. // The returned chunk ref is the reference from where the chunk encoding starts for the chunk. -func (cdm *ChunkDiskMapper) WriteChunk(seriesRef HeadSeriesRef, mint, maxt int64, chk chunkenc.Chunk, callback func(err error)) (chkRef ChunkDiskMapperRef) { +func (cdm *ChunkDiskMapper) WriteChunk(seriesRef HeadSeriesRef, mint, maxt int64, chk chunkenc.Chunk, isOOO bool, callback func(err error)) (chkRef ChunkDiskMapperRef) { // cdm.evtlPosMtx must be held to serialize the calls to cdm.evtlPos.getNextChunkRef() and the writing of the chunk (either with or without queue). cdm.evtlPosMtx.Lock() defer cdm.evtlPosMtx.Unlock() ref, cutFile := cdm.evtlPos.getNextChunkRef(chk) if cdm.writeQueue != nil { - return cdm.writeChunkViaQueue(ref, cutFile, seriesRef, mint, maxt, chk, callback) + return cdm.writeChunkViaQueue(ref, isOOO, cutFile, seriesRef, mint, maxt, chk, callback) } - err := cdm.writeChunk(seriesRef, mint, maxt, chk, ref, cutFile) + err := cdm.writeChunk(seriesRef, mint, maxt, chk, ref, isOOO, cutFile) if callback != nil { callback(err) } @@ -421,7 +441,7 @@ func (cdm *ChunkDiskMapper) WriteChunk(seriesRef HeadSeriesRef, mint, maxt int64 return ref } -func (cdm *ChunkDiskMapper) writeChunkViaQueue(ref ChunkDiskMapperRef, cutFile bool, seriesRef HeadSeriesRef, mint, maxt int64, chk chunkenc.Chunk, callback func(err error)) (chkRef ChunkDiskMapperRef) { +func (cdm *ChunkDiskMapper) writeChunkViaQueue(ref ChunkDiskMapperRef, isOOO, cutFile bool, seriesRef HeadSeriesRef, mint, maxt int64, chk chunkenc.Chunk, callback func(err error)) (chkRef ChunkDiskMapperRef) { var err error if callback != nil { defer func() { @@ -438,13 +458,14 @@ func (cdm *ChunkDiskMapper) writeChunkViaQueue(ref ChunkDiskMapperRef, cutFile b maxt: maxt, chk: chk, ref: ref, + isOOO: isOOO, callback: callback, }) return ref } -func (cdm *ChunkDiskMapper) writeChunk(seriesRef HeadSeriesRef, mint, maxt int64, chk chunkenc.Chunk, ref ChunkDiskMapperRef, cutFile bool) (err error) { +func (cdm *ChunkDiskMapper) writeChunk(seriesRef HeadSeriesRef, mint, maxt int64, chk chunkenc.Chunk, ref ChunkDiskMapperRef, isOOO, cutFile bool) (err error) { cdm.writePathMtx.Lock() defer cdm.writePathMtx.Unlock() @@ -476,7 +497,11 @@ func (cdm *ChunkDiskMapper) writeChunk(seriesRef HeadSeriesRef, mint, maxt int64 bytesWritten += MintMaxtSize binary.BigEndian.PutUint64(cdm.byteBuf[bytesWritten:], uint64(maxt)) bytesWritten += MintMaxtSize - cdm.byteBuf[bytesWritten] = byte(chk.Encoding()) + enc := chk.Encoding() + if isOOO { + enc = cdm.ApplyOutOfOrderMask(enc) + } + cdm.byteBuf[bytesWritten] = byte(enc) bytesWritten += ChunkEncodingSize n := binary.PutUvarint(cdm.byteBuf[bytesWritten:], uint64(len(chk.Bytes()))) bytesWritten += n @@ -696,7 +721,9 @@ func (cdm *ChunkDiskMapper) Chunk(ref ChunkDiskMapperRef) (chunkenc.Chunk, error // Encoding. chkEnc := mmapFile.byteSlice.Range(chkStart, chkStart+ChunkEncodingSize)[0] - + sourceChkEnc := chunkenc.Encoding(chkEnc) + // Extract the encoding from the byte. ChunkDiskMapper uses only the last 7 bits for the encoding. + chkEnc = byte(cdm.RemoveMasks(sourceChkEnc)) // Data length. // With the minimum chunk length this should never cause us reading // over the end of the slice. @@ -762,7 +789,7 @@ func (cdm *ChunkDiskMapper) Chunk(ref ChunkDiskMapperRef) (chunkenc.Chunk, error // and runs the provided function with information about each chunk. It returns on the first error encountered. // NOTE: This method needs to be called at least once after creating ChunkDiskMapper // to set the maxt of all the file. -func (cdm *ChunkDiskMapper) IterateAllChunks(f func(seriesRef HeadSeriesRef, chunkRef ChunkDiskMapperRef, mint, maxt int64, numSamples uint16, encoding chunkenc.Encoding) error) (err error) { +func (cdm *ChunkDiskMapper) IterateAllChunks(f func(seriesRef HeadSeriesRef, chunkRef ChunkDiskMapperRef, mint, maxt int64, numSamples uint16, encoding chunkenc.Encoding, isOOO bool) error) (err error) { cdm.writePathMtx.Lock() defer cdm.writePathMtx.Unlock() @@ -860,8 +887,10 @@ func (cdm *ChunkDiskMapper) IterateAllChunks(f func(seriesRef HeadSeriesRef, chu if maxt > mmapFile.maxt { mmapFile.maxt = maxt } - - if err := f(seriesRef, chunkRef, mint, maxt, numSamples, chkEnc); err != nil { + isOOO := cdm.IsOutOfOrderChunk(chkEnc) + // Extract the encoding from the byte. ChunkDiskMapper uses only the last 7 bits for the encoding. + chkEnc = cdm.RemoveMasks(chkEnc) + if err := f(seriesRef, chunkRef, mint, maxt, numSamples, chkEnc, isOOO); err != nil { if cerr, ok := err.(*CorruptionErr); ok { cerr.Dir = cdm.dir.Name() cerr.FileIndex = segID diff --git a/tsdb/chunks/head_chunks_test.go b/tsdb/chunks/head_chunks_test.go index 0b5bc460d2..ac89ae3e59 100644 --- a/tsdb/chunks/head_chunks_test.go +++ b/tsdb/chunks/head_chunks_test.go @@ -98,7 +98,11 @@ func TestChunkDiskMapper_WriteChunk_Chunk_IterateChunks(t *testing.T) { bytesWritten += MintMaxtSize binary.BigEndian.PutUint64(buf[bytesWritten:], uint64(maxt)) bytesWritten += MintMaxtSize - buf[bytesWritten] = byte(chunk.Encoding()) + enc := chunk.Encoding() + if isOOO { + enc = hrw.ApplyOutOfOrderMask(enc) + } + buf[bytesWritten] = byte(enc) bytesWritten += ChunkEncodingSize n := binary.PutUvarint(buf[bytesWritten:], uint64(len(chunk.Bytes()))) bytesWritten += n @@ -149,7 +153,7 @@ func TestChunkDiskMapper_WriteChunk_Chunk_IterateChunks(t *testing.T) { hrw = createChunkDiskMapper(t, dir) idx := 0 - require.NoError(t, hrw.IterateAllChunks(func(seriesRef HeadSeriesRef, chunkRef ChunkDiskMapperRef, mint, maxt int64, numSamples uint16, encoding chunkenc.Encoding) error { + require.NoError(t, hrw.IterateAllChunks(func(seriesRef HeadSeriesRef, chunkRef ChunkDiskMapperRef, mint, maxt int64, numSamples uint16, encoding chunkenc.Encoding, isOOO bool) error { t.Helper() expData := expectedData[idx] @@ -158,7 +162,7 @@ func TestChunkDiskMapper_WriteChunk_Chunk_IterateChunks(t *testing.T) { require.Equal(t, expData.maxt, maxt) require.Equal(t, expData.maxt, maxt) require.Equal(t, expData.numSamples, numSamples) - require.Equal(t, expData.isOOO, chunkenc.IsOutOfOrderChunk(encoding)) + require.Equal(t, expData.isOOO, isOOO) actChunk, err := hrw.Chunk(expData.chunkRef) require.NoError(t, err) @@ -188,7 +192,7 @@ func TestChunkDiskMapper_Truncate(t *testing.T) { mint, maxt := timeRange+1, timeRange+step-1 var err error awaitCb := make(chan struct{}) - hrw.WriteChunk(1, int64(mint), int64(maxt), randomChunk(t), func(cbErr error) { + hrw.WriteChunk(1, int64(mint), int64(maxt), randomChunk(t), false, func(cbErr error) { err = cbErr close(awaitCb) }) @@ -282,7 +286,7 @@ func TestChunkDiskMapper_Truncate_PreservesFileSequence(t *testing.T) { step := 100 mint, maxt := timeRange+1, timeRange+step-1 - hrw.WriteChunk(1, int64(mint), int64(maxt), randomChunk(t), func(err error) { + hrw.WriteChunk(1, int64(mint), int64(maxt), randomChunk(t), false, func(err error) { close(awaitCb) require.NoError(t, err) }) @@ -363,7 +367,7 @@ func TestHeadReadWriter_TruncateAfterFailedIterateChunks(t *testing.T) { // Write a chunks to iterate on it later. var err error awaitCb := make(chan struct{}) - hrw.WriteChunk(1, 0, 1000, randomChunk(t), func(cbErr error) { + hrw.WriteChunk(1, 0, 1000, randomChunk(t), false, func(cbErr error) { err = cbErr close(awaitCb) }) @@ -377,7 +381,7 @@ func TestHeadReadWriter_TruncateAfterFailedIterateChunks(t *testing.T) { hrw = createChunkDiskMapper(t, dir) // Forcefully failing IterateAllChunks. - require.Error(t, hrw.IterateAllChunks(func(_ HeadSeriesRef, _ ChunkDiskMapperRef, _, _ int64, _ uint16, _ chunkenc.Encoding) error { + require.Error(t, hrw.IterateAllChunks(func(_ HeadSeriesRef, _ ChunkDiskMapperRef, _, _ int64, _ uint16, _ chunkenc.Encoding, _ bool) error { return errors.New("random error") })) @@ -396,7 +400,7 @@ func TestHeadReadWriter_ReadRepairOnEmptyLastFile(t *testing.T) { mint, maxt := timeRange+1, timeRange+step-1 var err error awaitCb := make(chan struct{}) - hrw.WriteChunk(1, int64(mint), int64(maxt), randomChunk(t), func(cbErr error) { + hrw.WriteChunk(1, int64(mint), int64(maxt), randomChunk(t), false, func(cbErr error) { err = cbErr close(awaitCb) }) @@ -489,7 +493,7 @@ func createChunkDiskMapper(t *testing.T, dir string) *ChunkDiskMapper { hrw, err := NewChunkDiskMapper(nil, dir, chunkenc.NewPool(), DefaultWriteBufferSize, writeQueueSize) require.NoError(t, err) require.False(t, hrw.fileMaxtSet) - require.NoError(t, hrw.IterateAllChunks(func(_ HeadSeriesRef, _ ChunkDiskMapperRef, _, _ int64, _ uint16, _ chunkenc.Encoding) error { + require.NoError(t, hrw.IterateAllChunks(func(_ HeadSeriesRef, _ ChunkDiskMapperRef, _, _ int64, _ uint16, _ chunkenc.Encoding, _ bool) error { return nil })) require.True(t, hrw.fileMaxtSet) @@ -517,9 +521,8 @@ func createChunk(t *testing.T, idx int, hrw *ChunkDiskMapper) (seriesRef HeadSer awaitCb := make(chan struct{}) if rand.Intn(2) == 0 { isOOO = true - chunk = &chunkenc.OOOXORChunk{XORChunk: chunk.(*chunkenc.XORChunk)} } - chunkRef = hrw.WriteChunk(seriesRef, mint, maxt, chunk, func(cbErr error) { + chunkRef = hrw.WriteChunk(seriesRef, mint, maxt, chunk, isOOO, func(cbErr error) { require.NoError(t, err) close(awaitCb) }) diff --git a/tsdb/head.go b/tsdb/head.go index b28f5aca5e..b5239bdf84 100644 --- a/tsdb/head.go +++ b/tsdb/head.go @@ -784,10 +784,9 @@ func (h *Head) loadMmappedChunks(refSeries map[chunks.HeadSeriesRef]*memSeries) mmappedChunks := map[chunks.HeadSeriesRef][]*mmappedChunk{} oooMmappedChunks := map[chunks.HeadSeriesRef][]*mmappedChunk{} var lastRef, secondLastRef chunks.ChunkDiskMapperRef - if err := h.chunkDiskMapper.IterateAllChunks(func(seriesRef chunks.HeadSeriesRef, chunkRef chunks.ChunkDiskMapperRef, mint, maxt int64, numSamples uint16, encoding chunkenc.Encoding) error { + if err := h.chunkDiskMapper.IterateAllChunks(func(seriesRef chunks.HeadSeriesRef, chunkRef chunks.ChunkDiskMapperRef, mint, maxt int64, numSamples uint16, encoding chunkenc.Encoding, isOOO bool) error { secondLastRef = lastRef lastRef = chunkRef - isOOO := chunkenc.IsOutOfOrderChunk(encoding) if !isOOO && maxt < h.minValidTime.Load() { return nil } diff --git a/tsdb/head_append.go b/tsdb/head_append.go index 8a622fafe5..e3beaae17b 100644 --- a/tsdb/head_append.go +++ b/tsdb/head_append.go @@ -1453,8 +1453,7 @@ func (s *memSeries) mmapCurrentOOOHeadChunk(chunkDiskMapper *chunks.ChunkDiskMap return 0 } xor, _ := s.ooo.oooHeadChunk.chunk.ToXOR() // Encode to XorChunk which is more compact and implements all of the needed functionality. - oooXor := &chunkenc.OOOXORChunk{XORChunk: xor} - chunkRef := chunkDiskMapper.WriteChunk(s.ref, s.ooo.oooHeadChunk.minTime, s.ooo.oooHeadChunk.maxTime, oooXor, handleChunkWriteError) + chunkRef := chunkDiskMapper.WriteChunk(s.ref, s.ooo.oooHeadChunk.minTime, s.ooo.oooHeadChunk.maxTime, xor, true, handleChunkWriteError) s.ooo.oooMmappedChunks = append(s.ooo.oooMmappedChunks, &mmappedChunk{ ref: chunkRef, numSamples: uint16(xor.NumSamples()), @@ -1471,7 +1470,7 @@ func (s *memSeries) mmapCurrentHeadChunk(chunkDiskMapper *chunks.ChunkDiskMapper return } - chunkRef := chunkDiskMapper.WriteChunk(s.ref, s.headChunk.minTime, s.headChunk.maxTime, s.headChunk.chunk, handleChunkWriteError) + chunkRef := chunkDiskMapper.WriteChunk(s.ref, s.headChunk.minTime, s.headChunk.maxTime, s.headChunk.chunk, false, handleChunkWriteError) s.mmappedChunks = append(s.mmappedChunks, &mmappedChunk{ ref: chunkRef, numSamples: uint16(s.headChunk.chunk.NumSamples()), diff --git a/tsdb/head_test.go b/tsdb/head_test.go index b5afed64b3..1a0558cce3 100644 --- a/tsdb/head_test.go +++ b/tsdb/head_test.go @@ -69,7 +69,7 @@ func newTestHead(t testing.TB, chunkRange int64, compressWAL, oooEnabled bool) ( h, err := NewHead(nil, nil, wal, nil, opts, nil) require.NoError(t, err) - require.NoError(t, h.chunkDiskMapper.IterateAllChunks(func(_ chunks.HeadSeriesRef, _ chunks.ChunkDiskMapperRef, _, _ int64, _ uint16, _ chunkenc.Encoding) error { + require.NoError(t, h.chunkDiskMapper.IterateAllChunks(func(_ chunks.HeadSeriesRef, _ chunks.ChunkDiskMapperRef, _, _ int64, _ uint16, _ chunkenc.Encoding, _ bool) error { return nil })) @@ -4177,7 +4177,7 @@ func TestHeadInit_DiscardChunksWithUnsupportedEncoding(t *testing.T) { uc := newUnsupportedChunk() // Make this chunk not overlap with the previous and the next - h.chunkDiskMapper.WriteChunk(chunks.HeadSeriesRef(seriesRef), 500, 600, uc, func(err error) { require.NoError(t, err) }) + h.chunkDiskMapper.WriteChunk(chunks.HeadSeriesRef(seriesRef), 500, 600, uc, false, func(err error) { require.NoError(t, err) }) app = h.Appender(ctx) for i := 700; i < 1200; i++ { From 71c57a129282e0c0c6d15ef91b0d8404fc4cf375 Mon Sep 17 00:00:00 2001 From: beorn7 Date: Thu, 16 Mar 2023 13:55:57 +0100 Subject: [PATCH 018/231] docs: Clarify that range selectors use a closed interval Signed-off-by: beorn7 --- docs/querying/basics.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/querying/basics.md b/docs/querying/basics.md index bc4478f628..9eb95c66eb 100644 --- a/docs/querying/basics.md +++ b/docs/querying/basics.md @@ -157,9 +157,11 @@ syntax](https://github.com/google/re2/wiki/Syntax). Range vector literals work like instant vector literals, except that they select a range of samples back from the current instant. Syntactically, a [time -duration](#time-durations) is appended in square brackets (`[]`) at the end of a -vector selector to specify how far back in time values should be fetched for -each resulting range vector element. +duration](#time-durations) is appended in square brackets (`[]`) at the end of +a vector selector to specify how far back in time values should be fetched for +each resulting range vector element. The range is a closed interval, +i.e. samples with timestamps coinciding with either boundary of the range are +still included in the selection. In this example, we select all the values we have recorded within the last 5 minutes for all time series that have the metric name `http_requests_total` and From 3c4ab7a0691f94899c8215f2cb0202c6db45a7af Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Thu, 16 Mar 2023 13:25:55 +0000 Subject: [PATCH 019/231] labels: add test for Builder.Range Including mutating the Builder being Ranged over. Signed-off-by: Bryan Boreham --- model/labels/labels_test.go | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/model/labels/labels_test.go b/model/labels/labels_test.go index 4832be3375..588a84b984 100644 --- a/model/labels/labels_test.go +++ b/model/labels/labels_test.go @@ -529,6 +529,11 @@ func TestBuilder(t *testing.T) { base: FromStrings("aaa", "111"), want: FromStrings("aaa", "111"), }, + { + base: FromStrings("aaa", "111", "bbb", "222", "ccc", "333"), + set: []Label{{"aaa", "444"}, {"bbb", "555"}, {"ccc", "666"}}, + want: FromStrings("aaa", "444", "bbb", "555", "ccc", "666"), + }, { base: FromStrings("aaa", "111", "bbb", "222", "ccc", "333"), del: []string{"bbb"}, @@ -591,7 +596,15 @@ func TestBuilder(t *testing.T) { b.Keep(tcase.keep...) } b.Del(tcase.del...) - require.Equal(t, tcase.want, b.Labels(tcase.base)) + require.Equal(t, tcase.want, b.Labels(EmptyLabels())) + + // Check what happens when we call Range and mutate the builder. + b.Range(func(l Label) { + if l.Name == "aaa" || l.Name == "bbb" { + b.Del(l.Name) + } + }) + require.Equal(t, tcase.want.BytesWithoutLabels(nil, "aaa", "bbb"), b.Labels(tcase.base).Bytes(nil)) }) } } From 3743d87c56a610fbcdd7d49e6f85e2e873d37bc5 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Thu, 16 Mar 2023 13:28:13 +0000 Subject: [PATCH 020/231] labels: cope with mutating Builder during Range call Although we had a different slice, the underlying memory was the same so any changes meant we could skip some values. Signed-off-by: Bryan Boreham --- model/labels/labels.go | 7 +++++-- model/labels/labels_string.go | 7 +++++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/model/labels/labels.go b/model/labels/labels.go index 5e06e3b8da..6de001c3ce 100644 --- a/model/labels/labels.go +++ b/model/labels/labels.go @@ -545,9 +545,12 @@ func (b *Builder) Get(n string) string { } // Range calls f on each label in the Builder. -// If f calls Set or Del on b then this may affect what callbacks subsequently happen. func (b *Builder) Range(f func(l Label)) { - origAdd, origDel := b.add, b.del + // Stack-based arrays to avoid heap allocation in most cases. + var addStack [1024]Label + var delStack [1024]string + // Take a copy of add and del, so they are unaffected by calls to Set() or Del(). + origAdd, origDel := append(addStack[:0], b.add...), append(delStack[:0], b.del...) b.base.Range(func(l Label) { if !slices.Contains(origDel, l.Name) && !contains(origAdd, l.Name) { f(l) diff --git a/model/labels/labels_string.go b/model/labels/labels_string.go index 6fe14bedc2..98db29d254 100644 --- a/model/labels/labels_string.go +++ b/model/labels/labels_string.go @@ -599,9 +599,12 @@ func (b *Builder) Get(n string) string { } // Range calls f on each label in the Builder. -// If f calls Set or Del on b then this may affect what callbacks subsequently happen. func (b *Builder) Range(f func(l Label)) { - origAdd, origDel := b.add, b.del + // Stack-based arrays to avoid heap allocation in most cases. + var addStack [1024]Label + var delStack [1024]string + // Take a copy of add and del, so they are unaffected by calls to Set() or Del(). + origAdd, origDel := append(addStack[:0], b.add...), append(delStack[:0], b.del...) b.base.Range(func(l Label) { if !slices.Contains(origDel, l.Name) && !contains(origAdd, l.Name) { f(l) From bd23e8899d022cd0461adc063546f1bd3259283d Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Thu, 16 Mar 2023 13:25:55 +0000 Subject: [PATCH 021/231] labels: add test for Builder.Range Including mutating the Builder being Ranged over. Signed-off-by: Bryan Boreham --- model/labels/labels_test.go | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/model/labels/labels_test.go b/model/labels/labels_test.go index 4832be3375..588a84b984 100644 --- a/model/labels/labels_test.go +++ b/model/labels/labels_test.go @@ -529,6 +529,11 @@ func TestBuilder(t *testing.T) { base: FromStrings("aaa", "111"), want: FromStrings("aaa", "111"), }, + { + base: FromStrings("aaa", "111", "bbb", "222", "ccc", "333"), + set: []Label{{"aaa", "444"}, {"bbb", "555"}, {"ccc", "666"}}, + want: FromStrings("aaa", "444", "bbb", "555", "ccc", "666"), + }, { base: FromStrings("aaa", "111", "bbb", "222", "ccc", "333"), del: []string{"bbb"}, @@ -591,7 +596,15 @@ func TestBuilder(t *testing.T) { b.Keep(tcase.keep...) } b.Del(tcase.del...) - require.Equal(t, tcase.want, b.Labels(tcase.base)) + require.Equal(t, tcase.want, b.Labels(EmptyLabels())) + + // Check what happens when we call Range and mutate the builder. + b.Range(func(l Label) { + if l.Name == "aaa" || l.Name == "bbb" { + b.Del(l.Name) + } + }) + require.Equal(t, tcase.want.BytesWithoutLabels(nil, "aaa", "bbb"), b.Labels(tcase.base).Bytes(nil)) }) } } From 934c520d374053b2350f24d9c6337f7873c33364 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Thu, 16 Mar 2023 13:28:13 +0000 Subject: [PATCH 022/231] labels: cope with mutating Builder during Range call Although we had a different slice, the underlying memory was the same so any changes meant we could skip some values. Signed-off-by: Bryan Boreham --- model/labels/labels.go | 7 +++++-- model/labels/labels_string.go | 7 +++++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/model/labels/labels.go b/model/labels/labels.go index 5e06e3b8da..6de001c3ce 100644 --- a/model/labels/labels.go +++ b/model/labels/labels.go @@ -545,9 +545,12 @@ func (b *Builder) Get(n string) string { } // Range calls f on each label in the Builder. -// If f calls Set or Del on b then this may affect what callbacks subsequently happen. func (b *Builder) Range(f func(l Label)) { - origAdd, origDel := b.add, b.del + // Stack-based arrays to avoid heap allocation in most cases. + var addStack [1024]Label + var delStack [1024]string + // Take a copy of add and del, so they are unaffected by calls to Set() or Del(). + origAdd, origDel := append(addStack[:0], b.add...), append(delStack[:0], b.del...) b.base.Range(func(l Label) { if !slices.Contains(origDel, l.Name) && !contains(origAdd, l.Name) { f(l) diff --git a/model/labels/labels_string.go b/model/labels/labels_string.go index 6fe14bedc2..98db29d254 100644 --- a/model/labels/labels_string.go +++ b/model/labels/labels_string.go @@ -599,9 +599,12 @@ func (b *Builder) Get(n string) string { } // Range calls f on each label in the Builder. -// If f calls Set or Del on b then this may affect what callbacks subsequently happen. func (b *Builder) Range(f func(l Label)) { - origAdd, origDel := b.add, b.del + // Stack-based arrays to avoid heap allocation in most cases. + var addStack [1024]Label + var delStack [1024]string + // Take a copy of add and del, so they are unaffected by calls to Set() or Del(). + origAdd, origDel := append(addStack[:0], b.add...), append(delStack[:0], b.del...) b.base.Range(func(l Label) { if !slices.Contains(origDel, l.Name) && !contains(origAdd, l.Name) { f(l) From 2c6168be5f96a232b5c87e18b59920ba611891a0 Mon Sep 17 00:00:00 2001 From: Julien Pivotto Date: Thu, 16 Mar 2023 20:21:40 +0100 Subject: [PATCH 023/231] Release 2.43.0-rc.1 Signed-off-by: Julien Pivotto --- CHANGELOG.md | 4 ++++ VERSION | 2 +- web/ui/module/codemirror-promql/package.json | 4 ++-- web/ui/module/lezer-promql/package.json | 2 +- web/ui/package-lock.json | 14 +++++++------- web/ui/react-app/package.json | 4 ++-- 6 files changed, 17 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cccbae7dd5..bf725d9665 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,9 @@ # Changelog +## 2.43.0-rc.1 / 2023-03-16 + +* [BUGFIX] Fixed a bug where changes in the Builder's Range function could lead to skipped labels during iteration (#12145) + ## 2.43.0-rc.0 / 2023-03-09 We are working on some performance improvements in Prometheus, which are only diff --git a/VERSION b/VERSION index 1a38590b27..a55e12bb97 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.43.0-rc.0 +2.43.0-rc.1 diff --git a/web/ui/module/codemirror-promql/package.json b/web/ui/module/codemirror-promql/package.json index 027029c15a..8d7a73ec70 100644 --- a/web/ui/module/codemirror-promql/package.json +++ b/web/ui/module/codemirror-promql/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/codemirror-promql", - "version": "0.43.0-rc.0", + "version": "0.43.0-rc.1", "description": "a CodeMirror mode for the PromQL language", "types": "dist/esm/index.d.ts", "module": "dist/esm/index.js", @@ -29,7 +29,7 @@ }, "homepage": "https://github.com/prometheus/prometheus/blob/main/web/ui/module/codemirror-promql/README.md", "dependencies": { - "@prometheus-io/lezer-promql": "0.43.0-rc.0", + "@prometheus-io/lezer-promql": "0.43.0-rc.1", "lru-cache": "^6.0.0" }, "devDependencies": { diff --git a/web/ui/module/lezer-promql/package.json b/web/ui/module/lezer-promql/package.json index b3f5ccceeb..0fb3bda0e8 100644 --- a/web/ui/module/lezer-promql/package.json +++ b/web/ui/module/lezer-promql/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/lezer-promql", - "version": "0.43.0-rc.0", + "version": "0.43.0-rc.1", "description": "lezer-based PromQL grammar", "main": "dist/index.cjs", "type": "module", diff --git a/web/ui/package-lock.json b/web/ui/package-lock.json index 6ff46f054e..736c438fd7 100644 --- a/web/ui/package-lock.json +++ b/web/ui/package-lock.json @@ -28,10 +28,10 @@ }, "module/codemirror-promql": { "name": "@prometheus-io/codemirror-promql", - "version": "0.43.0-rc.0", + "version": "0.43.0-rc.1", "license": "Apache-2.0", "dependencies": { - "@prometheus-io/lezer-promql": "0.43.0-rc.0", + "@prometheus-io/lezer-promql": "0.43.0-rc.1", "lru-cache": "^6.0.0" }, "devDependencies": { @@ -61,7 +61,7 @@ }, "module/lezer-promql": { "name": "@prometheus-io/lezer-promql", - "version": "0.43.0-rc.0", + "version": "0.43.0-rc.1", "license": "Apache-2.0", "devDependencies": { "@lezer/generator": "^1.2.2", @@ -20763,7 +20763,7 @@ }, "react-app": { "name": "@prometheus-io/app", - "version": "0.43.0-rc.0", + "version": "0.43.0-rc.1", "dependencies": { "@codemirror/autocomplete": "^6.4.0", "@codemirror/commands": "^6.2.0", @@ -20781,7 +20781,7 @@ "@lezer/lr": "^1.3.1", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.43.0-rc.0", + "@prometheus-io/codemirror-promql": "0.43.0-rc.1", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", "downshift": "^7.2.0", @@ -23417,7 +23417,7 @@ "@lezer/lr": "^1.3.1", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.43.0-rc.0", + "@prometheus-io/codemirror-promql": "0.43.0-rc.1", "@testing-library/react-hooks": "^7.0.2", "@types/enzyme": "^3.10.12", "@types/flot": "0.0.32", @@ -23468,7 +23468,7 @@ "@lezer/common": "^1.0.2", "@lezer/highlight": "^1.1.3", "@lezer/lr": "^1.3.1", - "@prometheus-io/lezer-promql": "0.43.0-rc.0", + "@prometheus-io/lezer-promql": "0.43.0-rc.1", "@types/lru-cache": "^5.1.1", "isomorphic-fetch": "^3.0.0", "lru-cache": "^6.0.0", diff --git a/web/ui/react-app/package.json b/web/ui/react-app/package.json index 30ebd4922e..20f5819bb9 100644 --- a/web/ui/react-app/package.json +++ b/web/ui/react-app/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/app", - "version": "0.43.0-rc.0", + "version": "0.43.0-rc.1", "private": true, "dependencies": { "@codemirror/autocomplete": "^6.4.0", @@ -19,7 +19,7 @@ "@lezer/common": "^1.0.2", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.43.0-rc.0", + "@prometheus-io/codemirror-promql": "0.43.0-rc.1", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", "downshift": "^7.2.0", From 331a7dfd213baae7e798bb3fabde2ad0e6a00114 Mon Sep 17 00:00:00 2001 From: Julien Pivotto Date: Thu, 16 Mar 2023 20:29:51 +0100 Subject: [PATCH 024/231] Replace '+' with '_' in docker image tag for semver compatibility This change introduces a new variable, SANITIZED_DOCKER_IMAGE_TAG, which replaces any '+' characters in the original DOCKER_IMAGE_TAG with '_' characters. This ensures better compatibility with semver standards, particularly when using metadata in version tags. Signed-off-by: Julien Pivotto --- Makefile.common | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/Makefile.common b/Makefile.common index 6d8007c951..221941db9c 100644 --- a/Makefile.common +++ b/Makefile.common @@ -91,6 +91,8 @@ BUILD_DOCKER_ARCHS = $(addprefix common-docker-,$(DOCKER_ARCHS)) PUBLISH_DOCKER_ARCHS = $(addprefix common-docker-publish-,$(DOCKER_ARCHS)) TAG_DOCKER_ARCHS = $(addprefix common-docker-tag-latest-,$(DOCKER_ARCHS)) +SANITIZED_DOCKER_IMAGE_TAG := $(subst +,_,$(DOCKER_IMAGE_TAG)) + ifeq ($(GOHOSTARCH),amd64) ifeq ($(GOHOSTOS),$(filter $(GOHOSTOS),linux freebsd darwin windows)) # Only supported on amd64 @@ -205,7 +207,7 @@ common-tarball: promu .PHONY: common-docker $(BUILD_DOCKER_ARCHS) common-docker: $(BUILD_DOCKER_ARCHS) $(BUILD_DOCKER_ARCHS): common-docker-%: - docker build -t "$(DOCKER_REPO)/$(DOCKER_IMAGE_NAME)-linux-$*:$(DOCKER_IMAGE_TAG)" \ + docker build -t "$(DOCKER_REPO)/$(DOCKER_IMAGE_NAME)-linux-$*:$(SANITIZED_DOCKER_IMAGE_TAG)" \ -f $(DOCKERFILE_PATH) \ --build-arg ARCH="$*" \ --build-arg OS="linux" \ @@ -214,19 +216,19 @@ $(BUILD_DOCKER_ARCHS): common-docker-%: .PHONY: common-docker-publish $(PUBLISH_DOCKER_ARCHS) common-docker-publish: $(PUBLISH_DOCKER_ARCHS) $(PUBLISH_DOCKER_ARCHS): common-docker-publish-%: - docker push "$(DOCKER_REPO)/$(DOCKER_IMAGE_NAME)-linux-$*:$(DOCKER_IMAGE_TAG)" + docker push "$(DOCKER_REPO)/$(DOCKER_IMAGE_NAME)-linux-$*:$(SANITIZED_DOCKER_IMAGE_TAG)" DOCKER_MAJOR_VERSION_TAG = $(firstword $(subst ., ,$(shell cat VERSION))) .PHONY: common-docker-tag-latest $(TAG_DOCKER_ARCHS) common-docker-tag-latest: $(TAG_DOCKER_ARCHS) $(TAG_DOCKER_ARCHS): common-docker-tag-latest-%: - docker tag "$(DOCKER_REPO)/$(DOCKER_IMAGE_NAME)-linux-$*:$(DOCKER_IMAGE_TAG)" "$(DOCKER_REPO)/$(DOCKER_IMAGE_NAME)-linux-$*:latest" - docker tag "$(DOCKER_REPO)/$(DOCKER_IMAGE_NAME)-linux-$*:$(DOCKER_IMAGE_TAG)" "$(DOCKER_REPO)/$(DOCKER_IMAGE_NAME)-linux-$*:v$(DOCKER_MAJOR_VERSION_TAG)" + docker tag "$(DOCKER_REPO)/$(DOCKER_IMAGE_NAME)-linux-$*:$(SANITIZED_DOCKER_IMAGE_TAG)" "$(DOCKER_REPO)/$(DOCKER_IMAGE_NAME)-linux-$*:latest" + docker tag "$(DOCKER_REPO)/$(DOCKER_IMAGE_NAME)-linux-$*:$(SANITIZED_DOCKER_IMAGE_TAG)" "$(DOCKER_REPO)/$(DOCKER_IMAGE_NAME)-linux-$*:v$(DOCKER_MAJOR_VERSION_TAG)" .PHONY: common-docker-manifest common-docker-manifest: - DOCKER_CLI_EXPERIMENTAL=enabled docker manifest create -a "$(DOCKER_REPO)/$(DOCKER_IMAGE_NAME):$(DOCKER_IMAGE_TAG)" $(foreach ARCH,$(DOCKER_ARCHS),$(DOCKER_REPO)/$(DOCKER_IMAGE_NAME)-linux-$(ARCH):$(DOCKER_IMAGE_TAG)) - DOCKER_CLI_EXPERIMENTAL=enabled docker manifest push "$(DOCKER_REPO)/$(DOCKER_IMAGE_NAME):$(DOCKER_IMAGE_TAG)" + DOCKER_CLI_EXPERIMENTAL=enabled docker manifest create -a "$(DOCKER_REPO)/$(DOCKER_IMAGE_NAME):$(SANITIZED_DOCKER_IMAGE_TAG)" $(foreach ARCH,$(DOCKER_ARCHS),$(DOCKER_REPO)/$(DOCKER_IMAGE_NAME)-linux-$(ARCH):$(SANITIZED_DOCKER_IMAGE_TAG)) + DOCKER_CLI_EXPERIMENTAL=enabled docker manifest push "$(DOCKER_REPO)/$(DOCKER_IMAGE_NAME):$(SANITIZED_DOCKER_IMAGE_TAG)" .PHONY: promu promu: $(PROMU) From 2fde2fb37d257f085d7abd08c1de6eda82383409 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Tue, 7 Mar 2023 17:10:15 +0000 Subject: [PATCH 025/231] scrape: add Target.LabelsRange This allows users of a Target to iterate labels without allocating heap memory. Signed-off-by: Bryan Boreham --- scrape/target.go | 9 +++++++++ scrape/target_test.go | 11 +++++++++++ 2 files changed, 20 insertions(+) diff --git a/scrape/target.go b/scrape/target.go index ae952b420a..59f6e2873e 100644 --- a/scrape/target.go +++ b/scrape/target.go @@ -181,6 +181,15 @@ func (t *Target) Labels() labels.Labels { return b.Labels() } +// LabelsRange calls f on each public label of the target. +func (t *Target) LabelsRange(f func(l labels.Label)) { + t.labels.Range(func(l labels.Label) { + if !strings.HasPrefix(l.Name, model.ReservedLabelPrefix) { + f(l) + } + }) +} + // DiscoveredLabels returns a copy of the target's labels before any processing. func (t *Target) DiscoveredLabels() labels.Labels { t.mtx.Lock() diff --git a/scrape/target_test.go b/scrape/target_test.go index 991195f5b8..4937359ed7 100644 --- a/scrape/target_test.go +++ b/scrape/target_test.go @@ -43,6 +43,17 @@ func TestTargetLabels(t *testing.T) { want := labels.FromStrings(model.JobLabel, "some_job", "foo", "bar") got := target.Labels() require.Equal(t, want, got) + i := 0 + target.LabelsRange(func(l labels.Label) { + switch i { + case 0: + require.Equal(t, labels.Label{Name: "foo", Value: "bar"}, l) + case 1: + require.Equal(t, labels.Label{Name: model.JobLabel, Value: "some_job"}, l) + } + i++ + }) + require.Equal(t, 2, i) } func TestTargetOffset(t *testing.T) { From 0dfa1e73f8c248fa19282b56b5c8fdf58b44caf4 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Tue, 7 Mar 2023 17:11:24 +0000 Subject: [PATCH 026/231] scrape: use LabelsRange instead of Labels, for performance Includes a rewrite of `resolveConflictingExposedLabels` to use `labels.Builder.Get`, which simplifies it considerably. Signed-off-by: Bryan Boreham --- scrape/scrape.go | 37 ++++++++++++------------------------- 1 file changed, 12 insertions(+), 25 deletions(-) diff --git a/scrape/scrape.go b/scrape/scrape.go index 3fce6f9dd4..5c71a0110d 100644 --- a/scrape/scrape.go +++ b/scrape/scrape.go @@ -500,7 +500,10 @@ func (sp *scrapePool) Sync(tgs []*targetgroup.Group) { } targetSyncFailed.WithLabelValues(sp.config.JobName).Add(float64(len(failures))) for _, t := range targets { - if !t.Labels().IsEmpty() { + // Replicate .Labels().IsEmpty() with a loop here to avoid generating garbage. + nonEmpty := false + t.LabelsRange(func(l labels.Label) { nonEmpty = true }) + if nonEmpty { all = append(all, t) } else if !t.DiscoveredLabels().IsEmpty() { sp.droppedTargets = append(sp.droppedTargets, t) @@ -666,17 +669,16 @@ func verifyLabelLimits(lset labels.Labels, limits *labelLimits) error { func mutateSampleLabels(lset labels.Labels, target *Target, honor bool, rc []*relabel.Config) labels.Labels { lb := labels.NewBuilder(lset) - targetLabels := target.Labels() if honor { - targetLabels.Range(func(l labels.Label) { + target.LabelsRange(func(l labels.Label) { if !lset.Has(l.Name) { lb.Set(l.Name, l.Value) } }) } else { var conflictingExposedLabels []labels.Label - targetLabels.Range(func(l labels.Label) { + target.LabelsRange(func(l labels.Label) { existingValue := lset.Get(l.Name) if existingValue != "" { conflictingExposedLabels = append(conflictingExposedLabels, labels.Label{Name: l.Name, Value: existingValue}) @@ -686,7 +688,7 @@ func mutateSampleLabels(lset labels.Labels, target *Target, honor bool, rc []*re }) if len(conflictingExposedLabels) > 0 { - resolveConflictingExposedLabels(lb, lset, targetLabels, conflictingExposedLabels) + resolveConflictingExposedLabels(lb, conflictingExposedLabels) } } @@ -699,42 +701,27 @@ func mutateSampleLabels(lset labels.Labels, target *Target, honor bool, rc []*re return res } -func resolveConflictingExposedLabels(lb *labels.Builder, exposedLabels, targetLabels labels.Labels, conflictingExposedLabels []labels.Label) { +func resolveConflictingExposedLabels(lb *labels.Builder, conflictingExposedLabels []labels.Label) { sort.SliceStable(conflictingExposedLabels, func(i, j int) bool { return len(conflictingExposedLabels[i].Name) < len(conflictingExposedLabels[j].Name) }) - for i, l := range conflictingExposedLabels { + for _, l := range conflictingExposedLabels { newName := l.Name for { newName = model.ExportedLabelPrefix + newName - if !exposedLabels.Has(newName) && - !targetLabels.Has(newName) && - !labelSliceHas(conflictingExposedLabels[:i], newName) { - conflictingExposedLabels[i].Name = newName + if lb.Get(newName) == "" { + lb.Set(newName, l.Value) break } } } - - for _, l := range conflictingExposedLabels { - lb.Set(l.Name, l.Value) - } -} - -func labelSliceHas(lbls []labels.Label, name string) bool { - for _, l := range lbls { - if l.Name == name { - return true - } - } - return false } func mutateReportSampleLabels(lset labels.Labels, target *Target) labels.Labels { lb := labels.NewBuilder(lset) - target.Labels().Range(func(l labels.Label) { + target.LabelsRange(func(l labels.Label) { lb.Set(model.ExportedLabelPrefix+l.Name, lset.Get(l.Name)) lb.Set(l.Name, l.Value) }) From 0c09c3feb021a45a87bd99dfc44ccadfa14deb21 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Tue, 7 Mar 2023 17:17:49 +0000 Subject: [PATCH 027/231] scrape sync: avoid copy of labels for dropped targets Since the Target object was just created in this function, nobody else has a reference to it and there are no concerns about it being modified concurrently so we don't need to copy the value. Signed-off-by: Bryan Boreham --- scrape/scrape.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scrape/scrape.go b/scrape/scrape.go index 5c71a0110d..01c66ca81d 100644 --- a/scrape/scrape.go +++ b/scrape/scrape.go @@ -505,7 +505,7 @@ func (sp *scrapePool) Sync(tgs []*targetgroup.Group) { t.LabelsRange(func(l labels.Label) { nonEmpty = true }) if nonEmpty { all = append(all, t) - } else if !t.DiscoveredLabels().IsEmpty() { + } else if !t.discoveredLabels.IsEmpty() { sp.droppedTargets = append(sp.droppedTargets, t) } } From 1cc28ce9ca52453d6fc73a398158c4c135f90a47 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ren=C3=A9=20Scheibe?= Date: Sat, 18 Mar 2023 20:11:35 +0100 Subject: [PATCH 028/231] chore: Fix documentation on signal to shut down instance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: René Scheibe --- docs/getting_started.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting_started.md b/docs/getting_started.md index 11d8d0fb82..e89ac705ee 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -264,4 +264,4 @@ process ID. While Prometheus does have recovery mechanisms in the case that there is an abrupt process failure it is recommend to use the `SIGTERM` signal to cleanly shutdown a Prometheus instance. If you're running on Linux this can be performed -by using `kill -s SIGHUP `, replacing `` with your Prometheus process ID. +by using `kill -s SIGTERM `, replacing `` with your Prometheus process ID. From 1b7d973f143d27d83cd9186f6ceec33cbe3796f0 Mon Sep 17 00:00:00 2001 From: Ganesh Vernekar Date: Tue, 21 Mar 2023 15:15:36 +0530 Subject: [PATCH 029/231] tsdb: Fix a comment in tsdb/head_read.go Signed-off-by: Ganesh Vernekar --- tsdb/head_read.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tsdb/head_read.go b/tsdb/head_read.go index b54e53aa07..9c40bcd7a8 100644 --- a/tsdb/head_read.go +++ b/tsdb/head_read.go @@ -340,7 +340,7 @@ func (h *headChunkReader) chunk(meta chunks.Meta, copyLastChunk bool) (chunkenc. } // chunk returns the chunk for the HeadChunkID from memory or by m-mapping it from the disk. -// If headChunk is true, it means that the returned *memChunk +// If headChunk is false, it means that the returned *memChunk // (and not the chunkenc.Chunk inside it) can be garbage collected after its usage. func (s *memSeries) chunk(id chunks.HeadChunkID, chunkDiskMapper *chunks.ChunkDiskMapper, memChunkPool *sync.Pool) (chunk *memChunk, headChunk bool, err error) { // ix represents the index of chunk in the s.mmappedChunks slice. The chunk id's are From f14665b9e3a6a30286a83c36d7f746259e5699aa Mon Sep 17 00:00:00 2001 From: SuperQ Date: Mon, 20 Mar 2023 14:14:39 +0100 Subject: [PATCH 030/231] Fix docker tag sanitizer Use a `-` instead of `_` to make the docker tag also pass semver checks. Signed-off-by: SuperQ --- Makefile.common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile.common b/Makefile.common index 221941db9c..b111d25620 100644 --- a/Makefile.common +++ b/Makefile.common @@ -91,7 +91,7 @@ BUILD_DOCKER_ARCHS = $(addprefix common-docker-,$(DOCKER_ARCHS)) PUBLISH_DOCKER_ARCHS = $(addprefix common-docker-publish-,$(DOCKER_ARCHS)) TAG_DOCKER_ARCHS = $(addprefix common-docker-tag-latest-,$(DOCKER_ARCHS)) -SANITIZED_DOCKER_IMAGE_TAG := $(subst +,_,$(DOCKER_IMAGE_TAG)) +SANITIZED_DOCKER_IMAGE_TAG := $(subst +,-,$(DOCKER_IMAGE_TAG)) ifeq ($(GOHOSTARCH),amd64) ifeq ($(GOHOSTOS),$(filter $(GOHOSTOS),linux freebsd darwin windows)) From 1070c9b06c3d9b6a627d284e7a475bb811833643 Mon Sep 17 00:00:00 2001 From: Julien Pivotto Date: Tue, 21 Mar 2023 11:30:28 +0100 Subject: [PATCH 031/231] Release 2.43.0 Signed-off-by: Julien Pivotto --- CHANGELOG.md | 12 ++++-------- VERSION | 2 +- web/ui/module/codemirror-promql/package.json | 4 ++-- web/ui/module/lezer-promql/package.json | 2 +- web/ui/package-lock.json | 14 +++++++------- web/ui/react-app/package.json | 4 ++-- 6 files changed, 17 insertions(+), 21 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bf725d9665..38f3c995a8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,10 +1,6 @@ # Changelog -## 2.43.0-rc.1 / 2023-03-16 - -* [BUGFIX] Fixed a bug where changes in the Builder's Range function could lead to skipped labels during iteration (#12145) - -## 2.43.0-rc.0 / 2023-03-09 +## 2.43.0 / 2023-03-21 We are working on some performance improvements in Prometheus, which are only built into Prometheus when compiling it using the Go tag `stringlabels` @@ -12,9 +8,9 @@ built into Prometheus when compiling it using the Go tag `stringlabels` structure for labels that uses a single string to hold all the label/values, resulting in a smaller heap size and some speedups in most cases. We would like to encourage users who are interested in these improvements to help us measure -the gains on their production architecture. Building Prometheus from source -with the `stringlabels` Go tag and providing feedback on its effectiveness in -their specific use cases would be incredibly helpful to us. #10991 +the gains on their production architecture. We are providing release artefacts +2.43.0+stringlabels and docker images tagged v2.43.0-stringlabels with those +improvements for testing. #10991 * [FEATURE] Promtool: Add HTTP client configuration to query commands. #11487 * [FEATURE] Scrape: Add `include_scrape_configs` to include scrape configs from different files. #12019 diff --git a/VERSION b/VERSION index a55e12bb97..5b9cd9afd5 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.43.0-rc.1 +2.43.0 diff --git a/web/ui/module/codemirror-promql/package.json b/web/ui/module/codemirror-promql/package.json index 8d7a73ec70..f20a634683 100644 --- a/web/ui/module/codemirror-promql/package.json +++ b/web/ui/module/codemirror-promql/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/codemirror-promql", - "version": "0.43.0-rc.1", + "version": "0.43.0", "description": "a CodeMirror mode for the PromQL language", "types": "dist/esm/index.d.ts", "module": "dist/esm/index.js", @@ -29,7 +29,7 @@ }, "homepage": "https://github.com/prometheus/prometheus/blob/main/web/ui/module/codemirror-promql/README.md", "dependencies": { - "@prometheus-io/lezer-promql": "0.43.0-rc.1", + "@prometheus-io/lezer-promql": "0.43.0", "lru-cache": "^6.0.0" }, "devDependencies": { diff --git a/web/ui/module/lezer-promql/package.json b/web/ui/module/lezer-promql/package.json index 0fb3bda0e8..411251e56c 100644 --- a/web/ui/module/lezer-promql/package.json +++ b/web/ui/module/lezer-promql/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/lezer-promql", - "version": "0.43.0-rc.1", + "version": "0.43.0", "description": "lezer-based PromQL grammar", "main": "dist/index.cjs", "type": "module", diff --git a/web/ui/package-lock.json b/web/ui/package-lock.json index 736c438fd7..2e334cf9e4 100644 --- a/web/ui/package-lock.json +++ b/web/ui/package-lock.json @@ -28,10 +28,10 @@ }, "module/codemirror-promql": { "name": "@prometheus-io/codemirror-promql", - "version": "0.43.0-rc.1", + "version": "0.43.0", "license": "Apache-2.0", "dependencies": { - "@prometheus-io/lezer-promql": "0.43.0-rc.1", + "@prometheus-io/lezer-promql": "0.43.0", "lru-cache": "^6.0.0" }, "devDependencies": { @@ -61,7 +61,7 @@ }, "module/lezer-promql": { "name": "@prometheus-io/lezer-promql", - "version": "0.43.0-rc.1", + "version": "0.43.0", "license": "Apache-2.0", "devDependencies": { "@lezer/generator": "^1.2.2", @@ -20763,7 +20763,7 @@ }, "react-app": { "name": "@prometheus-io/app", - "version": "0.43.0-rc.1", + "version": "0.43.0", "dependencies": { "@codemirror/autocomplete": "^6.4.0", "@codemirror/commands": "^6.2.0", @@ -20781,7 +20781,7 @@ "@lezer/lr": "^1.3.1", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.43.0-rc.1", + "@prometheus-io/codemirror-promql": "0.43.0", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", "downshift": "^7.2.0", @@ -23417,7 +23417,7 @@ "@lezer/lr": "^1.3.1", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.43.0-rc.1", + "@prometheus-io/codemirror-promql": "0.43.0", "@testing-library/react-hooks": "^7.0.2", "@types/enzyme": "^3.10.12", "@types/flot": "0.0.32", @@ -23468,7 +23468,7 @@ "@lezer/common": "^1.0.2", "@lezer/highlight": "^1.1.3", "@lezer/lr": "^1.3.1", - "@prometheus-io/lezer-promql": "0.43.0-rc.1", + "@prometheus-io/lezer-promql": "0.43.0", "@types/lru-cache": "^5.1.1", "isomorphic-fetch": "^3.0.0", "lru-cache": "^6.0.0", diff --git a/web/ui/react-app/package.json b/web/ui/react-app/package.json index 20f5819bb9..c8b115582a 100644 --- a/web/ui/react-app/package.json +++ b/web/ui/react-app/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/app", - "version": "0.43.0-rc.1", + "version": "0.43.0", "private": true, "dependencies": { "@codemirror/autocomplete": "^6.4.0", @@ -19,7 +19,7 @@ "@lezer/common": "^1.0.2", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.43.0-rc.1", + "@prometheus-io/codemirror-promql": "0.43.0", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", "downshift": "^7.2.0", From 67f48f47c7d2e3cb91b06adf190a275ed89b5081 Mon Sep 17 00:00:00 2001 From: Julien Pivotto Date: Tue, 21 Mar 2023 13:50:48 +0100 Subject: [PATCH 032/231] Address review comments Signed-off-by: Julien Pivotto --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 38f3c995a8..13bdde0316 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ structure for labels that uses a single string to hold all the label/values, resulting in a smaller heap size and some speedups in most cases. We would like to encourage users who are interested in these improvements to help us measure the gains on their production architecture. We are providing release artefacts -2.43.0+stringlabels and docker images tagged v2.43.0-stringlabels with those +`2.43.0+stringlabels` and Docker images tagged `v2.43.0-stringlabels` with those improvements for testing. #10991 * [FEATURE] Promtool: Add HTTP client configuration to query commands. #11487 From ca0abf26c526377e06a0024c0a1f410b601e6b0f Mon Sep 17 00:00:00 2001 From: Vernon Miller <96601789+aldernero@users.noreply.github.com> Date: Tue, 21 Mar 2023 08:03:43 -0600 Subject: [PATCH 033/231] Adds an affirmative log message for successful WAL repair (#12135) * Adds an affirmative log message for successful WAL repair Signed-off-by: Vernon Miller Signed-off-by: Vernon Miller <96601789+aldernero@users.noreply.github.com> Co-authored-by: Ganesh Vernekar --- tsdb/agent/db.go | 1 + tsdb/db.go | 2 ++ 2 files changed, 3 insertions(+) diff --git a/tsdb/agent/db.go b/tsdb/agent/db.go index da74fe4c9d..a17e0d1b98 100644 --- a/tsdb/agent/db.go +++ b/tsdb/agent/db.go @@ -303,6 +303,7 @@ func Open(l log.Logger, reg prometheus.Registerer, rs *remote.Storage, dir strin if err := w.Repair(err); err != nil { return nil, errors.Wrap(err, "repair corrupted WAL") } + level.Info(db.logger).Log("msg", "successfully repaired WAL") } go db.run() diff --git a/tsdb/db.go b/tsdb/db.go index 561867025b..659251c3ca 100644 --- a/tsdb/db.go +++ b/tsdb/db.go @@ -828,11 +828,13 @@ func open(dir string, l log.Logger, r prometheus.Registerer, opts *Options, rngs if err := wbl.Repair(initErr); err != nil { return nil, errors.Wrap(err, "repair corrupted OOO WAL") } + level.Info(db.logger).Log("msg", "Successfully repaired OOO WAL") } else { level.Warn(db.logger).Log("msg", "Encountered WAL read error, attempting repair", "err", initErr) if err := wal.Repair(initErr); err != nil { return nil, errors.Wrap(err, "repair corrupted WAL") } + level.Info(db.logger).Log("msg", "Successfully repaired WAL") } } From ae220724d4594a3d6c882e1d7e85a8e737948b8f Mon Sep 17 00:00:00 2001 From: Julien Pivotto Date: Tue, 21 Mar 2023 17:27:21 +0100 Subject: [PATCH 034/231] Docs: use boolean instead of bool boolean makes the type consistent and clickable on https://prometheus.io/docs/prometheus/latest/configuration/configuration/ Signed-off-by: Julien Pivotto --- docs/configuration/configuration.md | 98 ++++++++++++++--------------- docs/configuration/https.md | 2 +- 2 files changed, 50 insertions(+), 50 deletions(-) diff --git a/docs/configuration/configuration.md b/docs/configuration/configuration.md index 415ad4a279..f27f8256a5 100644 --- a/docs/configuration/configuration.md +++ b/docs/configuration/configuration.md @@ -205,7 +205,7 @@ oauth2: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] # Configures the scrape request's TLS settings. tls_config: @@ -218,7 +218,7 @@ tls_config: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -447,7 +447,7 @@ tls_config: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -535,7 +535,7 @@ oauth2: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -544,7 +544,7 @@ oauth2: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] # TLS configuration. tls_config: @@ -646,7 +646,7 @@ oauth2: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -655,7 +655,7 @@ oauth2: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] # TLS configuration. tls_config: @@ -733,7 +733,7 @@ oauth2: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -742,7 +742,7 @@ oauth2: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] # TLS configuration. tls_config: @@ -791,7 +791,7 @@ host: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -849,7 +849,7 @@ oauth2: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] ``` @@ -966,7 +966,7 @@ host: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -1026,7 +1026,7 @@ oauth2: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] ``` @@ -1173,7 +1173,7 @@ oauth2: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -1182,7 +1182,7 @@ oauth2: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] # TLS configuration. tls_config: @@ -1448,7 +1448,7 @@ oauth2: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -1457,7 +1457,7 @@ oauth2: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] ``` See [this example Prometheus configuration file](/documentation/examples/prometheus-puppetdb.yml) @@ -1665,7 +1665,7 @@ oauth2: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -1674,7 +1674,7 @@ oauth2: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] # TLS configuration. tls_config: @@ -1759,7 +1759,7 @@ oauth2: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -1768,7 +1768,7 @@ oauth2: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] # TLS configuration. tls_config: @@ -1842,7 +1842,7 @@ oauth2: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -1851,7 +1851,7 @@ oauth2: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] # TLS configuration. tls_config: @@ -2067,7 +2067,7 @@ oauth2: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -2076,7 +2076,7 @@ oauth2: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] # TLS configuration. tls_config: @@ -2153,7 +2153,7 @@ server: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -2193,7 +2193,7 @@ oauth2: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] ``` The [relabeling phase](#relabel_config) is the preferred and more powerful way @@ -2280,7 +2280,7 @@ oauth2: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -2289,7 +2289,7 @@ oauth2: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] # TLS configuration. tls_config: @@ -2361,7 +2361,7 @@ oauth2: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -2370,7 +2370,7 @@ oauth2: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] # TLS configuration. tls_config: @@ -2456,7 +2456,7 @@ oauth2: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] # TLS configuration for connecting to marathon servers tls_config: @@ -2469,7 +2469,7 @@ tls_config: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -2567,7 +2567,7 @@ oauth2: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -2576,7 +2576,7 @@ oauth2: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] # TLS configuration. tls_config: @@ -2753,7 +2753,7 @@ tls_config: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -2762,7 +2762,7 @@ tls_config: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] # Refresh interval to re-read the app instance list. [ refresh_interval: | default = 30s ] @@ -2869,7 +2869,7 @@ tags_filter: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] # Optional proxy URL. [ proxy_url: ] @@ -2878,7 +2878,7 @@ tags_filter: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -2954,7 +2954,7 @@ oauth2: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -2963,7 +2963,7 @@ oauth2: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] # TLS configuration. tls_config: @@ -3036,7 +3036,7 @@ oauth2: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -3045,7 +3045,7 @@ oauth2: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] # TLS configuration. tls_config: @@ -3238,7 +3238,7 @@ tls_config: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -3247,7 +3247,7 @@ tls_config: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] # List of Azure service discovery configurations. azure_sd_configs: @@ -3456,7 +3456,7 @@ tls_config: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -3465,7 +3465,7 @@ tls_config: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] # Configures the queue used to write to remote storage. queue_config: @@ -3569,7 +3569,7 @@ tls_config: # contain port numbers. [ no_proxy: ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) -[ proxy_from_environment: | default: false ] +[ proxy_from_environment: | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ : [, ...] ] ] @@ -3578,7 +3578,7 @@ tls_config: [ follow_redirects: | default = true ] # Whether to enable HTTP2. -[ enable_http2: | default: true ] +[ enable_http2: | default: true ] # Whether to use the external labels as selectors for the remote read endpoint. [ filter_external_labels: | default = true ] diff --git a/docs/configuration/https.md b/docs/configuration/https.md index 0513612f5c..d31aca7b4b 100644 --- a/docs/configuration/https.md +++ b/docs/configuration/https.md @@ -64,7 +64,7 @@ tls_server_config: # client's most preferred ciphersuite, or the server's most preferred # ciphersuite. If true then the server's preference, as expressed in # the order of elements in cipher_suites, is used. - [ prefer_server_cipher_suites: | default = true ] + [ prefer_server_cipher_suites: | default = true ] # Elliptic curves that will be used in an ECDHE handshake, in preference # order. Available curves are documented in the go documentation: From 97c7fffbb870a83625a7846e84e60c577d4575e4 Mon Sep 17 00:00:00 2001 From: Filip Petkovski Date: Wed, 22 Mar 2023 10:02:10 +0100 Subject: [PATCH 035/231] parser: Allow parsing arbitrary functions In Thanos we would like to start experimenting with custom functions that are currently not part of the PromQL spec. We would do this by adding an implementation for those functions in the Thanos engine: https://github.com/thanos-community/promql-engine and allow users to decide which engine they want to use on a per-query basis. Since we use the PromQL parser from Prometheus, injecting functions in the global `Functions` variable would mean they also become available for the Prometheus engine. To avoid this side-effect, this commit exposes a Parser interface in which the supported functions can be injected as an option. If not functions are injected, the parser implementation will default to the functions defined in the global Functions variable. Signed-off-by: Filip Petkovski --- promql/parser/functions.go | 4 +- promql/parser/generated_parser.y | 2 +- promql/parser/generated_parser.y.go | 2 +- promql/parser/parse.go | 117 ++++++++++++++++++---------- promql/parser/parse_test.go | 21 ++++- 5 files changed, 97 insertions(+), 49 deletions(-) diff --git a/promql/parser/functions.go b/promql/parser/functions.go index 450021328b..479c7f635d 100644 --- a/promql/parser/functions.go +++ b/promql/parser/functions.go @@ -387,7 +387,7 @@ var Functions = map[string]*Function{ } // getFunction returns a predefined Function object for the given name. -func getFunction(name string) (*Function, bool) { - function, ok := Functions[name] +func getFunction(name string, functions map[string]*Function) (*Function, bool) { + function, ok := functions[name] return function, ok } diff --git a/promql/parser/generated_parser.y b/promql/parser/generated_parser.y index 461e854ac1..fe3adfb9a0 100644 --- a/promql/parser/generated_parser.y +++ b/promql/parser/generated_parser.y @@ -339,7 +339,7 @@ grouping_label : maybe_label function_call : IDENTIFIER function_call_body { - fn, exist := getFunction($1.Val) + fn, exist := getFunction($1.Val, yylex.(*parser).functions) if !exist{ yylex.(*parser).addParseErrf($1.PositionRange(),"unknown function with name %q", $1.Val) } diff --git a/promql/parser/generated_parser.y.go b/promql/parser/generated_parser.y.go index 3a0e8bf69b..1ac95036af 100644 --- a/promql/parser/generated_parser.y.go +++ b/promql/parser/generated_parser.y.go @@ -1210,7 +1210,7 @@ yydefault: yyDollar = yyS[yypt-2 : yypt+1] //line promql/parser/generated_parser.y:341 { - fn, exist := getFunction(yyDollar[1].item.Val) + fn, exist := getFunction(yyDollar[1].item.Val, yylex.(*parser).functions) if !exist { yylex.(*parser).addParseErrf(yyDollar[1].item.PositionRange(), "unknown function with name %q", yyDollar[1].item.Val) } diff --git a/promql/parser/parse.go b/promql/parser/parse.go index 6c37ce6fc6..7e7ec71f55 100644 --- a/promql/parser/parse.go +++ b/promql/parser/parse.go @@ -37,12 +37,20 @@ var parserPool = sync.Pool{ }, } +type Parser interface { + ParseExpr() (Expr, error) + Close() +} + type parser struct { lex Lexer inject ItemType injecting bool + // functions contains all functions supported by the parser instance. + functions map[string]*Function + // Everytime an Item is lexed that could be the end // of certain expressions its end position is stored here. lastClosing Pos @@ -53,6 +61,62 @@ type parser struct { parseErrors ParseErrors } +type Opt func(p *parser) + +func WithFunctions(functions map[string]*Function) Opt { + return func(p *parser) { + p.functions = functions + } +} + +// NewParser returns a new parser. +func NewParser(input string, opts ...Opt) *parser { + p := parserPool.Get().(*parser) + + p.functions = Functions + p.injecting = false + p.parseErrors = nil + p.generatedParserResult = nil + + // Clear lexer struct before reusing. + p.lex = Lexer{ + input: input, + state: lexStatements, + } + + // Apply user define options. + for _, opt := range opts { + opt(p) + } + + return p +} + +func (p *parser) ParseExpr() (expr Expr, err error) { + defer p.recover(&err) + + parseResult := p.parseGenerated(START_EXPRESSION) + + if parseResult != nil { + expr = parseResult.(Expr) + } + + // Only typecheck when there are no syntax errors. + if len(p.parseErrors) == 0 { + p.checkAST(expr) + } + + if len(p.parseErrors) != 0 { + err = p.parseErrors + } + + return expr, err +} + +func (p *parser) Close() { + defer parserPool.Put(p) +} + // ParseErr wraps a parsing error with line and position context. type ParseErr struct { PositionRange PositionRange @@ -105,32 +169,15 @@ func (errs ParseErrors) Error() string { // ParseExpr returns the expression parsed from the input. func ParseExpr(input string) (expr Expr, err error) { - p := newParser(input) - defer parserPool.Put(p) - defer p.recover(&err) - - parseResult := p.parseGenerated(START_EXPRESSION) - - if parseResult != nil { - expr = parseResult.(Expr) - } - - // Only typecheck when there are no syntax errors. - if len(p.parseErrors) == 0 { - p.checkAST(expr) - } - - if len(p.parseErrors) != 0 { - err = p.parseErrors - } - - return expr, err + p := NewParser(input) + defer p.Close() + return p.ParseExpr() } // ParseMetric parses the input into a metric func ParseMetric(input string) (m labels.Labels, err error) { - p := newParser(input) - defer parserPool.Put(p) + p := NewParser(input) + defer p.Close() defer p.recover(&err) parseResult := p.parseGenerated(START_METRIC) @@ -148,8 +195,8 @@ func ParseMetric(input string) (m labels.Labels, err error) { // ParseMetricSelector parses the provided textual metric selector into a list of // label matchers. func ParseMetricSelector(input string) (m []*labels.Matcher, err error) { - p := newParser(input) - defer parserPool.Put(p) + p := NewParser(input) + defer p.Close() defer p.recover(&err) parseResult := p.parseGenerated(START_METRIC_SELECTOR) @@ -164,22 +211,6 @@ func ParseMetricSelector(input string) (m []*labels.Matcher, err error) { return m, err } -// newParser returns a new parser. -func newParser(input string) *parser { - p := parserPool.Get().(*parser) - - p.injecting = false - p.parseErrors = nil - p.generatedParserResult = nil - - // Clear lexer struct before reusing. - p.lex = Lexer{ - input: input, - state: lexStatements, - } - return p -} - // SequenceValue is an omittable value in a sequence of time series values. type SequenceValue struct { Value float64 @@ -200,10 +231,10 @@ type seriesDescription struct { // ParseSeriesDesc parses the description of a time series. func ParseSeriesDesc(input string) (labels labels.Labels, values []SequenceValue, err error) { - p := newParser(input) + p := NewParser(input) p.lex.seriesDesc = true - defer parserPool.Put(p) + defer p.Close() defer p.recover(&err) parseResult := p.parseGenerated(START_SERIES_DESCRIPTION) @@ -799,7 +830,7 @@ func MustLabelMatcher(mt labels.MatchType, name, val string) *labels.Matcher { } func MustGetFunction(name string) *Function { - f, ok := getFunction(name) + f, ok := getFunction(name, Functions) if !ok { panic(fmt.Errorf("function %q does not exist", name)) } diff --git a/promql/parser/parse_test.go b/promql/parser/parse_test.go index df66d9381a..ce609a4eea 100644 --- a/promql/parser/parse_test.go +++ b/promql/parser/parse_test.go @@ -3714,7 +3714,7 @@ func TestParseSeries(t *testing.T) { } func TestRecoverParserRuntime(t *testing.T) { - p := newParser("foo bar") + p := NewParser("foo bar") var err error defer func() { @@ -3728,7 +3728,7 @@ func TestRecoverParserRuntime(t *testing.T) { } func TestRecoverParserError(t *testing.T) { - p := newParser("foo bar") + p := NewParser("foo bar") var err error e := errors.New("custom error") @@ -3776,3 +3776,20 @@ func TestExtractSelectors(t *testing.T) { require.Equal(t, expected, ExtractSelectors(expr)) } } + +func TestParseCustomFunctions(t *testing.T) { + funcs := Functions + funcs["custom_func"] = &Function{ + Name: "custom_func", + ArgTypes: []ValueType{ValueTypeMatrix}, + ReturnType: ValueTypeVector, + } + input := "custom_func(metric[1m])" + p := NewParser(input, WithFunctions(funcs)) + expr, err := p.ParseExpr() + require.NoError(t, err) + + call, ok := expr.(*Call) + require.True(t, ok) + require.Equal(t, "custom_func", call.Func.Name) +} From 3d7783e6633dcfd79078ac30e100873c6f53fca3 Mon Sep 17 00:00:00 2001 From: Filip Petkovski Date: Wed, 22 Mar 2023 10:20:16 +0100 Subject: [PATCH 036/231] Add nolint for NewParser function Signed-off-by: Filip Petkovski --- promql/parser/parse.go | 1 + 1 file changed, 1 insertion(+) diff --git a/promql/parser/parse.go b/promql/parser/parse.go index 7e7ec71f55..ad8890f29f 100644 --- a/promql/parser/parse.go +++ b/promql/parser/parse.go @@ -70,6 +70,7 @@ func WithFunctions(functions map[string]*Function) Opt { } // NewParser returns a new parser. +// nolint:revive func NewParser(input string, opts ...Opt) *parser { p := parserPool.Get().(*parser) From b987afa7ef948461100b914fb6aea5e7195fd078 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Wed, 22 Mar 2023 15:46:02 +0000 Subject: [PATCH 037/231] labels: simplify call to get Labels from Builder It took a `Labels` where the memory could be re-used, but in practice this hardly ever benefitted. Especially after converting `relabel.Process` to `relabel.ProcessBuilder`. Comparing the parameter to `nil` was a bug; `EmptyLabels` is not `nil` so the slice was reallocated multiple times by `append`. Lastly `Builder.Labels()` now estimates that the final size will depend on labels added and deleted. Signed-off-by: Bryan Boreham --- cmd/promtool/rules.go | 2 +- model/labels/labels.go | 16 +++------------- model/labels/labels_string.go | 9 ++++----- model/labels/labels_test.go | 10 +++++----- model/relabel/relabel.go | 2 +- notifier/notifier.go | 2 +- promql/engine.go | 10 +++++----- promql/functions.go | 8 ++++---- promql/parser/generated_parser.y | 2 +- promql/parser/generated_parser.y.go | 2 +- rules/alerting.go | 6 +++--- rules/recording.go | 2 +- scrape/scrape.go | 4 ++-- scrape/target.go | 4 ++-- scrape/target_test.go | 2 +- storage/remote/read.go | 2 +- tsdb/compact_test.go | 6 +++--- 17 files changed, 39 insertions(+), 50 deletions(-) diff --git a/cmd/promtool/rules.go b/cmd/promtool/rules.go index 2b5ed1d781..aedc7bcb9d 100644 --- a/cmd/promtool/rules.go +++ b/cmd/promtool/rules.go @@ -163,7 +163,7 @@ func (importer *ruleImporter) importRule(ctx context.Context, ruleExpr, ruleName }) lb.Set(labels.MetricName, ruleName) - lbls := lb.Labels(labels.EmptyLabels()) + lbls := lb.Labels() for _, value := range sample.Values { if err := app.add(ctx, lbls, timestamp.FromTime(value.Timestamp.Time()), float64(value.Value)); err != nil { diff --git a/model/labels/labels.go b/model/labels/labels.go index 6de001c3ce..4e87eb9b34 100644 --- a/model/labels/labels.go +++ b/model/labels/labels.go @@ -570,24 +570,14 @@ func contains(s []Label, n string) bool { return false } -// Labels returns the labels from the builder, adding them to res if non-nil. -// Argument res can be the same as b.base, if caller wants to overwrite that slice. +// Labels returns the labels from the builder. // If no modifications were made, the original labels are returned. -func (b *Builder) Labels(res Labels) Labels { +func (b *Builder) Labels() Labels { if len(b.del) == 0 && len(b.add) == 0 { return b.base } - if res == nil { - // In the general case, labels are removed, modified or moved - // rather than added. - res = make(Labels, 0, len(b.base)) - } else { - res = res[:0] - } - // Justification that res can be the same slice as base: in this loop - // we move forward through base, and either skip an element or assign - // it to res at its current position or an earlier position. + res := make(Labels, 0, len(b.base)-len(b.del)+len(b.add)) for _, l := range b.base { if slices.Contains(b.del, l.Name) || contains(b.add, l.Name) { continue diff --git a/model/labels/labels_string.go b/model/labels/labels_string.go index 98db29d254..9badf667c1 100644 --- a/model/labels/labels_string.go +++ b/model/labels/labels_string.go @@ -158,7 +158,7 @@ func (ls Labels) MatchLabels(on bool, names ...string) Labels { b.Del(MetricName) b.Del(names...) } - return b.Labels(EmptyLabels()) + return b.Labels() } // Hash returns a hash value for the label set. @@ -624,10 +624,9 @@ func contains(s []Label, n string) bool { return false } -// Labels returns the labels from the builder, adding them to res if non-nil. -// Argument res can be the same as b.base, if caller wants to overwrite that slice. +// Labels returns the labels from the builder. // If no modifications were made, the original labels are returned. -func (b *Builder) Labels(res Labels) Labels { +func (b *Builder) Labels() Labels { if len(b.del) == 0 && len(b.add) == 0 { return b.base } @@ -637,7 +636,7 @@ func (b *Builder) Labels(res Labels) Labels { a, d := 0, 0 bufSize := len(b.base.data) + labelsSize(b.add) - buf := make([]byte, 0, bufSize) // TODO: see if we can re-use the buffer from res. + buf := make([]byte, 0, bufSize) for pos := 0; pos < len(b.base.data); { oldPos := pos var lName string diff --git a/model/labels/labels_test.go b/model/labels/labels_test.go index 588a84b984..9e60c22516 100644 --- a/model/labels/labels_test.go +++ b/model/labels/labels_test.go @@ -596,7 +596,7 @@ func TestBuilder(t *testing.T) { b.Keep(tcase.keep...) } b.Del(tcase.del...) - require.Equal(t, tcase.want, b.Labels(EmptyLabels())) + require.Equal(t, tcase.want, b.Labels()) // Check what happens when we call Range and mutate the builder. b.Range(func(l Label) { @@ -604,7 +604,7 @@ func TestBuilder(t *testing.T) { b.Del(l.Name) } }) - require.Equal(t, tcase.want.BytesWithoutLabels(nil, "aaa", "bbb"), b.Labels(tcase.base).Bytes(nil)) + require.Equal(t, tcase.want.BytesWithoutLabels(nil, "aaa", "bbb"), b.Labels().Bytes(nil)) }) } } @@ -669,7 +669,7 @@ func BenchmarkLabels_Hash(b *testing.B) { // Label ~20B name, 50B value. b.Set(fmt.Sprintf("abcdefghijabcdefghijabcdefghij%d", i), fmt.Sprintf("abcdefghijabcdefghijabcdefghijabcdefghijabcdefghij%d", i)) } - return b.Labels(EmptyLabels()) + return b.Labels() }(), }, { @@ -680,7 +680,7 @@ func BenchmarkLabels_Hash(b *testing.B) { // Label ~50B name, 50B value. b.Set(fmt.Sprintf("abcdefghijabcdefghijabcdefghijabcdefghijabcdefghij%d", i), fmt.Sprintf("abcdefghijabcdefghijabcdefghijabcdefghijabcdefghij%d", i)) } - return b.Labels(EmptyLabels()) + return b.Labels() }(), }, { @@ -729,7 +729,7 @@ func BenchmarkBuilder(b *testing.B) { for _, l := range m { builder.Set(l.Name, l.Value) } - l = builder.Labels(EmptyLabels()) + l = builder.Labels() } require.Equal(b, 9, l.Len()) } diff --git a/model/relabel/relabel.go b/model/relabel/relabel.go index 5ef79b4a7d..5027c3963c 100644 --- a/model/relabel/relabel.go +++ b/model/relabel/relabel.go @@ -211,7 +211,7 @@ func Process(lbls labels.Labels, cfgs ...*Config) (ret labels.Labels, keep bool) if !ProcessBuilder(lb, cfgs...) { return labels.EmptyLabels(), false } - return lb.Labels(lbls), true + return lb.Labels(), true } // ProcessBuilder is like Process, but the caller passes a labels.Builder diff --git a/notifier/notifier.go b/notifier/notifier.go index 79697d0796..c3b2e5c7e0 100644 --- a/notifier/notifier.go +++ b/notifier/notifier.go @@ -359,7 +359,7 @@ func (n *Manager) Send(alerts ...*Alert) { } }) - a.Labels = lb.Labels(a.Labels) + a.Labels = lb.Labels() } alerts = n.relabelAlerts(alerts) diff --git a/promql/engine.go b/promql/engine.go index ddfb26b13f..5682ea0f50 100644 --- a/promql/engine.go +++ b/promql/engine.go @@ -2193,7 +2193,7 @@ func resultMetric(lhs, rhs labels.Labels, op parser.ItemType, matching *parser.V } } - ret := enh.lb.Labels(labels.EmptyLabels()) + ret := enh.lb.Labels() enh.resultMetric[str] = ret return ret } @@ -2233,7 +2233,7 @@ func (ev *evaluator) VectorscalarBinop(op parser.ItemType, lhs Vector, rhs Scala } func dropMetricName(l labels.Labels) labels.Labels { - return labels.NewBuilder(l).Del(labels.MetricName).Labels(labels.EmptyLabels()) + return labels.NewBuilder(l).Del(labels.MetricName).Labels() } // scalarBinop evaluates a binary operation between two Scalars. @@ -2367,7 +2367,7 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without if op == parser.COUNT_VALUES { enh.resetBuilder(metric) enh.lb.Set(valueLabel, strconv.FormatFloat(s.V, 'f', -1, 64)) - metric = enh.lb.Labels(labels.EmptyLabels()) + metric = enh.lb.Labels() // We've changed the metric so we have to recompute the grouping key. recomputeGroupingKey = true @@ -2389,10 +2389,10 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without if without { enh.lb.Del(grouping...) enh.lb.Del(labels.MetricName) - m = enh.lb.Labels(labels.EmptyLabels()) + m = enh.lb.Labels() } else if len(grouping) > 0 { enh.lb.Keep(grouping...) - m = enh.lb.Labels(labels.EmptyLabels()) + m = enh.lb.Labels() } else { m = labels.EmptyLabels() } diff --git a/promql/functions.go b/promql/functions.go index 3da38ea0f3..e1a02ac8e3 100644 --- a/promql/functions.go +++ b/promql/functions.go @@ -960,7 +960,7 @@ func funcHistogramQuantile(vals []parser.Value, args parser.Expressions, enh *Ev if !ok { sample.Metric = labels.NewBuilder(sample.Metric). Del(excludedLabels...). - Labels(labels.EmptyLabels()) + Labels() mb = &metricWithBuckets{sample.Metric, nil} enh.signatureToMetricWithBuckets[string(enh.lblBuf)] = mb @@ -1080,7 +1080,7 @@ func funcLabelReplace(vals []parser.Value, args parser.Expressions, enh *EvalNod if len(res) > 0 { lb.Set(dst, string(res)) } - outMetric = lb.Labels(labels.EmptyLabels()) + outMetric = lb.Labels() enh.Dmn[h] = outMetric } } @@ -1148,7 +1148,7 @@ func funcLabelJoin(vals []parser.Value, args parser.Expressions, enh *EvalNodeHe lb.Set(dst, strval) } - outMetric = lb.Labels(labels.EmptyLabels()) + outMetric = lb.Labels() enh.Dmn[h] = outMetric } @@ -1414,7 +1414,7 @@ func createLabelsForAbsentFunction(expr parser.Expr) labels.Labels { } } - return b.Labels(labels.EmptyLabels()) + return b.Labels() } func stringFromArg(e parser.Expr) string { diff --git a/promql/parser/generated_parser.y b/promql/parser/generated_parser.y index 461e854ac1..b1c604eeca 100644 --- a/promql/parser/generated_parser.y +++ b/promql/parser/generated_parser.y @@ -567,7 +567,7 @@ label_matcher : IDENTIFIER match_op STRING */ metric : metric_identifier label_set - { b := labels.NewBuilder($2); b.Set(labels.MetricName, $1.Val); $$ = b.Labels(labels.EmptyLabels()) } + { b := labels.NewBuilder($2); b.Set(labels.MetricName, $1.Val); $$ = b.Labels() } | label_set {$$ = $1} ; diff --git a/promql/parser/generated_parser.y.go b/promql/parser/generated_parser.y.go index 3a0e8bf69b..2cf3e06b9a 100644 --- a/promql/parser/generated_parser.y.go +++ b/promql/parser/generated_parser.y.go @@ -1494,7 +1494,7 @@ yydefault: { b := labels.NewBuilder(yyDollar[2].labels) b.Set(labels.MetricName, yyDollar[1].item.Val) - yyVAL.labels = b.Labels(labels.EmptyLabels()) + yyVAL.labels = b.Labels() } case 96: yyDollar = yyS[yypt-1 : yypt+1] diff --git a/rules/alerting.go b/rules/alerting.go index 70e2241f7a..c793b90cb1 100644 --- a/rules/alerting.go +++ b/rules/alerting.go @@ -234,7 +234,7 @@ func (r *AlertingRule) sample(alert *Alert, ts time.Time) promql.Sample { lb.Set(alertStateLabel, alert.State.String()) s := promql.Sample{ - Metric: lb.Labels(labels.EmptyLabels()), + Metric: lb.Labels(), Point: promql.Point{T: timestamp.FromTime(ts), V: 1}, } return s @@ -252,7 +252,7 @@ func (r *AlertingRule) forStateSample(alert *Alert, ts time.Time, v float64) pro lb.Set(labels.AlertName, r.name) s := promql.Sample{ - Metric: lb.Labels(labels.EmptyLabels()), + Metric: lb.Labels(), Point: promql.Point{T: timestamp.FromTime(ts), V: v}, } return s @@ -381,7 +381,7 @@ func (r *AlertingRule) Eval(ctx context.Context, ts time.Time, query QueryFunc, }) annotations := sb.Labels() - lbs := lb.Labels(labels.EmptyLabels()) + lbs := lb.Labels() h := lbs.Hash() resultFPs[h] = struct{}{} diff --git a/rules/recording.go b/rules/recording.go index a07c779bb9..b6a886cddc 100644 --- a/rules/recording.go +++ b/rules/recording.go @@ -93,7 +93,7 @@ func (rule *RecordingRule) Eval(ctx context.Context, ts time.Time, query QueryFu lb.Set(l.Name, l.Value) }) - sample.Metric = lb.Labels(sample.Metric) + sample.Metric = lb.Labels() } // Check that the rule does not produce identical metrics after applying diff --git a/scrape/scrape.go b/scrape/scrape.go index 01c66ca81d..f38527ff30 100644 --- a/scrape/scrape.go +++ b/scrape/scrape.go @@ -692,7 +692,7 @@ func mutateSampleLabels(lset labels.Labels, target *Target, honor bool, rc []*re } } - res := lb.Labels(labels.EmptyLabels()) + res := lb.Labels() if len(rc) > 0 { res, _ = relabel.Process(res, rc...) @@ -726,7 +726,7 @@ func mutateReportSampleLabels(lset labels.Labels, target *Target) labels.Labels lb.Set(l.Name, l.Value) }) - return lb.Labels(labels.EmptyLabels()) + return lb.Labels() } // appender returns an appender for ingested samples from the target. diff --git a/scrape/target.go b/scrape/target.go index 59f6e2873e..f250910c10 100644 --- a/scrape/target.go +++ b/scrape/target.go @@ -380,7 +380,7 @@ func PopulateLabels(lb *labels.Builder, cfg *config.ScrapeConfig, noDefaultPort } } - preRelabelLabels := lb.Labels(labels.EmptyLabels()) + preRelabelLabels := lb.Labels() keep := relabel.ProcessBuilder(lb, cfg.RelabelConfigs...) // Check if the target was dropped. @@ -476,7 +476,7 @@ func PopulateLabels(lb *labels.Builder, cfg *config.ScrapeConfig, noDefaultPort lb.Set(model.InstanceLabel, addr) } - res = lb.Labels(labels.EmptyLabels()) + res = lb.Labels() err = res.Validate(func(l labels.Label) error { // Check label values are valid, drop the target if not. if !model.LabelValue(l.Value).IsValid() { diff --git a/scrape/target_test.go b/scrape/target_test.go index 4937359ed7..2bc3f000c5 100644 --- a/scrape/target_test.go +++ b/scrape/target_test.go @@ -140,7 +140,7 @@ func newTestTarget(targetURL string, deadline time.Duration, lbls labels.Labels) lb.Set(model.AddressLabel, strings.TrimPrefix(targetURL, "http://")) lb.Set(model.MetricsPathLabel, "/metrics") - return &Target{labels: lb.Labels(labels.EmptyLabels())} + return &Target{labels: lb.Labels()} } func TestNewHTTPBearerToken(t *testing.T) { diff --git a/storage/remote/read.go b/storage/remote/read.go index 21524d70db..af61334f48 100644 --- a/storage/remote/read.go +++ b/storage/remote/read.go @@ -278,5 +278,5 @@ func (sf seriesFilter) Labels() labels.Labels { b := labels.NewBuilder(sf.Series.Labels()) // todo: check if this is too inefficient. b.Del(sf.toFilter...) - return b.Labels(labels.EmptyLabels()) + return b.Labels() } diff --git a/tsdb/compact_test.go b/tsdb/compact_test.go index a3b99f87a0..cb5aa01222 100644 --- a/tsdb/compact_test.go +++ b/tsdb/compact_test.go @@ -1569,20 +1569,20 @@ func TestSparseHistogramSpaceSavings(t *testing.T) { for it.Next() { numOldSeriesPerHistogram++ b := it.At() - lbls := labels.NewBuilder(ah.baseLabels).Set("le", fmt.Sprintf("%.16f", b.Upper)).Labels(labels.EmptyLabels()) + lbls := labels.NewBuilder(ah.baseLabels).Set("le", fmt.Sprintf("%.16f", b.Upper)).Labels() refs[itIdx], err = oldApp.Append(refs[itIdx], lbls, ts, float64(b.Count)) require.NoError(t, err) itIdx++ } baseName := ah.baseLabels.Get(labels.MetricName) // _count metric. - countLbls := labels.NewBuilder(ah.baseLabels).Set(labels.MetricName, baseName+"_count").Labels(labels.EmptyLabels()) + countLbls := labels.NewBuilder(ah.baseLabels).Set(labels.MetricName, baseName+"_count").Labels() _, err = oldApp.Append(0, countLbls, ts, float64(h.Count)) require.NoError(t, err) numOldSeriesPerHistogram++ // _sum metric. - sumLbls := labels.NewBuilder(ah.baseLabels).Set(labels.MetricName, baseName+"_sum").Labels(labels.EmptyLabels()) + sumLbls := labels.NewBuilder(ah.baseLabels).Set(labels.MetricName, baseName+"_sum").Labels() _, err = oldApp.Append(0, sumLbls, ts, h.Sum) require.NoError(t, err) numOldSeriesPerHistogram++ From ee1157c14ad5a182e1b1e03aaa5a5aef67b0a682 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Wed, 22 Mar 2023 17:11:24 +0000 Subject: [PATCH 038/231] labels: shrink stack arrays in Builder.Range Go spends some time initializing all the elements of these arrays to zero, so reduce the size from 1024 to 128. This is still much bigger than we ever expect for a set of labels. (If someone does have more than 128 labels it will still work, but via heap allocation.) Signed-off-by: Bryan Boreham --- model/labels/labels.go | 4 ++-- model/labels/labels_string.go | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/model/labels/labels.go b/model/labels/labels.go index 6de001c3ce..8fc401564b 100644 --- a/model/labels/labels.go +++ b/model/labels/labels.go @@ -547,8 +547,8 @@ func (b *Builder) Get(n string) string { // Range calls f on each label in the Builder. func (b *Builder) Range(f func(l Label)) { // Stack-based arrays to avoid heap allocation in most cases. - var addStack [1024]Label - var delStack [1024]string + var addStack [128]Label + var delStack [128]string // Take a copy of add and del, so they are unaffected by calls to Set() or Del(). origAdd, origDel := append(addStack[:0], b.add...), append(delStack[:0], b.del...) b.base.Range(func(l Label) { diff --git a/model/labels/labels_string.go b/model/labels/labels_string.go index 98db29d254..560b643db3 100644 --- a/model/labels/labels_string.go +++ b/model/labels/labels_string.go @@ -601,8 +601,8 @@ func (b *Builder) Get(n string) string { // Range calls f on each label in the Builder. func (b *Builder) Range(f func(l Label)) { // Stack-based arrays to avoid heap allocation in most cases. - var addStack [1024]Label - var delStack [1024]string + var addStack [128]Label + var delStack [128]string // Take a copy of add and del, so they are unaffected by calls to Set() or Del(). origAdd, origDel := append(addStack[:0], b.add...), append(delStack[:0], b.del...) b.base.Range(func(l Label) { From b73040cff6c81e10d759cbd2ea23c8484689565c Mon Sep 17 00:00:00 2001 From: Julien Pivotto Date: Thu, 23 Mar 2023 00:01:34 +0100 Subject: [PATCH 039/231] Fix changelog Signed-off-by: Julien Pivotto --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 13bdde0316..6e6b30f47a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,7 +13,7 @@ the gains on their production architecture. We are providing release artefacts improvements for testing. #10991 * [FEATURE] Promtool: Add HTTP client configuration to query commands. #11487 -* [FEATURE] Scrape: Add `include_scrape_configs` to include scrape configs from different files. #12019 +* [FEATURE] Scrape: Add `scrape_config_files` to include scrape configs from different files. #12019 * [FEATURE] HTTP client: Add `no_proxy` to exclude URLs from proxied requests. #12098 * [FEATURE] HTTP client: Add `proxy_from_enviroment` to read proxies from env variables. #12098 * [ENHANCEMENT] API: Add support for setting lookback delta per query via the API. #12088 From 2f32a9e3c3231faa74cb1ec5fe4ec6d21107541a Mon Sep 17 00:00:00 2001 From: Oleg Zaytsev Date: Thu, 23 Mar 2023 11:08:56 +0100 Subject: [PATCH 040/231] Test compaction not failed during shutdown Test that blocks are not marked as "compaction failed" during shutdown. This shouldn't happen but this test currently fails. Signed-off-by: Oleg Zaytsev --- tsdb/compact_test.go | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/tsdb/compact_test.go b/tsdb/compact_test.go index a3b99f87a0..58f339d02b 100644 --- a/tsdb/compact_test.go +++ b/tsdb/compact_test.go @@ -1186,12 +1186,11 @@ func TestCancelCompactions(t *testing.T) { require.Equal(t, 3, len(db.Blocks()), "initial block count mismatch") require.Equal(t, 0.0, prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.ran), "initial compaction counter mismatch") db.compactc <- struct{}{} // Trigger a compaction. - var start time.Time for prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.populatingBlocks) <= 0 { time.Sleep(3 * time.Millisecond) } - start = time.Now() + start := time.Now() for prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.ran) != 1 { time.Sleep(3 * time.Millisecond) } @@ -1206,21 +1205,29 @@ func TestCancelCompactions(t *testing.T) { require.Equal(t, 3, len(db.Blocks()), "initial block count mismatch") require.Equal(t, 0.0, prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.ran), "initial compaction counter mismatch") db.compactc <- struct{}{} // Trigger a compaction. - dbClosed := make(chan struct{}) for prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.populatingBlocks) <= 0 { time.Sleep(3 * time.Millisecond) } - go func() { - require.NoError(t, db.Close()) - close(dbClosed) - }() start := time.Now() - <-dbClosed + require.NoError(t, db.Close()) actT := time.Since(start) - expT := time.Duration(timeCompactionUninterrupted / 2) // Closing the db in the middle of compaction should less than half the time. + + expT := timeCompactionUninterrupted / 2 // Closing the db in the middle of compaction should less than half the time. require.True(t, actT < expT, "closing the db took more than expected. exp: <%v, act: %v", expT, actT) + + // Make sure that no blocks were marked as compaction failed. + // This checks that the `context.Canceled` error is properly checked at all levels: + // - tsdb_errors.NewMulti() should have the Is() method implemented for correct checks. + // - callers should check with errors.Is() instead of ==. + readOnlyDB, err := OpenDBReadOnly(tmpdirCopy, log.NewNopLogger()) + require.NoError(t, err) + blocks, err := readOnlyDB.Blocks() + require.NoError(t, err) + for i, b := range blocks { + require.Falsef(t, b.Meta().Compaction.Failed, "block %d (%s) should not be marked as compaction failed", i, b.Meta().ULID) + } } } From 344c6308573e5730a9050064aee1167378f0467d Mon Sep 17 00:00:00 2001 From: Oleg Zaytsev Date: Thu, 23 Mar 2023 11:10:00 +0100 Subject: [PATCH 041/231] Fix context.Canceled wrapping in compaction We need to make sure that `tsdb_errors.NewMulti` handles the errors.Is() calls properly, like it's done in grafana/dskit. Also we need to check that `errors.Is(err, context.Canceled)`, not that `err == context.Canceled`. Signed-off-by: Oleg Zaytsev --- tsdb/compact.go | 2 +- tsdb/errors/errors.go | 14 ++++++++++++++ 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/tsdb/compact.go b/tsdb/compact.go index f216ad46a4..26a7c78c8a 100644 --- a/tsdb/compact.go +++ b/tsdb/compact.go @@ -471,7 +471,7 @@ func (c *LeveledCompactor) Compact(dest string, dirs []string, open []*Block) (u } errs := tsdb_errors.NewMulti(err) - if err != context.Canceled { + if !errors.Is(err, context.Canceled) { for _, b := range bs { if err := b.setCompactionFailed(); err != nil { errs.Add(errors.Wrapf(err, "setting compaction failed for block: %s", b.Dir())) diff --git a/tsdb/errors/errors.go b/tsdb/errors/errors.go index 607a7782a2..aa0a4b1b36 100644 --- a/tsdb/errors/errors.go +++ b/tsdb/errors/errors.go @@ -16,6 +16,7 @@ package errors import ( "bytes" + "errors" "fmt" "io" ) @@ -79,6 +80,19 @@ func (es nonNilMultiError) Error() string { return buf.String() } +// Is attempts to match the provided error against errors in the error list. +// +// This function allows errors.Is to traverse the values stored in the MultiError. +// It returns true if any of the errors in the list match the target. +func (es nonNilMultiError) Is(target error) bool { + for _, err := range es.errs { + if errors.Is(err, target) { + return true + } + } + return false +} + // CloseAll closes all given closers while recording error in MultiError. func CloseAll(cs []io.Closer) error { errs := NewMulti() From 3ac49d4ae210869043e6c33e3a82f13f2f849361 Mon Sep 17 00:00:00 2001 From: Alison <32000834+AlisonB319@users.noreply.github.com> Date: Thu, 23 Mar 2023 14:25:48 -0700 Subject: [PATCH 042/231] Set SourceMap to false to fix UI in MCR builds (#12175) * Set sourceMap to false Signed-off-by: Alison Burgess * add sourcemap=false to package.json build Signed-off-by: Alison Burgess * set sourcemap to true in tsconfig Signed-off-by: Alison Burgess --------- Signed-off-by: Alison Burgess Co-authored-by: Alison Burgess --- web/ui/package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/ui/package.json b/web/ui/package.json index e5a9f751f9..22f2615ff8 100644 --- a/web/ui/package.json +++ b/web/ui/package.json @@ -2,7 +2,7 @@ "name": "prometheus-io", "private": true, "scripts": { - "build": "bash build_ui.sh --all", + "build": "GENERATE_SOURCEMAP=false bash build_ui.sh --all", "build:module": "bash build_ui.sh --build-module", "start": "npm run start -w react-app", "test": "npm run test --workspaces", From 6ddadd98b44cca7d55b27c20477123afac2201d7 Mon Sep 17 00:00:00 2001 From: Alan Protasio Date: Mon, 27 Mar 2023 04:40:45 -0700 Subject: [PATCH 043/231] Optimization on `mergedStringIter` (#12132) Optimization on NewMergedStringIter Signed-off-by: Alan Protasio --- tsdb/querier.go | 14 ++++++++++---- tsdb/querier_bench_test.go | 27 +++++++++++++++++++++++++-- 2 files changed, 35 insertions(+), 6 deletions(-) diff --git a/tsdb/querier.go b/tsdb/querier.go index b4513218e9..74c73f460d 100644 --- a/tsdb/querier.go +++ b/tsdb/querier.go @@ -958,6 +958,7 @@ type mergedStringIter struct { b index.StringIter aok, bok bool cur string + err error } func (m *mergedStringIter) Next() bool { @@ -968,29 +969,34 @@ func (m *mergedStringIter) Next() bool { if !m.aok { m.cur = m.b.At() m.bok = m.b.Next() + m.err = m.b.Err() } else if !m.bok { m.cur = m.a.At() m.aok = m.a.Next() + m.err = m.a.Err() } else if m.b.At() > m.a.At() { m.cur = m.a.At() m.aok = m.a.Next() + m.err = m.a.Err() } else if m.a.At() > m.b.At() { m.cur = m.b.At() m.bok = m.b.Next() + m.err = m.b.Err() } else { // Equal. m.cur = m.b.At() m.aok = m.a.Next() + m.err = m.a.Err() m.bok = m.b.Next() + if m.err == nil { + m.err = m.b.Err() + } } return true } func (m mergedStringIter) At() string { return m.cur } func (m mergedStringIter) Err() error { - if m.a.Err() != nil { - return m.a.Err() - } - return m.b.Err() + return m.err } // DeletedIterator wraps chunk Iterator and makes sure any deleted metrics are not returned. diff --git a/tsdb/querier_bench_test.go b/tsdb/querier_bench_test.go index 1feb48f6fe..78993d105f 100644 --- a/tsdb/querier_bench_test.go +++ b/tsdb/querier_bench_test.go @@ -19,9 +19,10 @@ import ( "strconv" "testing" - "github.com/stretchr/testify/require" - "github.com/prometheus/prometheus/model/labels" + "github.com/prometheus/prometheus/tsdb/index" + + "github.com/stretchr/testify/require" ) // Make entries ~50B in size, to emulate real-world high cardinality. @@ -202,6 +203,28 @@ func benchmarkLabelValuesWithMatchers(b *testing.B, ir IndexReader) { } } +func BenchmarkMergedStringIter(b *testing.B) { + numSymbols := 100000 + s := make([]string, numSymbols) + for i := 0; i < numSymbols; i++ { + s[i] = fmt.Sprintf("symbol%v", i) + } + + for i := 0; i < b.N; i++ { + it := NewMergedStringIter(index.NewStringListIter(s), index.NewStringListIter(s)) + for j := 0; j < 100; j++ { + it = NewMergedStringIter(it, index.NewStringListIter(s)) + } + + for it.Next() { + require.NotNil(b, it.At()) + require.NoError(b, it.Err()) + } + } + + b.ReportAllocs() +} + func BenchmarkQuerierSelect(b *testing.B) { chunkDir := b.TempDir() opts := DefaultHeadOptions() From 22806f2e83f3ba95e7036724a7061d4d0dc5a3f3 Mon Sep 17 00:00:00 2001 From: Ganesh Vernekar Date: Mon, 27 Mar 2023 17:17:57 +0530 Subject: [PATCH 044/231] Remove codesome as codeowner from promql/ Signed-off-by: Ganesh Vernekar --- .github/CODEOWNERS | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index a6df329beb..75d24cdef7 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -3,5 +3,5 @@ /storage/remote @csmarchbanks @cstyan @bwplotka @tomwilkie /discovery/kubernetes @brancz /tsdb @codesome -/promql @codesome @roidelapluie +/promql @roidelapluie /cmd/promtool @dgl From b33a382646a9265c70dd448999edf57f1c74d149 Mon Sep 17 00:00:00 2001 From: Ganesh Vernekar Date: Mon, 27 Mar 2023 20:44:03 +0530 Subject: [PATCH 045/231] Remove mistakenly added file It got added in https://github.com/prometheus/prometheus/pull/11992 Signed-off-by: Ganesh Vernekar --- tsdb/test.txt | 1 - 1 file changed, 1 deletion(-) delete mode 100644 tsdb/test.txt diff --git a/tsdb/test.txt b/tsdb/test.txt deleted file mode 100644 index a66a6fb729..0000000000 --- a/tsdb/test.txt +++ /dev/null @@ -1 +0,0 @@ -make: Nothing to be done for `test'. From cf54a14f9c13d8bed0191bd1010811d6dfe7c7b1 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Tue, 28 Mar 2023 10:51:04 +0000 Subject: [PATCH 046/231] promql: add a benchmark for topk with k > 1 I picked k = 5. Signed-off-by: Bryan Boreham --- promql/bench_test.go | 3 +++ 1 file changed, 3 insertions(+) diff --git a/promql/bench_test.go b/promql/bench_test.go index 8abfcfdd20..88025d9325 100644 --- a/promql/bench_test.go +++ b/promql/bench_test.go @@ -157,6 +157,9 @@ func rangeQueryCases() []benchCase { { expr: "topk(1, a_X)", }, + { + expr: "topk(5, a_X)", + }, // Combinations. { expr: "rate(a_X[1m]) + rate(b_X[1m])", From f2fd85df823feba4dcff666d62716a6038c0f33c Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Tue, 28 Mar 2023 10:42:15 +0000 Subject: [PATCH 047/231] promql: use faster heap method for topk/bottomk Call `Fix()` instead of `Pop()` followed by `Push()`. This is slightly faster. Signed-off-by: Bryan Boreham --- promql/engine.go | 44 ++++++++++++++++++++++---------------------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/promql/engine.go b/promql/engine.go index ddfb26b13f..ab3465707d 100644 --- a/promql/engine.go +++ b/promql/engine.go @@ -2508,39 +2508,39 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without group.value += delta * (s.V - group.mean) case parser.TOPK: - if int64(len(group.heap)) < k || group.heap[0].V < s.V || math.IsNaN(group.heap[0].V) { - if int64(len(group.heap)) == k { - if k == 1 { // For k==1 we can replace in-situ. - group.heap[0] = Sample{ - Point: Point{V: s.V}, - Metric: s.Metric, - } - break - } - heap.Pop(&group.heap) - } + // We build a heap of up to k elements, with the smallest element at heap[0]. + if int64(len(group.heap)) < k { heap.Push(&group.heap, &Sample{ Point: Point{V: s.V}, Metric: s.Metric, }) + } else if group.heap[0].V < s.V || (math.IsNaN(group.heap[0].V) && !math.IsNaN(s.V)) { + // This new element is bigger than the previous smallest element - overwrite that. + group.heap[0] = Sample{ + Point: Point{V: s.V}, + Metric: s.Metric, + } + if k > 1 { + heap.Fix(&group.heap, 0) // Maintain the heap invariant. + } } case parser.BOTTOMK: - if int64(len(group.reverseHeap)) < k || group.reverseHeap[0].V > s.V || math.IsNaN(group.reverseHeap[0].V) { - if int64(len(group.reverseHeap)) == k { - if k == 1 { // For k==1 we can replace in-situ. - group.reverseHeap[0] = Sample{ - Point: Point{V: s.V}, - Metric: s.Metric, - } - break - } - heap.Pop(&group.reverseHeap) - } + // We build a heap of up to k elements, with the biggest element at heap[0]. + if int64(len(group.reverseHeap)) < k { heap.Push(&group.reverseHeap, &Sample{ Point: Point{V: s.V}, Metric: s.Metric, }) + } else if group.reverseHeap[0].V > s.V || (math.IsNaN(group.reverseHeap[0].V) && !math.IsNaN(s.V)) { + // This new element is smaller than the previous biggest element - overwrite that. + group.reverseHeap[0] = Sample{ + Point: Point{V: s.V}, + Metric: s.Metric, + } + if k > 1 { + heap.Fix(&group.reverseHeap, 0) // Maintain the heap invariant. + } } case parser.QUANTILE: From 6e2905a4d4ff9b47b1f6d201333f5bd53633f921 Mon Sep 17 00:00:00 2001 From: Oleg Zaytsev Date: Wed, 29 Mar 2023 21:34:34 +0200 Subject: [PATCH 048/231] Use zeropool.Pool to workaround SA6002 (#12189) * Use zeropool.Pool to workaround SA6002 I built a tiny library called https://github.com/colega/zeropool to workaround the SA6002 staticheck issue. While searching for the references of that SA6002 staticheck issues on Github first results was Prometheus itself, with quite a lot of ignores of it. This changes the usages of `sync.Pool` to `zeropool.Pool[T]` where a pointer is not available. Also added a benchmark for HeadAppender Append/Commit when series already exist, which is one of the most usual cases IMO, as I didn't find any. Signed-off-by: Oleg Zaytsev * Improve BenchmarkHeadAppender with more cases Signed-off-by: Oleg Zaytsev * A little copying is better than a little dependency https://www.youtube.com/watch?v=PAAkCSZUG1c&t=9m28s Signed-off-by: Oleg Zaytsev * Fix imports order Signed-off-by: Oleg Zaytsev * Add license header Signed-off-by: Oleg Zaytsev * Copyright should be on one of the first 3 lines Signed-off-by: Oleg Zaytsev * Use require.Equal for testing I don't depend on testify in my lib, but here we have it available. Signed-off-by: Oleg Zaytsev * Avoid flaky test Signed-off-by: Oleg Zaytsev * Also use zeropool for pointsPool in engine.go Signed-off-by: Oleg Zaytsev --------- Signed-off-by: Oleg Zaytsev --- promql/engine.go | 9 +- tsdb/head.go | 15 ++-- tsdb/head_append.go | 21 ++--- tsdb/head_test.go | 47 +++++++++- tsdb/head_wal.go | 66 ++++---------- tsdb/wal.go | 28 +++--- util/zeropool/pool.go | 77 ++++++++++++++++ util/zeropool/pool_test.go | 178 +++++++++++++++++++++++++++++++++++++ 8 files changed, 345 insertions(+), 96 deletions(-) create mode 100644 util/zeropool/pool.go create mode 100644 util/zeropool/pool_test.go diff --git a/promql/engine.go b/promql/engine.go index ddfb26b13f..a93bd4fc96 100644 --- a/promql/engine.go +++ b/promql/engine.go @@ -45,6 +45,7 @@ import ( "github.com/prometheus/prometheus/storage" "github.com/prometheus/prometheus/tsdb/chunkenc" "github.com/prometheus/prometheus/util/stats" + "github.com/prometheus/prometheus/util/zeropool" ) const ( @@ -1794,18 +1795,16 @@ func (ev *evaluator) vectorSelectorSingle(it *storage.MemoizedSeriesIterator, no return t, v, h, true } -var pointPool = sync.Pool{} +var pointPool zeropool.Pool[[]Point] func getPointSlice(sz int) []Point { - p := pointPool.Get() - if p != nil { - return p.([]Point) + if p := pointPool.Get(); p != nil { + return p } return make([]Point, 0, sz) } func putPointSlice(p []Point) { - //nolint:staticcheck // Ignore SA6002 relax staticcheck verification. pointPool.Put(p[:0]) } diff --git a/tsdb/head.go b/tsdb/head.go index b5239bdf84..0ee1612778 100644 --- a/tsdb/head.go +++ b/tsdb/head.go @@ -44,6 +44,7 @@ import ( "github.com/prometheus/prometheus/tsdb/tombstones" "github.com/prometheus/prometheus/tsdb/tsdbutil" "github.com/prometheus/prometheus/tsdb/wlog" + "github.com/prometheus/prometheus/util/zeropool" ) var ( @@ -83,13 +84,13 @@ type Head struct { exemplarMetrics *ExemplarMetrics exemplars ExemplarStorage logger log.Logger - appendPool sync.Pool - exemplarsPool sync.Pool - histogramsPool sync.Pool - floatHistogramsPool sync.Pool - metadataPool sync.Pool - seriesPool sync.Pool - bytesPool sync.Pool + appendPool zeropool.Pool[[]record.RefSample] + exemplarsPool zeropool.Pool[[]exemplarWithSeriesRef] + histogramsPool zeropool.Pool[[]record.RefHistogramSample] + floatHistogramsPool zeropool.Pool[[]record.RefFloatHistogramSample] + metadataPool zeropool.Pool[[]record.RefMetadata] + seriesPool zeropool.Pool[[]*memSeries] + bytesPool zeropool.Pool[[]byte] memChunkPool sync.Pool // All series addressable by their ID or hash. diff --git a/tsdb/head_append.go b/tsdb/head_append.go index e3beaae17b..14e343f74d 100644 --- a/tsdb/head_append.go +++ b/tsdb/head_append.go @@ -199,11 +199,10 @@ func (h *Head) getAppendBuffer() []record.RefSample { if b == nil { return make([]record.RefSample, 0, 512) } - return b.([]record.RefSample) + return b } func (h *Head) putAppendBuffer(b []record.RefSample) { - //nolint:staticcheck // Ignore SA6002 safe to ignore and actually fixing it has some performance penalty. h.appendPool.Put(b[:0]) } @@ -212,7 +211,7 @@ func (h *Head) getExemplarBuffer() []exemplarWithSeriesRef { if b == nil { return make([]exemplarWithSeriesRef, 0, 512) } - return b.([]exemplarWithSeriesRef) + return b } func (h *Head) putExemplarBuffer(b []exemplarWithSeriesRef) { @@ -220,7 +219,6 @@ func (h *Head) putExemplarBuffer(b []exemplarWithSeriesRef) { return } - //nolint:staticcheck // Ignore SA6002 safe to ignore and actually fixing it has some performance penalty. h.exemplarsPool.Put(b[:0]) } @@ -229,11 +227,10 @@ func (h *Head) getHistogramBuffer() []record.RefHistogramSample { if b == nil { return make([]record.RefHistogramSample, 0, 512) } - return b.([]record.RefHistogramSample) + return b } func (h *Head) putHistogramBuffer(b []record.RefHistogramSample) { - //nolint:staticcheck // Ignore SA6002 safe to ignore and actually fixing it has some performance penalty. h.histogramsPool.Put(b[:0]) } @@ -242,11 +239,10 @@ func (h *Head) getFloatHistogramBuffer() []record.RefFloatHistogramSample { if b == nil { return make([]record.RefFloatHistogramSample, 0, 512) } - return b.([]record.RefFloatHistogramSample) + return b } func (h *Head) putFloatHistogramBuffer(b []record.RefFloatHistogramSample) { - //nolint:staticcheck // Ignore SA6002 safe to ignore and actually fixing it has some performance penalty. h.floatHistogramsPool.Put(b[:0]) } @@ -255,11 +251,10 @@ func (h *Head) getMetadataBuffer() []record.RefMetadata { if b == nil { return make([]record.RefMetadata, 0, 512) } - return b.([]record.RefMetadata) + return b } func (h *Head) putMetadataBuffer(b []record.RefMetadata) { - //nolint:staticcheck // Ignore SA6002 safe to ignore and actually fixing it has some performance penalty. h.metadataPool.Put(b[:0]) } @@ -268,11 +263,10 @@ func (h *Head) getSeriesBuffer() []*memSeries { if b == nil { return make([]*memSeries, 0, 512) } - return b.([]*memSeries) + return b } func (h *Head) putSeriesBuffer(b []*memSeries) { - //nolint:staticcheck // Ignore SA6002 safe to ignore and actually fixing it has some performance penalty. h.seriesPool.Put(b[:0]) } @@ -281,11 +275,10 @@ func (h *Head) getBytesBuffer() []byte { if b == nil { return make([]byte, 0, 1024) } - return b.([]byte) + return b } func (h *Head) putBytesBuffer(b []byte) { - //nolint:staticcheck // Ignore SA6002 safe to ignore and actually fixing it has some performance penalty. h.bytesPool.Put(b[:0]) } diff --git a/tsdb/head_test.go b/tsdb/head_test.go index 1a0558cce3..0a2a2ee6f4 100644 --- a/tsdb/head_test.go +++ b/tsdb/head_test.go @@ -79,9 +79,9 @@ func newTestHead(t testing.TB, chunkRange int64, compressWAL, oooEnabled bool) ( func BenchmarkCreateSeries(b *testing.B) { series := genSeries(b.N, 10, 0, 0) h, _ := newTestHead(b, 10000, false, false) - defer func() { + b.Cleanup(func() { require.NoError(b, h.Close()) - }() + }) b.ReportAllocs() b.ResetTimer() @@ -91,6 +91,49 @@ func BenchmarkCreateSeries(b *testing.B) { } } +func BenchmarkHeadAppender_Append_Commit_ExistingSeries(b *testing.B) { + seriesCounts := []int{100, 1000, 10000} + series := genSeries(10000, 10, 0, 0) + + for _, seriesCount := range seriesCounts { + b.Run(fmt.Sprintf("%d series", seriesCount), func(b *testing.B) { + for _, samplesPerAppend := range []int64{1, 2, 5, 100} { + b.Run(fmt.Sprintf("%d samples per append", samplesPerAppend), func(b *testing.B) { + h, _ := newTestHead(b, 10000, false, false) + b.Cleanup(func() { require.NoError(b, h.Close()) }) + + ts := int64(1000) + append := func() error { + var err error + app := h.Appender(context.Background()) + for _, s := range series[:seriesCount] { + var ref storage.SeriesRef + for sampleIndex := int64(0); sampleIndex < samplesPerAppend; sampleIndex++ { + ref, err = app.Append(ref, s.Labels(), ts+sampleIndex, float64(ts+sampleIndex)) + if err != nil { + return err + } + } + } + ts += 1000 // should increment more than highest samplesPerAppend + return app.Commit() + } + + // Init series, that's not what we're benchmarking here. + require.NoError(b, append()) + + b.ReportAllocs() + b.ResetTimer() + + for i := 0; i < b.N; i++ { + require.NoError(b, append()) + } + }) + } + }) + } +} + func populateTestWAL(t testing.TB, w *wlog.WL, recs []interface{}) { var enc record.Encoder for _, r := range recs { diff --git a/tsdb/head_wal.go b/tsdb/head_wal.go index dd55f438d8..6a8a30d5ad 100644 --- a/tsdb/head_wal.go +++ b/tsdb/head_wal.go @@ -40,6 +40,7 @@ import ( "github.com/prometheus/prometheus/tsdb/record" "github.com/prometheus/prometheus/tsdb/tombstones" "github.com/prometheus/prometheus/tsdb/wlog" + "github.com/prometheus/prometheus/util/zeropool" ) // histogramRecord combines both RefHistogramSample and RefFloatHistogramSample @@ -74,41 +75,14 @@ func (h *Head) loadWAL(r *wlog.Reader, multiRef map[chunks.HeadSeriesRef]chunks. decoded = make(chan interface{}, 10) decodeErr, seriesCreationErr error - seriesPool = sync.Pool{ - New: func() interface{} { - return []record.RefSeries{} - }, - } - samplesPool = sync.Pool{ - New: func() interface{} { - return []record.RefSample{} - }, - } - tstonesPool = sync.Pool{ - New: func() interface{} { - return []tombstones.Stone{} - }, - } - exemplarsPool = sync.Pool{ - New: func() interface{} { - return []record.RefExemplar{} - }, - } - histogramsPool = sync.Pool{ - New: func() interface{} { - return []record.RefHistogramSample{} - }, - } - floatHistogramsPool = sync.Pool{ - New: func() interface{} { - return []record.RefFloatHistogramSample{} - }, - } - metadataPool = sync.Pool{ - New: func() interface{} { - return []record.RefMetadata{} - }, - } + + seriesPool zeropool.Pool[[]record.RefSeries] + samplesPool zeropool.Pool[[]record.RefSample] + tstonesPool zeropool.Pool[[]tombstones.Stone] + exemplarsPool zeropool.Pool[[]record.RefExemplar] + histogramsPool zeropool.Pool[[]record.RefHistogramSample] + floatHistogramsPool zeropool.Pool[[]record.RefFloatHistogramSample] + metadataPool zeropool.Pool[[]record.RefMetadata] ) defer func() { @@ -167,7 +141,7 @@ func (h *Head) loadWAL(r *wlog.Reader, multiRef map[chunks.HeadSeriesRef]chunks. rec := r.Record() switch dec.Type(rec) { case record.Series: - series := seriesPool.Get().([]record.RefSeries)[:0] + series := seriesPool.Get()[:0] series, err = dec.Series(rec, series) if err != nil { decodeErr = &wlog.CorruptionErr{ @@ -179,7 +153,7 @@ func (h *Head) loadWAL(r *wlog.Reader, multiRef map[chunks.HeadSeriesRef]chunks. } decoded <- series case record.Samples: - samples := samplesPool.Get().([]record.RefSample)[:0] + samples := samplesPool.Get()[:0] samples, err = dec.Samples(rec, samples) if err != nil { decodeErr = &wlog.CorruptionErr{ @@ -191,7 +165,7 @@ func (h *Head) loadWAL(r *wlog.Reader, multiRef map[chunks.HeadSeriesRef]chunks. } decoded <- samples case record.Tombstones: - tstones := tstonesPool.Get().([]tombstones.Stone)[:0] + tstones := tstonesPool.Get()[:0] tstones, err = dec.Tombstones(rec, tstones) if err != nil { decodeErr = &wlog.CorruptionErr{ @@ -203,7 +177,7 @@ func (h *Head) loadWAL(r *wlog.Reader, multiRef map[chunks.HeadSeriesRef]chunks. } decoded <- tstones case record.Exemplars: - exemplars := exemplarsPool.Get().([]record.RefExemplar)[:0] + exemplars := exemplarsPool.Get()[:0] exemplars, err = dec.Exemplars(rec, exemplars) if err != nil { decodeErr = &wlog.CorruptionErr{ @@ -215,7 +189,7 @@ func (h *Head) loadWAL(r *wlog.Reader, multiRef map[chunks.HeadSeriesRef]chunks. } decoded <- exemplars case record.HistogramSamples: - hists := histogramsPool.Get().([]record.RefHistogramSample)[:0] + hists := histogramsPool.Get()[:0] hists, err = dec.HistogramSamples(rec, hists) if err != nil { decodeErr = &wlog.CorruptionErr{ @@ -227,7 +201,7 @@ func (h *Head) loadWAL(r *wlog.Reader, multiRef map[chunks.HeadSeriesRef]chunks. } decoded <- hists case record.FloatHistogramSamples: - hists := floatHistogramsPool.Get().([]record.RefFloatHistogramSample)[:0] + hists := floatHistogramsPool.Get()[:0] hists, err = dec.FloatHistogramSamples(rec, hists) if err != nil { decodeErr = &wlog.CorruptionErr{ @@ -239,7 +213,7 @@ func (h *Head) loadWAL(r *wlog.Reader, multiRef map[chunks.HeadSeriesRef]chunks. } decoded <- hists case record.Metadata: - meta := metadataPool.Get().([]record.RefMetadata)[:0] + meta := metadataPool.Get()[:0] meta, err := dec.Metadata(rec, meta) if err != nil { decodeErr = &wlog.CorruptionErr{ @@ -278,7 +252,6 @@ Outer: idx := uint64(mSeries.ref) % uint64(concurrency) processors[idx].input <- walSubsetProcessorInputItem{walSeriesRef: walSeries.Ref, existingSeries: mSeries} } - //nolint:staticcheck // Ignore SA6002 relax staticcheck verification. seriesPool.Put(v) case []record.RefSample: samples := v @@ -315,7 +288,6 @@ Outer: } samples = samples[m:] } - //nolint:staticcheck // Ignore SA6002 relax staticcheck verification. samplesPool.Put(v) case []tombstones.Stone: for _, s := range v { @@ -330,13 +302,11 @@ Outer: h.tombstones.AddInterval(storage.SeriesRef(s.Ref), itv) } } - //nolint:staticcheck // Ignore SA6002 relax staticcheck verification. tstonesPool.Put(v) case []record.RefExemplar: for _, e := range v { exemplarsInput <- e } - //nolint:staticcheck // Ignore SA6002 relax staticcheck verification. exemplarsPool.Put(v) case []record.RefHistogramSample: samples := v @@ -373,7 +343,6 @@ Outer: } samples = samples[m:] } - //nolint:staticcheck // Ignore SA6002 relax staticcheck verification. histogramsPool.Put(v) case []record.RefFloatHistogramSample: samples := v @@ -410,7 +379,6 @@ Outer: } samples = samples[m:] } - //nolint:staticcheck // Ignore SA6002 relax staticcheck verification. floatHistogramsPool.Put(v) case []record.RefMetadata: for _, m := range v { @@ -425,7 +393,6 @@ Outer: Help: m.Help, } } - //nolint:staticcheck // Ignore SA6002 relax staticcheck verification. metadataPool.Put(v) default: panic(fmt.Errorf("unexpected decoded type: %T", d)) @@ -793,7 +760,6 @@ func (h *Head) loadWBL(r *wlog.Reader, multiRef map[chunks.HeadSeriesRef]chunks. } samples = samples[m:] } - //nolint:staticcheck // Ignore SA6002 relax staticcheck verification. samplesPool.Put(d) case []record.RefMmapMarker: markers := v diff --git a/tsdb/wal.go b/tsdb/wal.go index 38584847eb..e0bc1ec699 100644 --- a/tsdb/wal.go +++ b/tsdb/wal.go @@ -38,6 +38,7 @@ import ( "github.com/prometheus/prometheus/tsdb/record" "github.com/prometheus/prometheus/tsdb/tombstones" "github.com/prometheus/prometheus/tsdb/wlog" + "github.com/prometheus/prometheus/util/zeropool" ) // WALEntryType indicates what data a WAL entry contains. @@ -870,9 +871,9 @@ func (r *walReader) Read( // Historically, the processing is the bottleneck with reading and decoding using only // 15% of the CPU. var ( - seriesPool sync.Pool - samplePool sync.Pool - deletePool sync.Pool + seriesPool zeropool.Pool[[]record.RefSeries] + samplePool zeropool.Pool[[]record.RefSample] + deletePool zeropool.Pool[[]tombstones.Stone] ) donec := make(chan struct{}) datac := make(chan interface{}, 100) @@ -886,19 +887,16 @@ func (r *walReader) Read( if seriesf != nil { seriesf(v) } - //nolint:staticcheck // Ignore SA6002 safe to ignore and actually fixing it has some performance penalty. seriesPool.Put(v[:0]) case []record.RefSample: if samplesf != nil { samplesf(v) } - //nolint:staticcheck // Ignore SA6002 safe to ignore and actually fixing it has some performance penalty. samplePool.Put(v[:0]) case []tombstones.Stone: if deletesf != nil { deletesf(v) } - //nolint:staticcheck // Ignore SA6002 safe to ignore and actually fixing it has some performance penalty. deletePool.Put(v[:0]) default: level.Error(r.logger).Log("msg", "unexpected data type") @@ -915,11 +913,9 @@ func (r *walReader) Read( // Those should generally be caught by entry decoding before. switch et { case WALEntrySeries: - var series []record.RefSeries - if v := seriesPool.Get(); v == nil { + series := seriesPool.Get() + if series == nil { series = make([]record.RefSeries, 0, 512) - } else { - series = v.([]record.RefSeries) } err = r.decodeSeries(flag, b, &series) @@ -936,11 +932,9 @@ func (r *walReader) Read( } } case WALEntrySamples: - var samples []record.RefSample - if v := samplePool.Get(); v == nil { + samples := samplePool.Get() + if samples == nil { samples = make([]record.RefSample, 0, 512) - } else { - samples = v.([]record.RefSample) } err = r.decodeSamples(flag, b, &samples) @@ -958,11 +952,9 @@ func (r *walReader) Read( } } case WALEntryDeletes: - var deletes []tombstones.Stone - if v := deletePool.Get(); v == nil { + deletes := deletePool.Get() + if deletes == nil { deletes = make([]tombstones.Stone, 0, 512) - } else { - deletes = v.([]tombstones.Stone) } err = r.decodeDeletes(flag, b, &deletes) diff --git a/util/zeropool/pool.go b/util/zeropool/pool.go new file mode 100644 index 0000000000..4f6deddfb1 --- /dev/null +++ b/util/zeropool/pool.go @@ -0,0 +1,77 @@ +// Copyright 2023 The Prometheus Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// Package zeropool provides a zero-allocation type-safe alternative for sync.Pool, used to workaround staticheck SA6002. +// The contents of this package are brought from https://github.com/colega/zeropool because "little copying is better than little dependency". + +package zeropool + +import "sync" + +// Pool is a type-safe pool of items that does not allocate pointers to items. +// That is not entirely true, it does allocate sometimes, but not most of the time, +// just like the usual sync.Pool pools items most of the time, except when they're evicted. +// It does that by storing the allocated pointers in a secondary pool instead of letting them go, +// so they can be used later to store the items again. +// +// Zero value of Pool[T] is valid, and it will return zero values of T if nothing is pooled. +type Pool[T any] struct { + // items holds pointers to the pooled items, which are valid to be used. + items sync.Pool + // pointers holds just pointers to the pooled item types. + // The values referenced by pointers are not valid to be used (as they're used by some other caller) + // and it is safe to overwrite these pointers. + pointers sync.Pool +} + +// New creates a new Pool[T] with the given function to create new items. +// A Pool must not be copied after first use. +func New[T any](item func() T) Pool[T] { + return Pool[T]{ + items: sync.Pool{ + New: func() interface{} { + val := item() + return &val + }, + }, + } +} + +// Get returns an item from the pool, creating a new one if necessary. +// Get may be called concurrently from multiple goroutines. +func (p *Pool[T]) Get() T { + pooled := p.items.Get() + if pooled == nil { + // The only way this can happen is when someone is using the zero-value of zeropool.Pool, and items pool is empty. + // We don't have a pointer to store in p.pointers, so just return the empty value. + var zero T + return zero + } + + ptr := pooled.(*T) + item := *ptr // ptr still holds a reference to a copy of item, but nobody will use it. + p.pointers.Put(ptr) + return item +} + +// Put adds an item to the pool. +func (p *Pool[T]) Put(item T) { + var ptr *T + if pooled := p.pointers.Get(); pooled != nil { + ptr = pooled.(*T) + } else { + ptr = new(T) + } + *ptr = item + p.items.Put(ptr) +} diff --git a/util/zeropool/pool_test.go b/util/zeropool/pool_test.go new file mode 100644 index 0000000000..507687886f --- /dev/null +++ b/util/zeropool/pool_test.go @@ -0,0 +1,178 @@ +// Copyright 2023 The Prometheus Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package zeropool_test + +import ( + "math" + "sync" + "testing" + + "github.com/stretchr/testify/require" + "go.uber.org/atomic" + + "github.com/prometheus/prometheus/util/zeropool" +) + +func TestPool(t *testing.T) { + t.Run("provides correct values", func(t *testing.T) { + pool := zeropool.New(func() []byte { return make([]byte, 1024) }) + item1 := pool.Get() + require.Equal(t, 1024, len(item1)) + + item2 := pool.Get() + require.Equal(t, 1024, len(item2)) + + pool.Put(item1) + pool.Put(item2) + + item1 = pool.Get() + require.Equal(t, 1024, len(item1)) + + item2 = pool.Get() + require.Equal(t, 1024, len(item2)) + }) + + t.Run("is not racy", func(t *testing.T) { + pool := zeropool.New(func() []byte { return make([]byte, 1024) }) + + const iterations = 1e6 + const concurrency = math.MaxUint8 + var counter atomic.Int64 + + do := make(chan struct{}, 1e6) + for i := 0; i < iterations; i++ { + do <- struct{}{} + } + close(do) + + run := make(chan struct{}) + done := sync.WaitGroup{} + done.Add(concurrency) + for i := 0; i < concurrency; i++ { + go func(worker int) { + <-run + for range do { + item := pool.Get() + item[0] = byte(worker) + counter.Add(1) // Counts and also adds some delay to add raciness. + if item[0] != byte(worker) { + panic("wrong value") + } + pool.Put(item) + } + done.Done() + }(i) + } + close(run) + done.Wait() + t.Logf("Done %d iterations", counter.Load()) + }) + + t.Run("does not allocate", func(t *testing.T) { + pool := zeropool.New(func() []byte { return make([]byte, 1024) }) + // Warm up, this will alloate one slice. + slice := pool.Get() + pool.Put(slice) + + allocs := testing.AllocsPerRun(1000, func() { + slice := pool.Get() + pool.Put(slice) + }) + // Don't compare to 0, as when passing all the tests the GC could flush the pools during this test and we would allocate. + // Just check that it's less than 1 on average, which is mostly the same thing. + require.Less(t, allocs, 1., "Should not allocate.") + }) + + t.Run("zero value is valid", func(t *testing.T) { + var pool zeropool.Pool[[]byte] + slice := pool.Get() + pool.Put(slice) + + allocs := testing.AllocsPerRun(1000, func() { + slice := pool.Get() + pool.Put(slice) + }) + // Don't compare to 0, as when passing all the tests the GC could flush the pools during this test and we would allocate. + // Just check that it's less than 1 on average, which is mostly the same thing. + require.Less(t, allocs, 1., "Should not allocate.") + }) +} + +func BenchmarkZeropoolPool(b *testing.B) { + pool := zeropool.New(func() []byte { return make([]byte, 1024) }) + + // Warmup + item := pool.Get() + pool.Put(item) + + b.ResetTimer() + for i := 0; i < b.N; i++ { + item := pool.Get() + pool.Put(item) + } +} + +// BenchmarkSyncPoolValue uses sync.Pool to store values, which makes an allocation on each Put call. +func BenchmarkSyncPoolValue(b *testing.B) { + pool := sync.Pool{New: func() any { + return make([]byte, 1024) + }} + + // Warmup + item := pool.Get().([]byte) + pool.Put(item) //nolint:staticcheck // This allocates. + + b.ResetTimer() + for i := 0; i < b.N; i++ { + item := pool.Get().([]byte) + pool.Put(item) //nolint:staticcheck // This allocates. + } +} + +// BenchmarkSyncPoolNewPointer uses sync.Pool to store pointers, but it calls Put with a new pointer every time. +func BenchmarkSyncPoolNewPointer(b *testing.B) { + pool := sync.Pool{New: func() any { + v := make([]byte, 1024) + return &v + }} + + // Warmup + item := pool.Get().(*[]byte) + pool.Put(item) //nolint:staticcheck // This allocates. + + b.ResetTimer() + for i := 0; i < b.N; i++ { + item := pool.Get().(*[]byte) + buf := *item + pool.Put(&buf) //nolint:staticcheck // New pointer. + } +} + +// BenchmarkSyncPoolPointer illustrates the optimal usage of sync.Pool, not always possible. +func BenchmarkSyncPoolPointer(b *testing.B) { + pool := sync.Pool{New: func() any { + v := make([]byte, 1024) + return &v + }} + + // Warmup + item := pool.Get().(*[]byte) + pool.Put(item) + + b.ResetTimer() + for i := 0; i < b.N; i++ { + item := pool.Get().(*[]byte) + pool.Put(item) + } +} From 3ded84e649179b6b830b8aa29d980af04cef8bb7 Mon Sep 17 00:00:00 2001 From: Oleg Zaytsev Date: Thu, 30 Mar 2023 13:38:43 +0200 Subject: [PATCH 049/231] Fix TestCancelCompactions on windows It seems that readOnlyDB was still opened which blocked the temp dir cleanup. Also changed the copy dir to be another TempDir instead of manually creating one. Signed-off-by: Oleg Zaytsev --- tsdb/compact_test.go | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/tsdb/compact_test.go b/tsdb/compact_test.go index 58f339d02b..b58755e63e 100644 --- a/tsdb/compact_test.go +++ b/tsdb/compact_test.go @@ -1171,12 +1171,9 @@ func TestCancelCompactions(t *testing.T) { createBlock(t, tmpdir, genSeries(1, 1, 2000, 2001)) // The most recent block is ignored so can be e small one. // Copy the db so we have an exact copy to compare compaction times. - tmpdirCopy := tmpdir + "Copy" + tmpdirCopy := t.TempDir() err := fileutil.CopyDirs(tmpdir, tmpdirCopy) require.NoError(t, err) - defer func() { - require.NoError(t, os.RemoveAll(tmpdirCopy)) - }() // Measure the compaction time without interrupting it. var timeCompactionUninterrupted time.Duration @@ -1228,6 +1225,7 @@ func TestCancelCompactions(t *testing.T) { for i, b := range blocks { require.Falsef(t, b.Meta().Compaction.Failed, "block %d (%s) should not be marked as compaction failed", i, b.Meta().ULID) } + require.NoError(t, readOnlyDB.Close()) } } From 2fc2e233a68303af8d23df0e95e8e546d20e9123 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Fri, 31 Mar 2023 08:47:06 +0000 Subject: [PATCH 050/231] remote-write: raise default samples per send to 2,000 Larger messages cost less, because the per-message overheads at sender and receiver are spread over more samples. Example: scraping 1.5 million series every 15 seconds will send 50 messages per second. Previous default was 500. Signed-off-by: Bryan Boreham --- config/config.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/config/config.go b/config/config.go index a29c98eed2..9b0f4ad1e8 100644 --- a/config/config.go +++ b/config/config.go @@ -174,10 +174,10 @@ var ( // DefaultQueueConfig is the default remote queue configuration. DefaultQueueConfig = QueueConfig{ // With a maximum of 200 shards, assuming an average of 100ms remote write - // time and 500 samples per batch, we will be able to push 1M samples/s. + // time and 2000 samples per batch, we will be able to push 4M samples/s. MaxShards: 200, MinShards: 1, - MaxSamplesPerSend: 500, + MaxSamplesPerSend: 2000, // Each shard will have a max of 2500 samples pending in its channel, plus the pending // samples that have been enqueued. Theoretically we should only ever have about 3000 samples @@ -194,7 +194,7 @@ var ( DefaultMetadataConfig = MetadataConfig{ Send: true, SendInterval: model.Duration(1 * time.Minute), - MaxSamplesPerSend: 500, + MaxSamplesPerSend: 2000, } // DefaultRemoteReadConfig is the default remote read configuration. From 2bd2a6006dc9050cdf85e7f76653a32e0a14af64 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sat, 1 Apr 2023 23:57:09 +0000 Subject: [PATCH 051/231] build(deps): bump actions/cache from 3.2.4 to 3.3.1 Bumps [actions/cache](https://github.com/actions/cache) from 3.2.4 to 3.3.1. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/v3.2.4...v3.3.1) --- updated-dependencies: - dependency-name: actions/cache dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ceb374c8c5..a1497e5606 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -197,7 +197,7 @@ jobs: with: node-version-file: "web/ui/.nvmrc" registry-url: "https://registry.npmjs.org" - - uses: actions/cache@v3.2.4 + - uses: actions/cache@v3.3.1 with: path: ~/.npm key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }} From e15b0a29acaa9709c85230ed8b06619251d8179b Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sat, 1 Apr 2023 23:57:17 +0000 Subject: [PATCH 052/231] build(deps): bump actions/setup-go from 3 to 4 Bumps [actions/setup-go](https://github.com/actions/setup-go) from 3 to 4. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](https://github.com/actions/setup-go/compare/v3...v4) --- updated-dependencies: - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] --- .github/workflows/ci.yml | 4 ++-- .github/workflows/codeql-analysis.yml | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ceb374c8c5..8294d0756b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -52,7 +52,7 @@ jobs: runs-on: windows-latest steps: - uses: actions/checkout@v3 - - uses: actions/setup-go@v3 + - uses: actions/setup-go@v4 with: go-version: '>=1.20 <1.21' - run: | @@ -139,7 +139,7 @@ jobs: - name: Checkout repository uses: actions/checkout@v3 - name: Install Go - uses: actions/setup-go@v3 + uses: actions/setup-go@v4 with: go-version: 1.20.x - name: Install snmp_exporter/generator dependencies diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/codeql-analysis.yml index 43c9778575..6036e80ae9 100644 --- a/.github/workflows/codeql-analysis.yml +++ b/.github/workflows/codeql-analysis.yml @@ -21,7 +21,7 @@ jobs: steps: - name: Checkout repository uses: actions/checkout@v3 - - uses: actions/setup-go@v3 + - uses: actions/setup-go@v4 with: go-version: '>=1.20 <1.21' From a9af885de26aee57ed664c8708e15676f1f4cefe Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sat, 1 Apr 2023 23:57:22 +0000 Subject: [PATCH 053/231] build(deps): bump prometheus/promci from 0.0.2 to 0.1.0 Bumps [prometheus/promci](https://github.com/prometheus/promci) from 0.0.2 to 0.1.0. - [Release notes](https://github.com/prometheus/promci/releases) - [Commits](https://github.com/prometheus/promci/compare/v0.0.2...v0.1.0) --- updated-dependencies: - dependency-name: prometheus/promci dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- .github/workflows/ci.yml | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ceb374c8c5..4af85cc551 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -13,7 +13,7 @@ jobs: image: quay.io/prometheus/golang-builder:1.20-base steps: - uses: actions/checkout@v3 - - uses: prometheus/promci@v0.0.2 + - uses: prometheus/promci@v0.1.0 - uses: ./.github/promci/actions/setup_environment - run: make GO_ONLY=1 SKIP_GOLANGCI_LINT=1 - run: go test ./tsdb/ -test.tsdb-isolation=false @@ -35,7 +35,7 @@ jobs: steps: - uses: actions/checkout@v3 - - uses: prometheus/promci@v0.0.2 + - uses: prometheus/promci@v0.1.0 - uses: ./.github/promci/actions/setup_environment with: enable_go: false @@ -104,7 +104,7 @@ jobs: thread: [ 0, 1, 2 ] steps: - uses: actions/checkout@v3 - - uses: prometheus/promci@v0.0.2 + - uses: prometheus/promci@v0.1.0 - uses: ./.github/promci/actions/build with: promu_opts: "-p linux/amd64 -p windows/amd64 -p linux/arm64 -p darwin/amd64 -p darwin/arm64 -p linux/386" @@ -127,7 +127,7 @@ jobs: # should also be updated. steps: - uses: actions/checkout@v3 - - uses: prometheus/promci@v0.0.2 + - uses: prometheus/promci@v0.1.0 - uses: ./.github/promci/actions/build with: parallelism: 12 @@ -162,7 +162,7 @@ jobs: if: github.event_name == 'push' && github.event.ref == 'refs/heads/main' steps: - uses: actions/checkout@v3 - - uses: prometheus/promci@v0.0.2 + - uses: prometheus/promci@v0.1.0 - uses: ./.github/promci/actions/publish_main with: docker_hub_login: ${{ secrets.docker_hub_login }} @@ -176,7 +176,7 @@ jobs: if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v2.') steps: - uses: actions/checkout@v3 - - uses: prometheus/promci@v0.0.2 + - uses: prometheus/promci@v0.1.0 - uses: ./.github/promci/actions/publish_release with: docker_hub_login: ${{ secrets.docker_hub_login }} @@ -191,7 +191,7 @@ jobs: steps: - name: Checkout uses: actions/checkout@v3 - - uses: prometheus/promci@v0.0.2 + - uses: prometheus/promci@v0.1.0 - name: Install nodejs uses: actions/setup-node@v3 with: From babdbd8add4e5efc6b457854ee2e999f9c121482 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sat, 1 Apr 2023 23:59:48 +0000 Subject: [PATCH 054/231] build(deps): bump github.com/ionos-cloud/sdk-go/v6 from 6.1.4 to 6.1.5 Bumps [github.com/ionos-cloud/sdk-go/v6](https://github.com/ionos-cloud/sdk-go) from 6.1.4 to 6.1.5. - [Release notes](https://github.com/ionos-cloud/sdk-go/releases) - [Changelog](https://github.com/ionos-cloud/sdk-go/blob/master/docs/CHANGELOG.md) - [Commits](https://github.com/ionos-cloud/sdk-go/compare/v6.1.4...v6.1.5) --- updated-dependencies: - dependency-name: github.com/ionos-cloud/sdk-go/v6 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 4b672b8628..2075fd8912 100644 --- a/go.mod +++ b/go.mod @@ -30,7 +30,7 @@ require ( github.com/hashicorp/consul/api v1.20.0 github.com/hashicorp/nomad/api v0.0.0-20230308192510-48e7d70fcd4b github.com/hetznercloud/hcloud-go v1.41.0 - github.com/ionos-cloud/sdk-go/v6 v6.1.4 + github.com/ionos-cloud/sdk-go/v6 v6.1.5 github.com/json-iterator/go v1.1.12 github.com/kolo/xmlrpc v0.0.0-20220921171641-a4b6fa1dd06b github.com/linode/linodego v1.14.1 diff --git a/go.sum b/go.sum index 2414991588..da2a663360 100644 --- a/go.sum +++ b/go.sum @@ -464,8 +464,8 @@ github.com/imdario/mergo v0.3.13 h1:lFzP57bqS/wsqKssCGmtLAb8A0wKjLGrve2q3PPVcBk= github.com/imdario/mergo v0.3.13/go.mod h1:4lJ1jqUDcsbIECGy0RUJAXNIhg+6ocWgb1ALK2O4oXg= github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8= github.com/influxdata/influxdb1-client v0.0.0-20191209144304-8bf82d3c094d/go.mod h1:qj24IKcXYK6Iy9ceXlo3Tc+vtHo9lIhSX5JddghvEPo= -github.com/ionos-cloud/sdk-go/v6 v6.1.4 h1:BJHhFA8Q1SZC7VOXqKKr2BV2ysQ2/4hlk1e4hZte7GY= -github.com/ionos-cloud/sdk-go/v6 v6.1.4/go.mod h1:Ox3W0iiEz0GHnfY9e5LmAxwklsxguuNFEUSu0gVRTME= +github.com/ionos-cloud/sdk-go/v6 v6.1.5 h1:BFqThLOgrGJWeo7w6UDyYuNxyi/GqEmNPl7C/YcQ8Fw= +github.com/ionos-cloud/sdk-go/v6 v6.1.5/go.mod h1:EzEgRIDxBELvfoa/uBN0kOQaqovLjUWEB7iW4/Q+t4k= github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k= github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg= github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo= From bf988304d77c64241255ca43ec4e05362cdaf1be Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sat, 1 Apr 2023 23:59:59 +0000 Subject: [PATCH 055/231] build(deps): bump github.com/digitalocean/godo from 1.97.0 to 1.98.0 Bumps [github.com/digitalocean/godo](https://github.com/digitalocean/godo) from 1.97.0 to 1.98.0. - [Release notes](https://github.com/digitalocean/godo/releases) - [Changelog](https://github.com/digitalocean/godo/blob/main/CHANGELOG.md) - [Commits](https://github.com/digitalocean/godo/compare/v1.97.0...v1.98.0) --- updated-dependencies: - dependency-name: github.com/digitalocean/godo dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 4b672b8628..f0968cd63c 100644 --- a/go.mod +++ b/go.mod @@ -11,7 +11,7 @@ require ( github.com/aws/aws-sdk-go v1.44.217 github.com/cespare/xxhash/v2 v2.2.0 github.com/dennwc/varint v1.0.0 - github.com/digitalocean/godo v1.97.0 + github.com/digitalocean/godo v1.98.0 github.com/docker/docker v23.0.1+incompatible github.com/edsrzf/mmap-go v1.1.0 github.com/envoyproxy/go-control-plane v0.11.0 diff --git a/go.sum b/go.sum index 2414991588..9ed1279b96 100644 --- a/go.sum +++ b/go.sum @@ -149,8 +149,8 @@ github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs github.com/dennwc/varint v1.0.0 h1:kGNFFSSw8ToIy3obO/kKr8U9GZYUAxQEVuix4zfDWzE= github.com/dennwc/varint v1.0.0/go.mod h1:hnItb35rvZvJrbTALZtY/iQfDs48JKRG1RPpgziApxA= github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ= -github.com/digitalocean/godo v1.97.0 h1:p9w1yCcWMZcxFSLPToNGXA96WfUVLXqoHti6GzVomL4= -github.com/digitalocean/godo v1.97.0/go.mod h1:NRpFznZFvhHjBoqZAaOD3khVzsJ3EibzKqFL4R60dmA= +github.com/digitalocean/godo v1.98.0 h1:potyC1eD0N9n5/P4/WmJuKgg+OGYZOBWEW+/aKTX6QQ= +github.com/digitalocean/godo v1.98.0/go.mod h1:NRpFznZFvhHjBoqZAaOD3khVzsJ3EibzKqFL4R60dmA= github.com/dnaeon/go-vcr v1.2.0 h1:zHCHvJYTMh1N7xnV7zf1m1GPBF9Ad0Jk/whtQ1663qI= github.com/docker/distribution v2.8.1+incompatible h1:Q50tZOPR6T/hjNsyc9g8/syEs6bk8XXApsHjKukMl68= github.com/docker/distribution v2.8.1+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w= From 6f2507656c97937a3a9856d3c94945569cb009f7 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 2 Apr 2023 00:00:12 +0000 Subject: [PATCH 056/231] build(deps): bump github.com/miekg/dns from 1.1.51 to 1.1.53 Bumps [github.com/miekg/dns](https://github.com/miekg/dns) from 1.1.51 to 1.1.53. - [Release notes](https://github.com/miekg/dns/releases) - [Changelog](https://github.com/miekg/dns/blob/master/Makefile.release) - [Commits](https://github.com/miekg/dns/compare/v1.1.51...v1.1.53) --- updated-dependencies: - dependency-name: github.com/miekg/dns dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] --- go.mod | 2 +- go.sum | 9 ++------- 2 files changed, 3 insertions(+), 8 deletions(-) diff --git a/go.mod b/go.mod index 4b672b8628..c010570297 100644 --- a/go.mod +++ b/go.mod @@ -34,7 +34,7 @@ require ( github.com/json-iterator/go v1.1.12 github.com/kolo/xmlrpc v0.0.0-20220921171641-a4b6fa1dd06b github.com/linode/linodego v1.14.1 - github.com/miekg/dns v1.1.51 + github.com/miekg/dns v1.1.53 github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f github.com/oklog/run v1.1.0 github.com/oklog/ulid v1.3.1 diff --git a/go.sum b/go.sum index 2414991588..1e23176bea 100644 --- a/go.sum +++ b/go.sum @@ -546,8 +546,8 @@ github.com/matttproud/golang_protobuf_extensions v1.0.4/go.mod h1:BSXmuO+STAnVfr github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg= github.com/miekg/dns v1.1.26/go.mod h1:bPDLeHnStXmXAq1m/Ch/hvfNHr14JKNPMBo3VZKjuso= github.com/miekg/dns v1.1.41/go.mod h1:p6aan82bvRIyn+zDIv9xYNUpwa73JcSh9BKwknJysuI= -github.com/miekg/dns v1.1.51 h1:0+Xg7vObnhrz/4ZCZcZh7zPXlmU0aveS2HDBd0m0qSo= -github.com/miekg/dns v1.1.51/go.mod h1:2Z9d3CP1LQWihRZUf29mQ19yDThaI4DAYzte2CaQW5c= +github.com/miekg/dns v1.1.53 h1:ZBkuHr5dxHtB1caEOlZTLPo7D3L3TWckgUUs/RHfDxw= +github.com/miekg/dns v1.1.53/go.mod h1:uInx36IzPl7FYnDcMeVWxj9byh7DutNykX4G9Sj60FY= github.com/mitchellh/cli v1.0.0/go.mod h1:hNIlj7HEI86fIcpObd7a0FcrxTWetlwJDGcceTlRvqc= github.com/mitchellh/cli v1.1.0/go.mod h1:xcISNoH86gajksDmfB23e/pu+B+GeFRMYmoHXxx3xhI= github.com/mitchellh/go-homedir v1.0.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= @@ -861,7 +861,6 @@ golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzB golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= -golang.org/x/mod v0.7.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= golang.org/x/mod v0.9.0 h1:KENHtAZL2y3NLMYZeHY9DW8HW8V+kQyJsY/V9JlKvCs= golang.org/x/mod v0.9.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -910,7 +909,6 @@ golang.org/x/net v0.0.0-20211029224645-99673261e6eb/go.mod h1:9nx3DQGgdP8bBQD5qx golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.1.0/go.mod h1:Cx3nUiGt4eDBEyega/BKRp+/AlGL8hYe7U9odMt2Cco= -golang.org/x/net v0.2.0/go.mod h1:KqCZLdyyvdV855qA2rE3GC2aiw5xGR5TEjj8smXukLY= golang.org/x/net v0.8.0 h1:Zrh2ngAOFYneWTAIAPethzeaQLuHwhuBkuV6ZiRnUaQ= golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= @@ -1002,13 +1000,11 @@ golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.6.0 h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ= golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.1.0/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= -golang.org/x/term v0.2.0/go.mod h1:TVmDHMZPmdnySmBfhjOoOdhjzdE1h4u1VwSiw2l1Nuc= golang.org/x/term v0.6.0 h1:clScbb1cHjoCkyRbWwBEUZ5H/tIFu5TAXIqaZD0Gcjw= golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U= golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= @@ -1081,7 +1077,6 @@ golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roY golang.org/x/tools v0.0.0-20200825202427-b303f430e36d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA= golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= -golang.org/x/tools v0.3.0/go.mod h1:/rWhSS2+zyEVwoJf8YAX6L2f0ntZ7Kn/mGgAWcipA5k= golang.org/x/tools v0.7.0 h1:W4OVu8VVOaIO0yzWMNdepAulS7YfoS3Zabrm8DOXXU4= golang.org/x/tools v0.7.0/go.mod h1:4pg6aUX35JBAogB10C9AtvVL+qowtN4pT3CGSQex14s= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= From 0b7cf5b368326658baec5fa789895121472e7fd2 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 2 Apr 2023 00:00:56 +0000 Subject: [PATCH 057/231] build(deps): bump google.golang.org/api from 0.111.0 to 0.114.0 Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.111.0 to 0.114.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.111.0...v0.114.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- go.mod | 6 +++--- go.sum | 12 ++++++------ 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/go.mod b/go.mod index 4b672b8628..496e7f0628 100644 --- a/go.mod +++ b/go.mod @@ -67,10 +67,10 @@ require ( golang.org/x/sys v0.6.0 golang.org/x/time v0.3.0 golang.org/x/tools v0.7.0 - google.golang.org/api v0.111.0 + google.golang.org/api v0.114.0 google.golang.org/genproto v0.0.0-20230306155012-7f2fa6fef1f4 google.golang.org/grpc v1.53.0 - google.golang.org/protobuf v1.29.0 + google.golang.org/protobuf v1.29.1 gopkg.in/yaml.v2 v2.4.0 gopkg.in/yaml.v3 v3.0.1 k8s.io/api v0.26.2 @@ -134,7 +134,7 @@ require ( github.com/google/gofuzz v1.2.0 // indirect github.com/google/uuid v1.3.0 // indirect github.com/googleapis/enterprise-certificate-proxy v0.2.3 // indirect - github.com/googleapis/gax-go/v2 v2.7.0 // indirect + github.com/googleapis/gax-go/v2 v2.7.1 // indirect github.com/gorilla/websocket v1.5.0 // indirect github.com/grpc-ecosystem/grpc-gateway/v2 v2.15.2 // indirect github.com/hashicorp/cronexpr v1.1.1 // indirect diff --git a/go.sum b/go.sum index 2414991588..a3c3008dfc 100644 --- a/go.sum +++ b/go.sum @@ -378,8 +378,8 @@ github.com/googleapis/enterprise-certificate-proxy v0.2.3 h1:yk9/cqRKtT9wXZSsRH9 github.com/googleapis/enterprise-certificate-proxy v0.2.3/go.mod h1:AwSRAtLfXpU5Nm3pW+v7rGDHp09LsPtGY9MduiEsR9k= github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= -github.com/googleapis/gax-go/v2 v2.7.0 h1:IcsPKeInNvYi7eqSaDjiZqDDKu5rsmunY0Y1YupQSSQ= -github.com/googleapis/gax-go/v2 v2.7.0/go.mod h1:TEop28CZZQ2y+c0VxMUmu1lV+fQx57QpBWsYpwqHJx8= +github.com/googleapis/gax-go/v2 v2.7.1 h1:gF4c0zjUP2H/s/hEGyLA3I0fA2ZWjzYiONAD6cvPr8A= +github.com/googleapis/gax-go/v2 v2.7.1/go.mod h1:4orTrqY6hXxxaUL4LHIPl6lGo8vAE38/qKbhSAKP6QI= github.com/gophercloud/gophercloud v1.2.0 h1:1oXyj4g54KBg/kFtCdMM6jtxSzeIyg8wv4z1HoGPp1E= github.com/gophercloud/gophercloud v1.2.0/go.mod h1:aAVqcocTSXh2vYFZ1JTvx4EQmfgzxRcNupUfxZbBNDM= github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY= @@ -1103,8 +1103,8 @@ google.golang.org/api v0.20.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/ google.golang.org/api v0.22.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE= google.golang.org/api v0.24.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0MncE= google.golang.org/api v0.28.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0MncE= -google.golang.org/api v0.111.0 h1:bwKi+z2BsdwYFRKrqwutM+axAlYLz83gt5pDSXCJT+0= -google.golang.org/api v0.111.0/go.mod h1:qtFHvU9mhgTJegR31csQ+rwxyUTHOKFqCKWp1J0fdw0= +google.golang.org/api v0.114.0 h1:1xQPji6cO2E2vLiI+C/XiFAnsn1WV3mjaEwGLhi3grE= +google.golang.org/api v0.114.0/go.mod h1:ifYI2ZsFK6/uGddGfAD5BMxlnkBqCmqHSDUVi45N5Yg= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.2.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= @@ -1183,8 +1183,8 @@ google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlba google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= -google.golang.org/protobuf v1.29.0 h1:44S3JjaKmLEE4YIkjzexaP+NzZsudE3Zin5Njn/pYX0= -google.golang.org/protobuf v1.29.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= +google.golang.org/protobuf v1.29.1 h1:7QBf+IK2gx70Ap/hDsOmam3GE0v9HicjfEdAxE62UoM= +google.golang.org/protobuf v1.29.1/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= From ce353727f116f5432192267621fbd1a39c40146b Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 2 Apr 2023 00:02:05 +0000 Subject: [PATCH 058/231] build(deps): bump github.com/prometheus/prometheus Bumps [github.com/prometheus/prometheus](https://github.com/prometheus/prometheus) from 0.42.0 to 0.43.0. - [Release notes](https://github.com/prometheus/prometheus/releases) - [Changelog](https://github.com/prometheus/prometheus/blob/main/CHANGELOG.md) - [Commits](https://github.com/prometheus/prometheus/compare/v0.42.0...v0.43.0) --- updated-dependencies: - dependency-name: github.com/prometheus/prometheus dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- documentation/examples/remote_storage/go.mod | 32 ++-- documentation/examples/remote_storage/go.sum | 148 +++++++++---------- 2 files changed, 89 insertions(+), 91 deletions(-) diff --git a/documentation/examples/remote_storage/go.mod b/documentation/examples/remote_storage/go.mod index 8eb97aaf5c..5e0e6ddb13 100644 --- a/documentation/examples/remote_storage/go.mod +++ b/documentation/examples/remote_storage/go.mod @@ -10,22 +10,22 @@ require ( github.com/influxdata/influxdb v1.11.0 github.com/prometheus/client_golang v1.14.0 github.com/prometheus/common v0.42.0 - github.com/prometheus/prometheus v0.42.0 + github.com/prometheus/prometheus v0.43.0 github.com/stretchr/testify v1.8.2 ) require ( github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 // indirect - github.com/aws/aws-sdk-go v1.44.187 // indirect + github.com/aws/aws-sdk-go v1.44.217 // indirect github.com/beorn7/perks v1.0.1 // indirect github.com/cespare/xxhash/v2 v2.2.0 // indirect github.com/davecgh/go-spew v1.1.1 // indirect github.com/dennwc/varint v1.0.0 // indirect github.com/felixge/httpsnoop v1.0.3 // indirect - github.com/go-logfmt/logfmt v0.5.1 // indirect + github.com/go-logfmt/logfmt v0.6.0 // indirect github.com/go-logr/logr v1.2.3 // indirect github.com/go-logr/stdr v1.2.2 // indirect - github.com/golang/protobuf v1.5.2 // indirect + github.com/golang/protobuf v1.5.3 // indirect github.com/grafana/regexp v0.0.0-20221122212121-6b5c0a4cb7fd // indirect github.com/jmespath/go-jmespath v0.4.0 // indirect github.com/jpillora/backoff v1.0.0 // indirect @@ -36,22 +36,22 @@ require ( github.com/pmezard/go-difflib v1.0.0 // indirect github.com/prometheus/client_model v0.3.0 // indirect github.com/prometheus/common/sigv4 v0.1.0 // indirect - github.com/prometheus/procfs v0.8.0 // indirect + github.com/prometheus/procfs v0.9.0 // indirect github.com/xhit/go-str2duration/v2 v2.1.0 // indirect - go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.37.0 // indirect - go.opentelemetry.io/otel v1.11.2 // indirect - go.opentelemetry.io/otel/metric v0.34.0 // indirect - go.opentelemetry.io/otel/trace v1.11.2 // indirect + go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.40.0 // indirect + go.opentelemetry.io/otel v1.14.0 // indirect + go.opentelemetry.io/otel/metric v0.37.0 // indirect + go.opentelemetry.io/otel/trace v1.14.0 // indirect go.uber.org/atomic v1.10.0 // indirect - go.uber.org/goleak v1.2.0 // indirect - golang.org/x/exp v0.0.0-20230124195608-d38c7dcee874 // indirect - golang.org/x/net v0.7.0 // indirect - golang.org/x/oauth2 v0.5.0 // indirect - golang.org/x/sys v0.5.0 // indirect - golang.org/x/text v0.7.0 // indirect + go.uber.org/goleak v1.2.1 // indirect + golang.org/x/exp v0.0.0-20230307190834-24139beb5833 // indirect + golang.org/x/net v0.8.0 // indirect + golang.org/x/oauth2 v0.6.0 // indirect + golang.org/x/sys v0.6.0 // indirect + golang.org/x/text v0.8.0 // indirect golang.org/x/time v0.3.0 // indirect google.golang.org/appengine v1.6.7 // indirect - google.golang.org/protobuf v1.28.1 // indirect + google.golang.org/protobuf v1.29.0 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect ) diff --git a/documentation/examples/remote_storage/go.sum b/documentation/examples/remote_storage/go.sum index e7ba9f53b3..11c2945151 100644 --- a/documentation/examples/remote_storage/go.sum +++ b/documentation/examples/remote_storage/go.sum @@ -7,7 +7,7 @@ github.com/Azure/go-autorest/autorest/to v0.4.0 h1:oXVqrxakqqV1UZdSazDOPOLvOIz+X github.com/Azure/go-autorest/autorest/validation v0.3.1 h1:AgyqjAd94fwNAoTjl/WQXg4VvFeRFpO+UhNyRXqF1ac= github.com/Azure/go-autorest/logger v0.2.1 h1:IG7i4p/mDa2Ce4TRyAO8IHnVhAVF3RFU+ZtXWSmf4Tg= github.com/Azure/go-autorest/tracing v0.6.0 h1:TYi4+3m5t6K48TGI9AUdb+IzbnSxvnvUMfuitfgcfuo= -github.com/Microsoft/go-winio v0.5.1 h1:aPJp2QD7OOrhO5tQXqQoGSJc+DjDtWTGLOmNyAm6FgY= +github.com/Microsoft/go-winio v0.6.0 h1:slsWYD/zyx7lCXoZVlvQrj0hPTM1HI4+v1sIda2yDvg= github.com/alecthomas/kingpin/v2 v2.3.2 h1:H0aULhgmSzN8xQ3nX1uxtdlTHYoPLu5AhHxWrKI6ocU= github.com/alecthomas/kingpin/v2 v2.3.2/go.mod h1:0gyi0zQnjuFk8xrkNKamJoyUo382HRL7ATRpFZCw6tE= github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= @@ -17,10 +17,10 @@ github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRF github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d/go.mod h1:rBZYJk541a8SKzHPHnH3zbiI+7dagKZ0cgpgrD7Fyho= github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 h1:s6gZFSlWYmbqAuRjVTiNNhvNRfY2Wxp9nhfyel4rklc= github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137/go.mod h1:OMCwj8VM1Kc9e19TLln2VL61YJF0x1XFtfdL4JdbSyE= -github.com/armon/go-metrics v0.3.10 h1:FR+drcQStOe+32sYyJYyZ7FIdgoGGBnwLl+flodp8Uo= +github.com/armon/go-metrics v0.4.1 h1:hR91U9KYmb6bLBYLQjyM+3j+rcd/UhE+G78SFnF8gJA= github.com/aws/aws-sdk-go v1.38.35/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro= -github.com/aws/aws-sdk-go v1.44.187 h1:D5CsRomPnlwDHJCanL2mtaLIcbhjiWxNh5j8zvaWdJA= -github.com/aws/aws-sdk-go v1.44.187/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI= +github.com/aws/aws-sdk-go v1.44.217 h1:FcWC56MRl+k756aH3qeMQTylSdeJ58WN0iFz3fkyRz0= +github.com/aws/aws-sdk-go v1.44.217/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI= github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8= github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= @@ -28,23 +28,23 @@ github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6r github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44= github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= -github.com/cncf/xds/go v0.0.0-20220314180256-7f1daf1720fc h1:PYXxkRUBGUMa5xgMVMDl62vEklZvKpVaxQeN9ie7Hfk= +github.com/cncf/xds/go v0.0.0-20230112175826-46e39c7b9b43 h1:XP+uhjN0yBCN/tPkr8Z0BNDc5rZam9RG6UWyf2FrSQ0= github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/dennwc/varint v1.0.0 h1:kGNFFSSw8ToIy3obO/kKr8U9GZYUAxQEVuix4zfDWzE= github.com/dennwc/varint v1.0.0/go.mod h1:hnItb35rvZvJrbTALZtY/iQfDs48JKRG1RPpgziApxA= -github.com/digitalocean/godo v1.95.0 h1:S48/byPKui7RHZc1wYEPfRvkcEvToADNb5I3guu95xg= +github.com/digitalocean/godo v1.97.0 h1:p9w1yCcWMZcxFSLPToNGXA96WfUVLXqoHti6GzVomL4= github.com/docker/distribution v2.8.1+incompatible h1:Q50tZOPR6T/hjNsyc9g8/syEs6bk8XXApsHjKukMl68= -github.com/docker/docker v20.10.23+incompatible h1:1ZQUUYAdh+oylOT85aA2ZcfRp22jmLhoaEcVEfK8dyA= +github.com/docker/docker v23.0.1+incompatible h1:vjgvJZxprTTE1A37nm+CLNAdwu6xZekyoiVlUZEINcY= github.com/docker/go-connections v0.4.0 h1:El9xVISelRB7BuFusrZozjnkIM5YnzCViNKohAFqRJQ= github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4= github.com/edsrzf/mmap-go v1.1.0 h1:6EUwBLQ/Mcr1EYLE4Tn1VdW1A4ckqCQWZBw8Hr0kjpQ= -github.com/emicklei/go-restful/v3 v3.9.0 h1:XwGDlfxEnQZzuopoqxwSEllNcCOM9DhhFyhFIIGKwxE= -github.com/envoyproxy/go-control-plane v0.10.3 h1:xdCVXxEe0Y3FQith+0cj2irwZudqGYvecuLB1HtdexY= +github.com/emicklei/go-restful/v3 v3.10.1 h1:rc42Y5YTp7Am7CS630D7JmhRjq4UlEUuEKfrDac4bSQ= +github.com/envoyproxy/go-control-plane v0.11.0 h1:jtLewhRR2vMRNnq2ZZUoCjUlgut+Y0+sDDWPOfwOi1o= github.com/envoyproxy/protoc-gen-validate v0.9.1 h1:PS7VIOgmSVhWUEeZwTe7z7zouA22Cr590PzXKbZHOVY= -github.com/fatih/color v1.13.0 h1:8LOYc1KYPPmyKMuN8QV2DNRWNbLo6LZ0iLs8+mlH53w= +github.com/fatih/color v1.14.1 h1:qfhVLaG5s+nCROl1zJsZRxFeYrHLqWroPOQ8BWiNb4w= github.com/felixge/httpsnoop v1.0.3 h1:s/nj+GCswXYzN5v2DpNMuMQYe+0DDwt5WVCU6CWBdXk= github.com/felixge/httpsnoop v1.0.3/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U= github.com/fsnotify/fsnotify v1.6.0 h1:n+5WquG0fcWoWp6xPWfHdbskMCQaFnG6PfBrh1Ky4HY= @@ -56,24 +56,24 @@ github.com/go-kit/log v0.2.1/go.mod h1:NwTd00d/i8cPZ3xOwwiv2PO5MOcx78fFErGNcVmBj github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE= github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk= github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A= -github.com/go-logfmt/logfmt v0.5.1 h1:otpy5pqBCBZ1ng9RQ0dPu4PN7ba75Y/aA+UpowDyNVA= -github.com/go-logfmt/logfmt v0.5.1/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs= +github.com/go-logfmt/logfmt v0.6.0 h1:wGYYu3uicYdqXVgoYbvnkrPVXkuLM1p1ifugDMEdRi4= +github.com/go-logfmt/logfmt v0.6.0/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs= github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= github.com/go-logr/logr v1.2.3 h1:2DntVwHkVopvECVRSlL5PSo9eG+cAkDCuckLubN+rq0= github.com/go-logr/logr v1.2.3/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= -github.com/go-openapi/jsonpointer v0.19.5 h1:gZr+CIYByUqjcgeLXnQu2gHYQC9o73G2XUeOFYEICuY= -github.com/go-openapi/jsonreference v0.20.0 h1:MYlu0sBgChmCfJxxUKZ8g1cPWFOB37YSZqewK7OKeyA= +github.com/go-openapi/jsonpointer v0.19.6 h1:eCs3fxoIi3Wh6vtgmLTOjdhSpiqphQ+DaPn38N2ZdrE= +github.com/go-openapi/jsonreference v0.20.2 h1:3sVjiK66+uXK/6oQ8xgcRKcFgQ5KXa2KvnJRumpMGbE= github.com/go-openapi/swag v0.22.3 h1:yMBqmnQ0gyZvEb/+KzuWZOXgllrXT4SADYbvDaXHv/g= -github.com/go-resty/resty/v2 v2.1.1-0.20191201195748-d7b97669fe48 h1:JVrqSeQfdhYRFk24TvhTZWU0q8lfCojxZQFi3Ou7+uY= +github.com/go-resty/resty/v2 v2.7.0 h1:me+K9p3uhSmXtrBZ4k9jcEAfJmuC8IivWHwaLZwPrFY= github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY= github.com/go-zookeeper/zk v1.0.3 h1:7M2kwOsc//9VeeFiPtf+uSJlVpU66x9Ba5+8XK7/TDg= github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ= github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= github.com/golang-jwt/jwt v3.2.1+incompatible h1:73Z+4BJcrTC+KczS6WvTPvRGOp1WmfEP4Q1lOd9Z/+c= -github.com/golang-jwt/jwt/v4 v4.2.0 h1:besgBTC8w8HjP6NzQdxwKH9Z5oQMZ24ThTrHp3cZ8eU= +github.com/golang-jwt/jwt/v4 v4.5.0 h1:7cYmW1XlMY7h7ii7UhUyChSgS5wUJEnm9uZVTGqOWzg= github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= @@ -86,11 +86,11 @@ github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvq github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= -github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw= -github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= +github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg= +github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= github.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM= github.com/golang/snappy v0.0.4/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= -github.com/google/gnostic v0.5.7-v3refs h1:FhTMOKj2VhjpouxvWJAV1TL304uMlb9zcDqkl6cEI54= +github.com/google/gnostic v0.6.9 h1:ZK/5VhkoX835RikCHpSUJV9a+S3e1zLh59YnyWeBW+0= github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= @@ -100,27 +100,27 @@ github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38= github.com/google/go-querystring v1.1.0 h1:AnCroh3fv4ZBgVIf1Iwtovgjaw/GiKJo8M8yD/fhyJ8= github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0= -github.com/gophercloud/gophercloud v1.1.1 h1:MuGyqbSxiuVBqkPZ3+Nhbytk1xZxhmfCB2Rg1cJWFWM= +github.com/gophercloud/gophercloud v1.2.0 h1:1oXyj4g54KBg/kFtCdMM6jtxSzeIyg8wv4z1HoGPp1E= github.com/gorilla/websocket v1.5.0 h1:PPwGk2jz7EePpoHN/+ClbZu8SPxiqlu12wZP/3sWmnc= github.com/grafana/regexp v0.0.0-20221122212121-6b5c0a4cb7fd h1:PpuIBO5P3e9hpqBD0O/HjhShYuM6XE0i/lbE6J94kww= github.com/grafana/regexp v0.0.0-20221122212121-6b5c0a4cb7fd/go.mod h1:M5qHK+eWfAv8VR/265dIuEpL3fNfeC21tXXp9itM24A= -github.com/hashicorp/consul/api v1.18.0 h1:R7PPNzTCeN6VuQNDwwhZWJvzCtGSrNpJqfb22h3yH9g= +github.com/hashicorp/consul/api v1.20.0 h1:9IHTjNVSZ7MIwjlW3N3a7iGiykCMDpxZu8jsxFJh0yc= github.com/hashicorp/cronexpr v1.1.1 h1:NJZDd87hGXjoZBdvyCF9mX4DCq5Wy7+A/w+A7q0wn6c= -github.com/hashicorp/errwrap v1.0.0 h1:hLrqtEDnRye3+sgx6z4qVLNuviH3MR5aQ0ykNJa/UYA= +github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I= github.com/hashicorp/go-cleanhttp v0.5.2 h1:035FKYIWjmULyFRBKPs8TBQoi0x6d9G4xc9neXJWAZQ= -github.com/hashicorp/go-hclog v0.16.2 h1:K4ev2ib4LdQETX5cSZBG0DVLk1jwGqSPXBjdah3veNs= +github.com/hashicorp/go-hclog v1.4.0 h1:ctuWFGrhFha8BnnzxqeRGidlEcQkDyL5u8J8t5eA11I= github.com/hashicorp/go-immutable-radix v1.3.1 h1:DKHmCUm2hRBK510BaiZlwvpD40f8bJFeZnpfm2KLowc= github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo= -github.com/hashicorp/go-retryablehttp v0.7.1 h1:sUiuQAnLlbvmExtFQs72iFW/HXeUn8Z1aJLQ4LJJbTQ= +github.com/hashicorp/go-retryablehttp v0.7.2 h1:AcYqCvkpalPnPF2pn0KamgwamS42TqUDDYFRKq/RAd0= github.com/hashicorp/go-rootcerts v1.0.2 h1:jzhAVGtqPKbwpyCPELlgNWhE1znq+qwJtW5Oi2viEzc= github.com/hashicorp/golang-lru v0.6.0 h1:uL2shRDx7RTrOrTCUZEGP/wJUFiUI8QT6E7z5o8jga4= -github.com/hashicorp/nomad/api v0.0.0-20230124213148-69fd1a0e4bf7 h1:XOdd3JHyeQnBRxotBo9ibxBFiYGuYhQU25s/YeV2cTU= +github.com/hashicorp/nomad/api v0.0.0-20230308192510-48e7d70fcd4b h1:EkuSTU8c/63q4LMayj8ilgg/4I5PXDFVcnqKfs9qcwI= github.com/hashicorp/serf v0.10.1 h1:Z1H2J60yRKvfDYAOZLd2MU0ND4AH/WDz7xYHDWQsIPY= -github.com/hetznercloud/hcloud-go v1.39.0 h1:RUlzI458nGnPR6dlcZlrsGXYC1hQlFbKdm8tVtEQQB0= -github.com/imdario/mergo v0.3.12 h1:b6R2BslTbIEToALKP7LxUvijTsNI9TAe80pLWN2g/HU= +github.com/hetznercloud/hcloud-go v1.41.0 h1:KJGFRRc68QiVu4PrEP5BmCQVveCP2CM26UGQUKGpIUs= +github.com/imdario/mergo v0.3.13 h1:lFzP57bqS/wsqKssCGmtLAb8A0wKjLGrve2q3PPVcBk= github.com/influxdata/influxdb v1.11.0 h1:0X+ZsbcOWc6AEi5MHee9BYqXCKmz8IZsljrRYjmV8Qg= github.com/influxdata/influxdb v1.11.0/go.mod h1:V93tJcidY0Zh0LtSONZWnXXGDyt20dtVf+Ddp4EnhaA= -github.com/ionos-cloud/sdk-go/v6 v6.1.3 h1:vb6yqdpiqaytvreM0bsn2pXw+1YDvEk2RKSmBAQvgDQ= +github.com/ionos-cloud/sdk-go/v6 v6.1.4 h1:BJHhFA8Q1SZC7VOXqKKr2BV2ysQ2/4hlk1e4hZte7GY= github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg= github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo= github.com/jmespath/go-jmespath/internal/testify v1.5.1 h1:shLQSRRSCCPj3f2gpwzGwWFoC7ycTf1rcQZHOlsJ6N8= @@ -147,14 +147,14 @@ github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= -github.com/linode/linodego v1.12.0 h1:33mOIrZ+gVva14gyJMKPZ85mQGovAvZCEP1ftgmFBjA= +github.com/linode/linodego v1.14.1 h1:uGxQyy0BidoEpLGdvfi4cPgEW+0YUFsEGrLEhcTfjNc= github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0= -github.com/mattn/go-colorable v0.1.12 h1:jF+Du6AlPIjs2BiUiQlKOX0rt3SujHxPnksPKZbaA40= -github.com/mattn/go-isatty v0.0.14 h1:yVuAays6BHfxijgZPzw+3Zlu5yQgKGP2/hcQbHb7S9Y= +github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA= +github.com/mattn/go-isatty v0.0.17 h1:BTarxUcIeDqL27Mc+vyvdWYSL28zpIhv3RoTdsLMPng= github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0= github.com/matttproud/golang_protobuf_extensions v1.0.4 h1:mmDVorXM7PCGKw94cs5zkfA9PSy5pEvNWRP0ET0TIVo= github.com/matttproud/golang_protobuf_extensions v1.0.4/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4= -github.com/miekg/dns v1.1.50 h1:DQUfb9uc6smULcREF09Uc+/Gd46YWqJd5DbpPE9xkcA= +github.com/miekg/dns v1.1.51 h1:0+Xg7vObnhrz/4ZCZcZh7zPXlmU0aveS2HDBd0m0qSo= github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y= github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= @@ -200,17 +200,16 @@ github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA= github.com/prometheus/procfs v0.1.3/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU= github.com/prometheus/procfs v0.6.0/go.mod h1:cz+aTbrPOrUb4q7XlbU9ygM+/jj0fzG6c1xBZuNvfVA= -github.com/prometheus/procfs v0.8.0 h1:ODq8ZFEaYeCaZOJlZZdJA2AbQR98dSHSM1KW/You5mo= -github.com/prometheus/procfs v0.8.0/go.mod h1:z7EfXMXOkbkqb9IINtpCn86r/to3BnA0uaxHdg830/4= -github.com/prometheus/prometheus v0.42.0 h1:G769v8covTkOiNckXFIwLx01XE04OE6Fr0JPA0oR2nI= -github.com/prometheus/prometheus v0.42.0/go.mod h1:Pfqb/MLnnR2KK+0vchiaH39jXxvLMBk+3lnIGP4N7Vk= -github.com/rogpeppe/go-internal v1.6.1 h1:/FiVV8dS/e+YqF2JvO3yXRFbBLTIuSDkuC7aBOAvL+k= +github.com/prometheus/procfs v0.9.0 h1:wzCHvIvM5SxWqYvwgVL7yJY8Lz3PKn49KQtpgMYJfhI= +github.com/prometheus/procfs v0.9.0/go.mod h1:+pB4zwohETzFnmlpe6yd2lSc+0/46IYZRB/chUwxUZY= +github.com/prometheus/prometheus v0.43.0 h1:18iCSfrbAHbXvYFvR38U1Pt4uZmU9SmDcCpCrBKUiGg= +github.com/prometheus/prometheus v0.43.0/go.mod h1:2BA14LgBeqlPuzObSEbh+Y+JwLH2GcqDlJKbF2sA6FM= github.com/rogpeppe/go-internal v1.6.1/go.mod h1:xXDCJY+GAPziupqXw64V24skbSoqbTEfhy4qGm1nDQc= -github.com/scaleway/scaleway-sdk-go v1.0.0-beta.12 h1:Aaz4T7dZp7cB2cv7D/tGtRdSMh48sRaDYr7Jh0HV4qQ= +github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8= +github.com/scaleway/scaleway-sdk-go v1.0.0-beta.14 h1:yFl3jyaSVLNYXlnNYM5z2pagEk1dYQhfr1p20T1NyKY= github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo= github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE= github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88= -github.com/sirupsen/logrus v1.8.1 h1:dJKuHgqk1NNQlqoA6BTlM1Wf9DOH3NBjQyu0h9+AZZE= github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= @@ -230,31 +229,30 @@ github.com/xhit/go-str2duration/v2 v2.1.0/go.mod h1:ohY8p+0f07DiV6Em5LKB0s2YpLtX github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= -go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.37.0 h1:yt2NKzK7Vyo6h0+X8BA4FpreZQTlVEIarnsBP/H5mzs= -go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.37.0/go.mod h1:+ARmXlUlc51J7sZeCBkBJNdHGySrdOzgzxp6VWRWM1U= -go.opentelemetry.io/otel v1.11.2 h1:YBZcQlsVekzFsFbjygXMOXSs6pialIZxcjfO/mBDmR0= -go.opentelemetry.io/otel v1.11.2/go.mod h1:7p4EUV+AqgdlNV9gL97IgUZiVR3yrFXYo53f9BM3tRI= -go.opentelemetry.io/otel/metric v0.34.0 h1:MCPoQxcg/26EuuJwpYN1mZTeCYAUGx8ABxfW07YkjP8= -go.opentelemetry.io/otel/metric v0.34.0/go.mod h1:ZFuI4yQGNCupurTXCwkeD/zHBt+C2bR7bw5JqUm/AP8= -go.opentelemetry.io/otel/trace v1.11.2 h1:Xf7hWSF2Glv0DE3MH7fBHvtpSBsjcBUe5MYAmZM/+y0= -go.opentelemetry.io/otel/trace v1.11.2/go.mod h1:4N+yC7QEz7TTsG9BSRLNAa63eg5E06ObSbKPmxQ/pKA= +go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.40.0 h1:lE9EJyw3/JhrjWH/hEy9FptnalDQgj7vpbgC2KCCCxE= +go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.40.0/go.mod h1:pcQ3MM3SWvrA71U4GDqv9UFDJ3HQsW7y5ZO3tDTlUdI= +go.opentelemetry.io/otel v1.14.0 h1:/79Huy8wbf5DnIPhemGB+zEPVwnN6fuQybr/SRXa6hM= +go.opentelemetry.io/otel v1.14.0/go.mod h1:o4buv+dJzx8rohcUeRmWUZhqupFvzWis188WlggnNeU= +go.opentelemetry.io/otel/metric v0.37.0 h1:pHDQuLQOZwYD+Km0eb657A25NaRzy0a+eLyKfDXedEs= +go.opentelemetry.io/otel/metric v0.37.0/go.mod h1:DmdaHfGt54iV6UKxsV9slj2bBRJcKC1B1uvDLIioc1s= +go.opentelemetry.io/otel/trace v1.14.0 h1:wp2Mmvj41tDsyAJXiWDWpfNsOiIyd38fy85pyKcFq/M= +go.opentelemetry.io/otel/trace v1.14.0/go.mod h1:8avnQLK+CG77yNLUae4ea2JDQ6iT+gozhnZjy/rw9G8= go.uber.org/atomic v1.10.0 h1:9qC72Qh0+3MqyJbAn8YU5xVq1frD8bn3JtD2oXtafVQ= go.uber.org/atomic v1.10.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0= -go.uber.org/goleak v1.2.0 h1:xqgm/S+aQvhWFTtR0XK3Jvg7z8kGV8P4X14IzwN3Eqk= -go.uber.org/goleak v1.2.0/go.mod h1:XJYK+MuIchqpmGmUSAzotztawfKvYLUIgg7guXrwVUo= +go.uber.org/goleak v1.2.1 h1:NBol2c7O1ZokfZ0LEU9K6Whx/KnwvepVetCUhtKja4A= +go.uber.org/goleak v1.2.1/go.mod h1:qlT2yGI9QafXHhZZLxlSuNsMw3FFLxBr+tBRlmO1xH4= golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= -golang.org/x/crypto v0.1.0 h1:MDRAIl0xIo9Io2xV565hzXHw3zVseKrJKodhohM5CjU= -golang.org/x/exp v0.0.0-20230124195608-d38c7dcee874 h1:kWC3b7j6Fu09SnEBr7P4PuQyM0R6sqyH9R+EjIvT1nQ= -golang.org/x/exp v0.0.0-20230124195608-d38c7dcee874/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc= -golang.org/x/lint v0.0.0-20210508222113-6edffad5e616 h1:VLliZ0d+/avPrXXH+OakdXhpJuEoBZuwh1m2j7U6Iug= +golang.org/x/crypto v0.7.0 h1:AvwMYaRytfdeVt3u6mLaxYtErKYjxA2OXjJ1HHq6t3A= +golang.org/x/exp v0.0.0-20230307190834-24139beb5833 h1:SChBja7BCQewoTAU7IgvucQKMIXrEpFxNMs0spT3/5s= +golang.org/x/exp v0.0.0-20230307190834-24139beb5833/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= -golang.org/x/mod v0.7.0 h1:LapD9S96VoQRhi/GrNTqeBJFrUjs5UHCAtTlgwA5oZA= +golang.org/x/mod v0.9.0 h1:KENHtAZL2y3NLMYZeHY9DW8HW8V+kQyJsY/V9JlKvCs= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -271,12 +269,12 @@ golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.1.0/go.mod h1:Cx3nUiGt4eDBEyega/BKRp+/AlGL8hYe7U9odMt2Cco= -golang.org/x/net v0.7.0 h1:rJrUqqhjsgNp7KqAIc25s9pZnjU7TUcSY7HcVZjdn1g= -golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= +golang.org/x/net v0.8.0 h1:Zrh2ngAOFYneWTAIAPethzeaQLuHwhuBkuV6ZiRnUaQ= +golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.5.0 h1:HuArIo48skDwlrvM3sEdHXElYslAMsf3KwRkkW4MC4s= -golang.org/x/oauth2 v0.5.0/go.mod h1:9/XBHVqLaWO3/BRHs5jbpYCnOZVjj5V0ndyaAM7KB4I= +golang.org/x/oauth2 v0.6.0 h1:Lh8GPgSKBfWSwFvtuWOfeI3aAAnbXTSutYxJiOJFgIw= +golang.org/x/oauth2 v0.6.0/go.mod h1:ycmewcwgD4Rpr3eZJLSB4Kyyljb3qDh40vJ8STE5HKw= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -303,20 +301,20 @@ golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.5.0 h1:MUK/U/4lj1t1oPg0HfuXDN/Z1wv31ZJ/YcPiGccS4DU= -golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.6.0 h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ= +golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.1.0/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= -golang.org/x/term v0.5.0 h1:n2a8QNdAb0sZNpU9R1ALUXBbY+w51fCQDN+7EdxNBsY= +golang.org/x/term v0.6.0 h1:clScbb1cHjoCkyRbWwBEUZ5H/tIFu5TAXIqaZD0Gcjw= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= -golang.org/x/text v0.7.0 h1:4BRB4x83lYWy72KwLD/qYDuTu7q9PjSagHvijDw7cLo= -golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.8.0 h1:57P1ETyNKtuIjB4SRd15iJxuhj8Gc416Y78H3qgMh68= +golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4= golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= @@ -324,7 +322,7 @@ golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtn golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= -golang.org/x/tools v0.5.0 h1:+bSpV5HIeWkuvgaMfI3UmKRThoTA5ODJTUd8T17NO+4= +golang.org/x/tools v0.7.0 h1:W4OVu8VVOaIO0yzWMNdepAulS7YfoS3Zabrm8DOXXU4= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= @@ -333,8 +331,8 @@ google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7 google.golang.org/appengine v1.6.6/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc= google.golang.org/appengine v1.6.7 h1:FZR1q0exgwxzPzp/aF+VccGrSfxfPpkBqjIIEq3ru6c= google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc= -google.golang.org/genproto v0.0.0-20230124163310-31e0e69b6fc2 h1:O97sLx/Xmb/KIZHB/2/BzofxBs5QmmR0LcihPtllmbc= -google.golang.org/grpc v1.52.1 h1:2NpOPk5g5Xtb0qebIEs7hNIa++PdtZLo2AQUpc1YnSU= +google.golang.org/genproto v0.0.0-20230306155012-7f2fa6fef1f4 h1:DdoeryqhaXp1LtT/emMP1BRJPHHKFi5akj/nbx/zNTA= +google.golang.org/grpc v1.53.0 h1:LAv2ds7cmFV/XTS3XG1NneeENYrXGmorPxsBbptIjNc= google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= @@ -343,8 +341,8 @@ google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzi google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= -google.golang.org/protobuf v1.28.1 h1:d0NfwRgPtno5B1Wa6L2DAG+KivqkdutMf1UhdNx175w= -google.golang.org/protobuf v1.28.1/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= +google.golang.org/protobuf v1.29.0 h1:44S3JjaKmLEE4YIkjzexaP+NzZsudE3Zin5Njn/pYX0= +google.golang.org/protobuf v1.29.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= @@ -352,7 +350,7 @@ gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc= -gopkg.in/ini.v1 v1.66.6 h1:LATuAqN/shcYAOkv3wl2L4rkaKqkcgTBQjOyYDvcPKI= +gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA= gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= @@ -364,13 +362,13 @@ gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= -k8s.io/api v0.26.1 h1:f+SWYiPd/GsiWwVRz+NbFyCgvv75Pk9NK6dlkZgpCRQ= -k8s.io/apimachinery v0.26.1 h1:8EZ/eGJL+hY/MYCNwhmDzVqq2lPl3N3Bo8rvweJwXUQ= -k8s.io/client-go v0.26.1 h1:87CXzYJnAMGaa/IDDfRdhTzxk/wzGZ+/HUQpqgVSZXU= +k8s.io/api v0.26.2 h1:dM3cinp3PGB6asOySalOZxEG4CZ0IAdJsrYZXE/ovGQ= +k8s.io/apimachinery v0.26.2 h1:da1u3D5wfR5u2RpLhE/ZtZS2P7QvDgLZTi9wrNZl/tQ= +k8s.io/client-go v0.26.2 h1:s1WkVujHX3kTp4Zn4yGNFK+dlDXy1bAAkIl+cFAiuYI= k8s.io/klog v1.0.0 h1:Pt+yjF5aB1xDSVbau4VsWe+dQNzA0qv1LlXdC2dF6Q8= -k8s.io/klog/v2 v2.80.1 h1:atnLQ121W371wYYFawwYx1aEY2eUfs4l3J72wtgAwV4= -k8s.io/kube-openapi v0.0.0-20221207184640-f3cff1453715 h1:tBEbstoM+K0FiBV5KGAKQ0kuvf54v/hwpldiJt69w1s= -k8s.io/utils v0.0.0-20221128185143-99ec85e7a448 h1:KTgPnR10d5zhztWptI952TNtt/4u5h3IzDXkdIMuo2Y= +k8s.io/klog/v2 v2.90.1 h1:m4bYOKall2MmOiRaR1J+We67Do7vm9KiQVlT96lnHUw= +k8s.io/kube-openapi v0.0.0-20230303024457-afdc3dddf62d h1:VcFq5n7wCJB2FQMCIHfC+f+jNcGgNMar1uKd6rVlifU= +k8s.io/utils v0.0.0-20230308161112-d77c459e9343 h1:m7tbIjXGcGIAtpmQr7/NAi7RsWoW3E7Zcm4jI1HicTc= sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMmr1bNJefnuqLsRAsHZo= sigs.k8s.io/structured-merge-diff/v4 v4.2.3 h1:PRbqxJClWWYMNV1dhaG4NsibJbArud9kFxnAMREiWFE= sigs.k8s.io/yaml v1.3.0 h1:a2VclLzOGrwOHDiV8EfBGhvjHvP46CtW5j6POvhYGGo= From 408f31f78669a27e92aac41452a4f67dfb44f917 Mon Sep 17 00:00:00 2001 From: Hayk Davtyan <46712946+hayk96@users.noreply.github.com> Date: Sun, 2 Apr 2023 13:37:58 +0400 Subject: [PATCH 059/231] [WebUI/ScrapePoolList] Case-insensitive search of "Scrape Pools" (#12207) Signed-off-by: hayk96 --- .../react-app/src/pages/targets/ScrapePoolList.tsx | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/web/ui/react-app/src/pages/targets/ScrapePoolList.tsx b/web/ui/react-app/src/pages/targets/ScrapePoolList.tsx index c9e2947e63..29585a2007 100644 --- a/web/ui/react-app/src/pages/targets/ScrapePoolList.tsx +++ b/web/ui/react-app/src/pages/targets/ScrapePoolList.tsx @@ -31,6 +31,8 @@ const ScrapePoolDropDown: FC = ({ selectedPool, scrapeP const [filter, setFilter] = useState(''); + const filteredPools = scrapePools.filter((pool) => pool.toLowerCase().includes(filter.toLowerCase())); + return ( @@ -51,13 +53,11 @@ const ScrapePoolDropDown: FC = ({ selectedPool, scrapeP {scrapePools.length === 0 ? ( No scrape pools configured ) : ( - scrapePools - .filter((name) => filter === '' || name.includes(filter)) - .map((name) => ( - onScrapePoolChange(name)} active={name === selectedPool}> - {name} - - )) + filteredPools.map((name) => ( + onScrapePoolChange(name)} active={name === selectedPool}> + {name} + + )) )} From 889ef998f4f73d0d1fbd961e59494584e3616e42 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Sun, 2 Apr 2023 11:00:14 +0100 Subject: [PATCH 060/231] remote-write: adjust MaxShards and Capacity for larger MaxSamplesPerSend Keep the target throughput and total pending samples the same. Signed-off-by: Bryan Boreham --- config/config.go | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/config/config.go b/config/config.go index 9b0f4ad1e8..5c51d5a0d8 100644 --- a/config/config.go +++ b/config/config.go @@ -173,16 +173,16 @@ var ( // DefaultQueueConfig is the default remote queue configuration. DefaultQueueConfig = QueueConfig{ - // With a maximum of 200 shards, assuming an average of 100ms remote write - // time and 2000 samples per batch, we will be able to push 4M samples/s. - MaxShards: 200, + // With a maximum of 50 shards, assuming an average of 100ms remote write + // time and 2000 samples per batch, we will be able to push 1M samples/s. + MaxShards: 50, MinShards: 1, MaxSamplesPerSend: 2000, - // Each shard will have a max of 2500 samples pending in its channel, plus the pending - // samples that have been enqueued. Theoretically we should only ever have about 3000 samples - // per shard pending. At 200 shards that's 600k. - Capacity: 2500, + // Each shard will have a max of 10,000 samples pending in its channel, plus the pending + // samples that have been enqueued. Theoretically we should only ever have about 12,000 samples + // per shard pending. At 50 shards that's 600k. + Capacity: 10000, BatchSendDeadline: model.Duration(5 * time.Second), // Backoff times for retrying a batch of samples on recoverable errors. From e91720276654d0180f5dfa655f4561cb5c87e986 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Sun, 2 Apr 2023 11:17:05 +0100 Subject: [PATCH 061/231] labels: make sure estimated size is not negative Deleted labels are remembered, even if they were not in `base` or were removed from `add`, so `base+add-del` could go negative. Signed-off-by: Bryan Boreham --- model/labels/labels.go | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/model/labels/labels.go b/model/labels/labels.go index 4e87eb9b34..e9d6c940e3 100644 --- a/model/labels/labels.go +++ b/model/labels/labels.go @@ -577,7 +577,11 @@ func (b *Builder) Labels() Labels { return b.base } - res := make(Labels, 0, len(b.base)-len(b.del)+len(b.add)) + expectedSize := len(b.base) + len(b.add) - len(b.del) + if expectedSize < 1 { + expectedSize = 1 + } + res := make(Labels, 0, expectedSize) for _, l := range b.base { if slices.Contains(b.del, l.Name) || contains(b.add, l.Name) { continue From 8b011893276e357181c65c4c6dd095228e92a49b Mon Sep 17 00:00:00 2001 From: SuperQ Date: Mon, 3 Apr 2023 09:05:10 +0200 Subject: [PATCH 062/231] Move errcheck excludes config Eliminate the need for a second config file for golangci-lint config file by moving the list of errcheck exclude functions into the yaml config. Signed-off-by: SuperQ --- .github/workflows/ci.yml | 1 + .golangci.yml | 13 ++++++++++++- scripts/errcheck_excludes.txt | 13 ------------- 3 files changed, 13 insertions(+), 14 deletions(-) delete mode 100644 scripts/errcheck_excludes.txt diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ceb374c8c5..1e625b63be 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -3,6 +3,7 @@ name: CI on: pull_request: push: + jobs: test_go: name: Go tests diff --git a/.golangci.yml b/.golangci.yml index 81790e6e3a..efa6b2044d 100644 --- a/.golangci.yml +++ b/.golangci.yml @@ -33,7 +33,18 @@ linters-settings: - io/ioutil: "Use corresponding 'os' or 'io' functions instead." - regexp: "Use github.com/grafana/regexp instead of regexp" errcheck: - exclude: scripts/errcheck_excludes.txt + exclude-functions: + # Don't flag lines such as "io.Copy(io.Discard, resp.Body)". + - io.Copy + # The next two are used in HTTP handlers, any error is handled by the server itself. + - io.WriteString + - (net/http.ResponseWriter).Write + # No need to check for errors on server's shutdown. + - (*net/http.Server).Shutdown + # Never check for logger errors. + - (github.com/go-kit/log.Logger).Log + # Never check for rollback errors as Rollback() is called when a previous error was detected. + - (github.com/prometheus/prometheus/storage.Appender).Rollback goimports: local-prefixes: github.com/prometheus/prometheus gofumpt: diff --git a/scripts/errcheck_excludes.txt b/scripts/errcheck_excludes.txt deleted file mode 100644 index 8c77ca148d..0000000000 --- a/scripts/errcheck_excludes.txt +++ /dev/null @@ -1,13 +0,0 @@ -// Don't flag lines such as "io.Copy(io.Discard, resp.Body)". -io.Copy -// The next two are used in HTTP handlers, any error is handled by the server itself. -io.WriteString -(net/http.ResponseWriter).Write -// No need to check for errors on server's shutdown. -(*net/http.Server).Shutdown - -// Never check for logger errors. -(github.com/go-kit/log.Logger).Log - -// Never check for rollback errors as Rollback() is called when a previous error was detected. -(github.com/prometheus/prometheus/storage.Appender).Rollback From f2b9a39a48350c04a64541c934eb2a6e475fc9b9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Mierzwa?= Date: Fri, 31 Mar 2023 14:31:14 +0100 Subject: [PATCH 063/231] Use a random port in cmd/prometheus tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit There are a few tests that will run prometheus command. This can test if there's already something listening on port :9090 since --web.listen-address defaults to 0.0.0.0:9090. To fix that we can tell prometheus to use a random port on loopback interface. Signed-off-by: Łukasz Mierzwa --- cmd/prometheus/main_test.go | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/cmd/prometheus/main_test.go b/cmd/prometheus/main_test.go index 26d11e21e6..21447d0369 100644 --- a/cmd/prometheus/main_test.go +++ b/cmd/prometheus/main_test.go @@ -121,7 +121,7 @@ func TestFailedStartupExitCode(t *testing.T) { fakeInputFile := "fake-input-file" expectedExitStatus := 2 - prom := exec.Command(promPath, "-test.main", "--config.file="+fakeInputFile) + prom := exec.Command(promPath, "-test.main", "--web.listen-address=0.0.0.0:0", "--config.file="+fakeInputFile) err := prom.Run() require.Error(t, err) @@ -358,7 +358,7 @@ func getCurrentGaugeValuesFor(t *testing.T, reg prometheus.Gatherer, metricNames } func TestAgentSuccessfulStartup(t *testing.T) { - prom := exec.Command(promPath, "-test.main", "--enable-feature=agent", "--config.file="+agentConfig) + prom := exec.Command(promPath, "-test.main", "--enable-feature=agent", "--web.listen-address=0.0.0.0:0", "--config.file="+agentConfig) require.NoError(t, prom.Start()) actualExitStatus := 0 @@ -376,7 +376,7 @@ func TestAgentSuccessfulStartup(t *testing.T) { } func TestAgentFailedStartupWithServerFlag(t *testing.T) { - prom := exec.Command(promPath, "-test.main", "--enable-feature=agent", "--storage.tsdb.path=.", "--config.file="+promConfig) + prom := exec.Command(promPath, "-test.main", "--enable-feature=agent", "--storage.tsdb.path=.", "--web.listen-address=0.0.0.0:0", "--config.file="+promConfig) output := bytes.Buffer{} prom.Stderr = &output @@ -403,7 +403,7 @@ func TestAgentFailedStartupWithServerFlag(t *testing.T) { } func TestAgentFailedStartupWithInvalidConfig(t *testing.T) { - prom := exec.Command(promPath, "-test.main", "--enable-feature=agent", "--config.file="+promConfig) + prom := exec.Command(promPath, "-test.main", "--enable-feature=agent", "--web.listen-address=0.0.0.0:0", "--config.file="+promConfig) require.NoError(t, prom.Start()) actualExitStatus := 0 @@ -438,7 +438,7 @@ func TestModeSpecificFlags(t *testing.T) { for _, tc := range testcases { t.Run(fmt.Sprintf("%s mode with option %s", tc.mode, tc.arg), func(t *testing.T) { - args := []string{"-test.main", tc.arg, t.TempDir()} + args := []string{"-test.main", tc.arg, t.TempDir(), "--web.listen-address=0.0.0.0:0"} if tc.mode == "agent" { args = append(args, "--enable-feature=agent", "--config.file="+agentConfig) From 291ab4d0bc6ea9f0e61073f303bd3982e9727b8d Mon Sep 17 00:00:00 2001 From: Ganesh Vernekar Date: Mon, 3 Apr 2023 19:41:31 +0530 Subject: [PATCH 064/231] =?UTF-8?q?Add=20Jes=C3=BAs=20V=C3=A1zquez=20as=20?= =?UTF-8?q?a=20TSDB=20maintainer=20(#12222)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Ganesh Vernekar --- MAINTAINERS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MAINTAINERS.md b/MAINTAINERS.md index 103ebdda64..1175bb9a6f 100644 --- a/MAINTAINERS.md +++ b/MAINTAINERS.md @@ -10,7 +10,7 @@ Julien Pivotto ( / @roidelapluie) and Levi Harrison * `prometheus-mixin`: Björn Rabenstein ( / @beorn7) * `storage` * `remote`: Chris Marchbanks ( / @csmarchbanks), Callum Styan ( / @cstyan), Bartłomiej Płotka ( / @bwplotka), Tom Wilkie ( / @tomwilkie) -* `tsdb`: Ganesh Vernekar ( / @codesome), Bartłomiej Płotka ( / @bwplotka) +* `tsdb`: Ganesh Vernekar ( / @codesome), Bartłomiej Płotka ( / @bwplotka), Jesús Vázquez ( / @jesusvazquez) * `agent`: Robert Fratto ( / @rfratto) * `web` * `ui`: Julius Volz ( / @juliusv) From c8b408cf5e60cc57fbcc087ca86ade99552097f5 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 3 Apr 2023 20:16:09 +0000 Subject: [PATCH 065/231] build(deps): bump github.com/Azure/go-autorest/autorest/adal Bumps [github.com/Azure/go-autorest/autorest/adal](https://github.com/Azure/go-autorest) from 0.9.22 to 0.9.23. - [Release notes](https://github.com/Azure/go-autorest/releases) - [Changelog](https://github.com/Azure/go-autorest/blob/main/CHANGELOG.md) - [Commits](https://github.com/Azure/go-autorest/compare/autorest/adal/v0.9.22...autorest/adal/v0.9.23) --- updated-dependencies: - dependency-name: github.com/Azure/go-autorest/autorest/adal dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] --- go.mod | 2 +- go.sum | 9 +++++++-- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index ef3e738158..92cc3931c7 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ go 1.19 require ( github.com/Azure/azure-sdk-for-go v65.0.0+incompatible github.com/Azure/go-autorest/autorest v0.11.28 - github.com/Azure/go-autorest/autorest/adal v0.9.22 + github.com/Azure/go-autorest/autorest/adal v0.9.23 github.com/alecthomas/kingpin/v2 v2.3.2 github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 github.com/aws/aws-sdk-go v1.44.217 diff --git a/go.sum b/go.sum index e9c973ba4a..7ea1be034d 100644 --- a/go.sum +++ b/go.sum @@ -45,8 +45,8 @@ github.com/Azure/go-autorest v14.2.0+incompatible/go.mod h1:r+4oMnoxhatjLLJ6zxSW github.com/Azure/go-autorest/autorest v0.11.28 h1:ndAExarwr5Y+GaHE6VCaY1kyS/HwwGGyuimVhWsHOEM= github.com/Azure/go-autorest/autorest v0.11.28/go.mod h1:MrkzG3Y3AH668QyF9KRk5neJnGgmhQ6krbhR8Q5eMvA= github.com/Azure/go-autorest/autorest/adal v0.9.18/go.mod h1:XVVeme+LZwABT8K5Lc3hA4nAe8LDBVle26gTrguhhPQ= -github.com/Azure/go-autorest/autorest/adal v0.9.22 h1:/GblQdIudfEM3AWWZ0mrYJQSd7JS4S/Mbzh6F0ov0Xc= -github.com/Azure/go-autorest/autorest/adal v0.9.22/go.mod h1:XuAbAEUv2Tta//+voMI038TrJBqjKam0me7qR+L8Cmk= +github.com/Azure/go-autorest/autorest/adal v0.9.23 h1:Yepx8CvFxwNKpH6ja7RZ+sKX+DWYNldbLiALMC3BTz8= +github.com/Azure/go-autorest/autorest/adal v0.9.23/go.mod h1:5pcMqFkdPhviJdlEy3kC/v1ZLnQl0MH6XA5YCcMhy4c= github.com/Azure/go-autorest/autorest/date v0.3.0 h1:7gUk1U5M/CQbp9WoqinNzJar+8KY+LPI6wiWrP/myHw= github.com/Azure/go-autorest/autorest/date v0.3.0/go.mod h1:BI0uouVdmngYNUzGWeSYnokU+TrmwEsOqdt8Y6sso74= github.com/Azure/go-autorest/autorest/mocks v0.4.1/go.mod h1:LTp+uSrOhSkaKrUy935gNZuuIPPVsHlr9DSOxSayd+k= @@ -826,6 +826,7 @@ golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5y golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.0.0-20220829220503-c86fa9a7ed90/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= +golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58= golang.org/x/crypto v0.7.0 h1:AvwMYaRytfdeVt3u6mLaxYtErKYjxA2OXjJ1HHq6t3A= golang.org/x/crypto v0.7.0/go.mod h1:pYwdfH91IfpZVANVyUOhSIPZaFoJGxTFbZhFTx+dXZU= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= @@ -909,6 +910,7 @@ golang.org/x/net v0.0.0-20211029224645-99673261e6eb/go.mod h1:9nx3DQGgdP8bBQD5qx golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.1.0/go.mod h1:Cx3nUiGt4eDBEyega/BKRp+/AlGL8hYe7U9odMt2Cco= +golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/net v0.8.0 h1:Zrh2ngAOFYneWTAIAPethzeaQLuHwhuBkuV6ZiRnUaQ= golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= @@ -1000,11 +1002,13 @@ golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.6.0 h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ= golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.1.0/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= +golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= golang.org/x/term v0.6.0 h1:clScbb1cHjoCkyRbWwBEUZ5H/tIFu5TAXIqaZD0Gcjw= golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U= golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= @@ -1016,6 +1020,7 @@ golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/text v0.8.0 h1:57P1ETyNKtuIjB4SRd15iJxuhj8Gc416Y78H3qgMh68= golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= From 3f7beeecc69ffe83ebaa4def073f644e178bd131 Mon Sep 17 00:00:00 2001 From: Nidhey Nitin Indurkar <46122307+nidhey27@users.noreply.github.com> Date: Tue, 4 Apr 2023 02:02:39 +0530 Subject: [PATCH 066/231] feat: health and readiness check of prometheus server in CLI (promtool) (#12096) * feat: health and readiness check of prometheus server in CLI (promtool) Signed-off-by: nidhey27 --- cmd/promtool/main.go | 60 +++++++++++++++++++++++++++++++++++ docs/command-line/promtool.md | 31 ++++++++++++++++++ 2 files changed, 91 insertions(+) diff --git a/cmd/promtool/main.go b/cmd/promtool/main.go index 3988957efa..064e7a04f7 100644 --- a/cmd/promtool/main.go +++ b/cmd/promtool/main.go @@ -71,6 +71,8 @@ const ( lintOptionAll = "all" lintOptionDuplicateRules = "duplicate-rules" lintOptionNone = "none" + checkHealth = "/-/healthy" + checkReadiness = "/-/ready" ) var lintOptions = []string{lintOptionAll, lintOptionDuplicateRules, lintOptionNone} @@ -87,6 +89,7 @@ func main() { app.HelpFlag.Short('h') checkCmd := app.Command("check", "Check the resources for validity.") + checkCmd.Flag("http.config.file", "HTTP client configuration file for promtool to connect to Prometheus.").PlaceHolder("").ExistingFileVar(&httpConfigFilePath) sdCheckCmd := checkCmd.Command("service-discovery", "Perform service discovery for the given job name and report the results, including relabeling.") sdConfigFile := sdCheckCmd.Arg("config-file", "The prometheus config file.").Required().ExistingFile() @@ -113,6 +116,18 @@ func main() { "The config files to check.", ).Required().ExistingFiles() + checkServerHealthCmd := checkCmd.Command("healthy", "Check if the Prometheus server is healthy.") + serverHealthURLArg := checkServerHealthCmd.Arg( + "server", + "The URL of the Prometheus server to check (e.g. http://localhost:9090)", + ).URL() + + checkServerReadyCmd := checkCmd.Command("ready", "Check if the Prometheus server is ready.") + serverReadyURLArg := checkServerReadyCmd.Arg( + "server", + "The URL of the Prometheus server to check (e.g. http://localhost:9090)", + ).URL() + checkRulesCmd := checkCmd.Command("rules", "Check if the rule files are valid or not.") ruleFiles := checkRulesCmd.Arg( "rule-files", @@ -276,6 +291,12 @@ func main() { case checkConfigCmd.FullCommand(): os.Exit(CheckConfig(*agentMode, *checkConfigSyntaxOnly, newLintConfig(*checkConfigLint, *checkConfigLintFatal), *configFiles...)) + case checkServerHealthCmd.FullCommand(): + os.Exit(checkErr(CheckServerStatus(*serverHealthURLArg, checkHealth, httpRoundTripper))) + + case checkServerReadyCmd.FullCommand(): + os.Exit(checkErr(CheckServerStatus(*serverReadyURLArg, checkReadiness, httpRoundTripper))) + case checkWebConfigCmd.FullCommand(): os.Exit(CheckWebConfig(*webConfigFiles...)) @@ -369,6 +390,45 @@ func (ls lintConfig) lintDuplicateRules() bool { return ls.all || ls.duplicateRules } +const promDefaultURL = "http://localhost:9090" + +// Check server status - healthy & ready. +func CheckServerStatus(serverURL *url.URL, checkEndpoint string, roundTripper http.RoundTripper) error { + if serverURL == nil { + serverURL, _ = url.Parse(promDefaultURL) + } + + config := api.Config{ + Address: serverURL.String() + checkEndpoint, + RoundTripper: roundTripper, + } + + // Create new client. + c, err := api.NewClient(config) + if err != nil { + fmt.Fprintln(os.Stderr, "error creating API client:", err) + return err + } + + request, err := http.NewRequest("GET", config.Address, nil) + if err != nil { + return err + } + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + response, dataBytes, err := c.Do(ctx, request) + if err != nil { + return err + } + + if response.StatusCode != http.StatusOK { + return fmt.Errorf("check failed: URL=%s, status=%d", serverURL, response.StatusCode) + } + + fmt.Fprintln(os.Stderr, " SUCCESS: ", string(dataBytes)) + return nil +} + // CheckConfig validates configuration files. func CheckConfig(agentMode, checkSyntaxOnly bool, lintSettings lintConfig, files ...string) int { failed := false diff --git a/docs/command-line/promtool.md b/docs/command-line/promtool.md index 42d853b856..543034f157 100644 --- a/docs/command-line/promtool.md +++ b/docs/command-line/promtool.md @@ -58,6 +58,7 @@ Check the resources for validity. | Flag | Description | | --- | --- | +| --http.config.file | HTTP client configuration file for promtool to connect to Prometheus. | | --extended | Print extended information related to the cardinality of the metrics. | @@ -130,6 +131,36 @@ Check if the web config files are valid or not. +##### `promtool check healthy` + +Check if the Prometheus server is healthy. + + + +###### Arguments + +| Argument | Description | +| --- | --- | +| server | The URL of the Prometheus server to check (e.g. http://localhost:9090) | + + + + +##### `promtool check ready` + +Check if the Prometheus server is ready. + + + +###### Arguments + +| Argument | Description | +| --- | --- | +| server | The URL of the Prometheus server to check (e.g. http://localhost:9090) | + + + + ##### `promtool check rules` Check if the rule files are valid or not. From 3923e83413075a17c5a10739696a879765a05542 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 3 Apr 2023 23:25:07 +0200 Subject: [PATCH 067/231] build(deps): bump bufbuild/buf-setup-action from 1.13.1 to 1.16.0 (#12209) Bumps [bufbuild/buf-setup-action](https://github.com/bufbuild/buf-setup-action) from 1.13.1 to 1.16.0. - [Release notes](https://github.com/bufbuild/buf-setup-action/releases) - [Commits](https://github.com/bufbuild/buf-setup-action/compare/v1.13.1...v1.16.0) --- updated-dependencies: - dependency-name: bufbuild/buf-setup-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/buf-lint.yml | 2 +- .github/workflows/buf.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/buf-lint.yml b/.github/workflows/buf-lint.yml index 5d93c7dfed..3275d08fc3 100644 --- a/.github/workflows/buf-lint.yml +++ b/.github/workflows/buf-lint.yml @@ -10,7 +10,7 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - - uses: bufbuild/buf-setup-action@v1.13.1 + - uses: bufbuild/buf-setup-action@v1.16.0 with: github_token: ${{ secrets.GITHUB_TOKEN }} - uses: bufbuild/buf-lint-action@v1 diff --git a/.github/workflows/buf.yml b/.github/workflows/buf.yml index 2bfee71383..c80183b18a 100644 --- a/.github/workflows/buf.yml +++ b/.github/workflows/buf.yml @@ -9,7 +9,7 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - - uses: bufbuild/buf-setup-action@v1.13.1 + - uses: bufbuild/buf-setup-action@v1.16.0 with: github_token: ${{ secrets.GITHUB_TOKEN }} - uses: bufbuild/buf-lint-action@v1 From 1936868e9d13e2aeda277f8d0782fc4739612ba8 Mon Sep 17 00:00:00 2001 From: Alex Le Date: Mon, 3 Apr 2023 23:31:49 -0700 Subject: [PATCH 068/231] Allow populate block logic in compact to be overriden outside Prometheus (#11711) Signed-off-by: Alex Le Signed-off-by: Alex Le --- tsdb/compact.go | 104 ++++++++++++++++++++++++------------------- tsdb/compact_test.go | 19 ++++---- tsdb/db_test.go | 10 ++--- tsdb/querier.go | 6 +-- 4 files changed, 75 insertions(+), 64 deletions(-) diff --git a/tsdb/compact.go b/tsdb/compact.go index 26a7c78c8a..b2d4123754 100644 --- a/tsdb/compact.go +++ b/tsdb/compact.go @@ -75,7 +75,7 @@ type Compactor interface { // LeveledCompactor implements the Compactor interface. type LeveledCompactor struct { - metrics *compactorMetrics + metrics *CompactorMetrics logger log.Logger ranges []int64 chunkPool chunkenc.Pool @@ -84,47 +84,47 @@ type LeveledCompactor struct { mergeFunc storage.VerticalChunkSeriesMergeFunc } -type compactorMetrics struct { - ran prometheus.Counter - populatingBlocks prometheus.Gauge - overlappingBlocks prometheus.Counter - duration prometheus.Histogram - chunkSize prometheus.Histogram - chunkSamples prometheus.Histogram - chunkRange prometheus.Histogram +type CompactorMetrics struct { + Ran prometheus.Counter + PopulatingBlocks prometheus.Gauge + OverlappingBlocks prometheus.Counter + Duration prometheus.Histogram + ChunkSize prometheus.Histogram + ChunkSamples prometheus.Histogram + ChunkRange prometheus.Histogram } -func newCompactorMetrics(r prometheus.Registerer) *compactorMetrics { - m := &compactorMetrics{} +func newCompactorMetrics(r prometheus.Registerer) *CompactorMetrics { + m := &CompactorMetrics{} - m.ran = prometheus.NewCounter(prometheus.CounterOpts{ + m.Ran = prometheus.NewCounter(prometheus.CounterOpts{ Name: "prometheus_tsdb_compactions_total", Help: "Total number of compactions that were executed for the partition.", }) - m.populatingBlocks = prometheus.NewGauge(prometheus.GaugeOpts{ + m.PopulatingBlocks = prometheus.NewGauge(prometheus.GaugeOpts{ Name: "prometheus_tsdb_compaction_populating_block", Help: "Set to 1 when a block is currently being written to the disk.", }) - m.overlappingBlocks = prometheus.NewCounter(prometheus.CounterOpts{ + m.OverlappingBlocks = prometheus.NewCounter(prometheus.CounterOpts{ Name: "prometheus_tsdb_vertical_compactions_total", Help: "Total number of compactions done on overlapping blocks.", }) - m.duration = prometheus.NewHistogram(prometheus.HistogramOpts{ + m.Duration = prometheus.NewHistogram(prometheus.HistogramOpts{ Name: "prometheus_tsdb_compaction_duration_seconds", Help: "Duration of compaction runs", Buckets: prometheus.ExponentialBuckets(1, 2, 14), }) - m.chunkSize = prometheus.NewHistogram(prometheus.HistogramOpts{ + m.ChunkSize = prometheus.NewHistogram(prometheus.HistogramOpts{ Name: "prometheus_tsdb_compaction_chunk_size_bytes", Help: "Final size of chunks on their first compaction", Buckets: prometheus.ExponentialBuckets(32, 1.5, 12), }) - m.chunkSamples = prometheus.NewHistogram(prometheus.HistogramOpts{ + m.ChunkSamples = prometheus.NewHistogram(prometheus.HistogramOpts{ Name: "prometheus_tsdb_compaction_chunk_samples", Help: "Final number of samples on their first compaction", Buckets: prometheus.ExponentialBuckets(4, 1.5, 12), }) - m.chunkRange = prometheus.NewHistogram(prometheus.HistogramOpts{ + m.ChunkRange = prometheus.NewHistogram(prometheus.HistogramOpts{ Name: "prometheus_tsdb_compaction_chunk_range_seconds", Help: "Final time range of chunks on their first compaction", Buckets: prometheus.ExponentialBuckets(100, 4, 10), @@ -132,13 +132,13 @@ func newCompactorMetrics(r prometheus.Registerer) *compactorMetrics { if r != nil { r.MustRegister( - m.ran, - m.populatingBlocks, - m.overlappingBlocks, - m.duration, - m.chunkRange, - m.chunkSamples, - m.chunkSize, + m.Ran, + m.PopulatingBlocks, + m.OverlappingBlocks, + m.Duration, + m.ChunkRange, + m.ChunkSamples, + m.ChunkSize, ) } return m @@ -392,6 +392,10 @@ func CompactBlockMetas(uid ulid.ULID, blocks ...*BlockMeta) *BlockMeta { // Compact creates a new block in the compactor's directory from the blocks in the // provided directories. func (c *LeveledCompactor) Compact(dest string, dirs []string, open []*Block) (uid ulid.ULID, err error) { + return c.CompactWithPopulateBlockFunc(dest, dirs, open, DefaultPopulateBlockFunc{}) +} + +func (c *LeveledCompactor) CompactWithPopulateBlockFunc(dest string, dirs []string, open []*Block, populateBlockFunc PopulateBlockFunc) (uid ulid.ULID, err error) { var ( blocks []BlockReader bs []*Block @@ -435,7 +439,7 @@ func (c *LeveledCompactor) Compact(dest string, dirs []string, open []*Block) (u uid = ulid.MustNew(ulid.Now(), rand.Reader) meta := CompactBlockMetas(uid, metas...) - err = c.write(dest, meta, blocks...) + err = c.write(dest, meta, populateBlockFunc, blocks...) if err == nil { if meta.Stats.NumSamples == 0 { for _, b := range bs { @@ -501,7 +505,7 @@ func (c *LeveledCompactor) Write(dest string, b BlockReader, mint, maxt int64, p } } - err := c.write(dest, meta, b) + err := c.write(dest, meta, DefaultPopulateBlockFunc{}, b) if err != nil { return uid, err } @@ -546,7 +550,7 @@ func (w *instrumentedChunkWriter) WriteChunks(chunks ...chunks.Meta) error { } // write creates a new block that is the union of the provided blocks into dir. -func (c *LeveledCompactor) write(dest string, meta *BlockMeta, blocks ...BlockReader) (err error) { +func (c *LeveledCompactor) write(dest string, meta *BlockMeta, populateBlockFunc PopulateBlockFunc, blocks ...BlockReader) (err error) { dir := filepath.Join(dest, meta.ULID.String()) tmp := dir + tmpForCreationBlockDirSuffix var closers []io.Closer @@ -557,8 +561,8 @@ func (c *LeveledCompactor) write(dest string, meta *BlockMeta, blocks ...BlockRe if err := os.RemoveAll(tmp); err != nil { level.Error(c.logger).Log("msg", "removed tmp folder after failed compaction", "err", err.Error()) } - c.metrics.ran.Inc() - c.metrics.duration.Observe(time.Since(t).Seconds()) + c.metrics.Ran.Inc() + c.metrics.Duration.Observe(time.Since(t).Seconds()) }(time.Now()) if err = os.RemoveAll(tmp); err != nil { @@ -582,9 +586,9 @@ func (c *LeveledCompactor) write(dest string, meta *BlockMeta, blocks ...BlockRe if meta.Compaction.Level == 1 { chunkw = &instrumentedChunkWriter{ ChunkWriter: chunkw, - size: c.metrics.chunkSize, - samples: c.metrics.chunkSamples, - trange: c.metrics.chunkRange, + size: c.metrics.ChunkSize, + samples: c.metrics.ChunkSamples, + trange: c.metrics.ChunkRange, } } @@ -594,7 +598,7 @@ func (c *LeveledCompactor) write(dest string, meta *BlockMeta, blocks ...BlockRe } closers = append(closers, indexw) - if err := c.populateBlock(blocks, meta, indexw, chunkw); err != nil { + if err := populateBlockFunc.PopulateBlock(c.ctx, c.metrics, c.logger, c.chunkPool, c.mergeFunc, blocks, meta, indexw, chunkw); err != nil { return errors.Wrap(err, "populate block") } @@ -659,10 +663,16 @@ func (c *LeveledCompactor) write(dest string, meta *BlockMeta, blocks ...BlockRe return nil } -// populateBlock fills the index and chunk writers with new data gathered as the union +type PopulateBlockFunc interface { + PopulateBlock(ctx context.Context, metrics *CompactorMetrics, logger log.Logger, chunkPool chunkenc.Pool, mergeFunc storage.VerticalChunkSeriesMergeFunc, blocks []BlockReader, meta *BlockMeta, indexw IndexWriter, chunkw ChunkWriter) error +} + +type DefaultPopulateBlockFunc struct{} + +// PopulateBlock fills the index and chunk writers with new data gathered as the union // of the provided blocks. It returns meta information for the new block. // It expects sorted blocks input by mint. -func (c *LeveledCompactor) populateBlock(blocks []BlockReader, meta *BlockMeta, indexw IndexWriter, chunkw ChunkWriter) (err error) { +func (c DefaultPopulateBlockFunc) PopulateBlock(ctx context.Context, metrics *CompactorMetrics, logger log.Logger, chunkPool chunkenc.Pool, mergeFunc storage.VerticalChunkSeriesMergeFunc, blocks []BlockReader, meta *BlockMeta, indexw IndexWriter, chunkw ChunkWriter) (err error) { if len(blocks) == 0 { return errors.New("cannot populate block from no readers") } @@ -679,23 +689,23 @@ func (c *LeveledCompactor) populateBlock(blocks []BlockReader, meta *BlockMeta, errs.Add(errors.Wrap(cerr, "close")) } err = errs.Err() - c.metrics.populatingBlocks.Set(0) + metrics.PopulatingBlocks.Set(0) }() - c.metrics.populatingBlocks.Set(1) + metrics.PopulatingBlocks.Set(1) globalMaxt := blocks[0].Meta().MaxTime for i, b := range blocks { select { - case <-c.ctx.Done(): - return c.ctx.Err() + case <-ctx.Done(): + return ctx.Err() default: } if !overlapping { if i > 0 && b.Meta().MinTime < globalMaxt { - c.metrics.overlappingBlocks.Inc() + metrics.OverlappingBlocks.Inc() overlapping = true - level.Info(c.logger).Log("msg", "Found overlapping blocks during compaction", "ulid", meta.ULID) + level.Info(logger).Log("msg", "Found overlapping blocks during compaction", "ulid", meta.ULID) } if b.Meta().MaxTime > globalMaxt { globalMaxt = b.Meta().MaxTime @@ -727,7 +737,7 @@ func (c *LeveledCompactor) populateBlock(blocks []BlockReader, meta *BlockMeta, } all = indexr.SortedPostings(all) // Blocks meta is half open: [min, max), so subtract 1 to ensure we don't hold samples with exact meta.MaxTime timestamp. - sets = append(sets, newBlockChunkSeriesSet(b.Meta().ULID, indexr, chunkr, tombsr, all, meta.MinTime, meta.MaxTime-1, false)) + sets = append(sets, NewBlockChunkSeriesSet(b.Meta().ULID, indexr, chunkr, tombsr, all, meta.MinTime, meta.MaxTime-1, false)) syms := indexr.Symbols() if i == 0 { symbols = syms @@ -755,14 +765,14 @@ func (c *LeveledCompactor) populateBlock(blocks []BlockReader, meta *BlockMeta, if len(sets) > 1 { // Merge series using specified chunk series merger. // The default one is the compacting series merger. - set = storage.NewMergeChunkSeriesSet(sets, c.mergeFunc) + set = storage.NewMergeChunkSeriesSet(sets, mergeFunc) } // Iterate over all sorted chunk series. for set.Next() { select { - case <-c.ctx.Done(): - return c.ctx.Err() + case <-ctx.Done(): + return ctx.Err() default: } s := set.At() @@ -797,7 +807,7 @@ func (c *LeveledCompactor) populateBlock(blocks []BlockReader, meta *BlockMeta, } for _, chk := range chks { - if err := c.chunkPool.Put(chk.Chunk); err != nil { + if err := chunkPool.Put(chk.Chunk); err != nil { return errors.Wrap(err, "put chunk") } } diff --git a/tsdb/compact_test.go b/tsdb/compact_test.go index b58755e63e..0a01892d55 100644 --- a/tsdb/compact_test.go +++ b/tsdb/compact_test.go @@ -441,7 +441,7 @@ func TestCompactionFailWillCleanUpTempDir(t *testing.T) { tmpdir := t.TempDir() - require.Error(t, compactor.write(tmpdir, &BlockMeta{}, erringBReader{})) + require.Error(t, compactor.write(tmpdir, &BlockMeta{}, DefaultPopulateBlockFunc{}, erringBReader{})) _, err = os.Stat(filepath.Join(tmpdir, BlockMeta{}.ULID.String()) + tmpForCreationBlockDirSuffix) require.True(t, os.IsNotExist(err), "directory is not cleaned up") } @@ -953,7 +953,8 @@ func TestCompaction_populateBlock(t *testing.T) { } iw := &mockIndexWriter{} - err = c.populateBlock(blocks, meta, iw, nopChunkWriter{}) + populateBlockFunc := DefaultPopulateBlockFunc{} + err = populateBlockFunc.PopulateBlock(c.ctx, c.metrics, c.logger, c.chunkPool, c.mergeFunc, blocks, meta, iw, nopChunkWriter{}) if tc.expErr != nil { require.Error(t, err) require.Equal(t, tc.expErr.Error(), err.Error()) @@ -1181,14 +1182,14 @@ func TestCancelCompactions(t *testing.T) { db, err := open(tmpdir, log.NewNopLogger(), nil, DefaultOptions(), []int64{1, 2000}, nil) require.NoError(t, err) require.Equal(t, 3, len(db.Blocks()), "initial block count mismatch") - require.Equal(t, 0.0, prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.ran), "initial compaction counter mismatch") + require.Equal(t, 0.0, prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.Ran), "initial compaction counter mismatch") db.compactc <- struct{}{} // Trigger a compaction. - for prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.populatingBlocks) <= 0 { + for prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.PopulatingBlocks) <= 0 { time.Sleep(3 * time.Millisecond) } start := time.Now() - for prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.ran) != 1 { + for prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.Ran) != 1 { time.Sleep(3 * time.Millisecond) } timeCompactionUninterrupted = time.Since(start) @@ -1200,10 +1201,10 @@ func TestCancelCompactions(t *testing.T) { db, err := open(tmpdirCopy, log.NewNopLogger(), nil, DefaultOptions(), []int64{1, 2000}, nil) require.NoError(t, err) require.Equal(t, 3, len(db.Blocks()), "initial block count mismatch") - require.Equal(t, 0.0, prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.ran), "initial compaction counter mismatch") + require.Equal(t, 0.0, prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.Ran), "initial compaction counter mismatch") db.compactc <- struct{}{} // Trigger a compaction. - for prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.populatingBlocks) <= 0 { + for prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.PopulatingBlocks) <= 0 { time.Sleep(3 * time.Millisecond) } @@ -1284,7 +1285,7 @@ func TestDeleteCompactionBlockAfterFailedReload(t *testing.T) { require.NoError(t, os.RemoveAll(lastBlockIndex)) // Corrupt the block by removing the index file. require.Equal(t, 0.0, prom_testutil.ToFloat64(db.metrics.reloadsFailed), "initial 'failed db reloadBlocks' count metrics mismatch") - require.Equal(t, 0.0, prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.ran), "initial `compactions` count metric mismatch") + require.Equal(t, 0.0, prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.Ran), "initial `compactions` count metric mismatch") require.Equal(t, 0.0, prom_testutil.ToFloat64(db.metrics.compactionsFailed), "initial `compactions failed` count metric mismatch") // Do the compaction and check the metrics. @@ -1292,7 +1293,7 @@ func TestDeleteCompactionBlockAfterFailedReload(t *testing.T) { // the new block created from the compaction should be deleted. require.Error(t, db.Compact()) require.Equal(t, 1.0, prom_testutil.ToFloat64(db.metrics.reloadsFailed), "'failed db reloadBlocks' count metrics mismatch") - require.Equal(t, 1.0, prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.ran), "`compaction` count metric mismatch") + require.Equal(t, 1.0, prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.Ran), "`compaction` count metric mismatch") require.Equal(t, 1.0, prom_testutil.ToFloat64(db.metrics.compactionsFailed), "`compactions failed` count metric mismatch") actBlocks, err = blockDirs(db.Dir()) diff --git a/tsdb/db_test.go b/tsdb/db_test.go index 33a47ea9ed..edcfe5f9ce 100644 --- a/tsdb/db_test.go +++ b/tsdb/db_test.go @@ -2055,7 +2055,7 @@ func TestNoEmptyBlocks(t *testing.T) { require.NoError(t, err) require.Equal(t, len(db.Blocks()), len(actBlocks)) require.Equal(t, 0, len(actBlocks)) - require.Equal(t, 0, int(prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.ran)), "no compaction should be triggered here") + require.Equal(t, 0, int(prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.Ran)), "no compaction should be triggered here") }) t.Run("Test no blocks after deleting all samples from head.", func(t *testing.T) { @@ -2069,7 +2069,7 @@ func TestNoEmptyBlocks(t *testing.T) { require.NoError(t, app.Commit()) require.NoError(t, db.Delete(math.MinInt64, math.MaxInt64, defaultMatcher)) require.NoError(t, db.Compact()) - require.Equal(t, 1, int(prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.ran)), "compaction should have been triggered here") + require.Equal(t, 1, int(prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.Ran)), "compaction should have been triggered here") actBlocks, err := blockDirs(db.Dir()) require.NoError(t, err) @@ -2091,7 +2091,7 @@ func TestNoEmptyBlocks(t *testing.T) { require.NoError(t, app.Commit()) require.NoError(t, db.Compact()) - require.Equal(t, 2, int(prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.ran)), "compaction should have been triggered here") + require.Equal(t, 2, int(prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.Ran)), "compaction should have been triggered here") actBlocks, err = blockDirs(db.Dir()) require.NoError(t, err) require.Equal(t, len(db.Blocks()), len(actBlocks)) @@ -2112,7 +2112,7 @@ func TestNoEmptyBlocks(t *testing.T) { require.NoError(t, app.Commit()) require.NoError(t, db.head.Delete(math.MinInt64, math.MaxInt64, defaultMatcher)) require.NoError(t, db.Compact()) - require.Equal(t, 3, int(prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.ran)), "compaction should have been triggered here") + require.Equal(t, 3, int(prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.Ran)), "compaction should have been triggered here") require.Equal(t, oldBlocks, db.Blocks()) }) @@ -2131,7 +2131,7 @@ func TestNoEmptyBlocks(t *testing.T) { require.Equal(t, len(blocks)+len(oldBlocks), len(db.Blocks())) // Ensure all blocks are registered. require.NoError(t, db.Delete(math.MinInt64, math.MaxInt64, defaultMatcher)) require.NoError(t, db.Compact()) - require.Equal(t, 5, int(prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.ran)), "compaction should have been triggered here once for each block that have tombstones") + require.Equal(t, 5, int(prom_testutil.ToFloat64(db.compactor.(*LeveledCompactor).metrics.Ran)), "compaction should have been triggered here once for each block that have tombstones") actBlocks, err := blockDirs(db.Dir()) require.NoError(t, err) diff --git a/tsdb/querier.go b/tsdb/querier.go index 74c73f460d..4b3144c71d 100644 --- a/tsdb/querier.go +++ b/tsdb/querier.go @@ -180,7 +180,7 @@ func (q *blockChunkQuerier) Select(sortSeries bool, hints *storage.SelectHints, if sortSeries { p = q.index.SortedPostings(p) } - return newBlockChunkSeriesSet(q.blockID, q.index, q.chunks, q.tombstones, p, mint, maxt, disableTrimming) + return NewBlockChunkSeriesSet(q.blockID, q.index, q.chunks, q.tombstones, p, mint, maxt, disableTrimming) } func findSetMatches(pattern string) []string { @@ -438,7 +438,7 @@ func (s *seriesData) Labels() labels.Labels { return s.labels } // blockBaseSeriesSet allows to iterate over all series in the single block. // Iterated series are trimmed with given min and max time as well as tombstones. -// See newBlockSeriesSet and newBlockChunkSeriesSet to use it for either sample or chunk iterating. +// See newBlockSeriesSet and NewBlockChunkSeriesSet to use it for either sample or chunk iterating. type blockBaseSeriesSet struct { blockID ulid.ULID p index.Postings @@ -924,7 +924,7 @@ type blockChunkSeriesSet struct { blockBaseSeriesSet } -func newBlockChunkSeriesSet(id ulid.ULID, i IndexReader, c ChunkReader, t tombstones.Reader, p index.Postings, mint, maxt int64, disableTrimming bool) storage.ChunkSeriesSet { +func NewBlockChunkSeriesSet(id ulid.ULID, i IndexReader, c ChunkReader, t tombstones.Reader, p index.Postings, mint, maxt int64, disableTrimming bool) storage.ChunkSeriesSet { return &blockChunkSeriesSet{ blockBaseSeriesSet{ blockID: id, From 6cecb8794164200c2010483b8f22676994bcbe16 Mon Sep 17 00:00:00 2001 From: Soon-Ping Date: Tue, 4 Apr 2023 11:21:13 -0700 Subject: [PATCH 069/231] Generalized rule group iteration evaluation hook (#11885) Signed-off-by: Soon-Ping Phang --- rules/manager.go | 149 ++++++++++++++++++++++++------------------ rules/manager_test.go | 58 +++++++++++----- 2 files changed, 126 insertions(+), 81 deletions(-) diff --git a/rules/manager.go b/rules/manager.go index f8dcf081fe..108b6a77a1 100644 --- a/rules/manager.go +++ b/rules/manager.go @@ -253,7 +253,8 @@ type Group struct { opts *ManagerOptions mtx sync.Mutex evaluationTime time.Duration - lastEvaluation time.Time + lastEvaluation time.Time // Wall-clock time of most recent evaluation. + lastEvalTimestamp time.Time // Time slot used for most recent evaluation. shouldRestore bool @@ -266,22 +267,27 @@ type Group struct { metrics *Metrics - ruleGroupPostProcessFunc RuleGroupPostProcessFunc + // Rule group evaluation iteration function, + // defaults to DefaultEvalIterationFunc. + evalIterationFunc GroupEvalIterationFunc } -// This function will be used before each rule group evaluation if not nil. -// Use this function type if the rule group post processing is needed. -type RuleGroupPostProcessFunc func(g *Group, lastEvalTimestamp time.Time, log log.Logger) error +// GroupEvalIterationFunc is used to implement and extend rule group +// evaluation iteration logic. It is configured in Group.evalIterationFunc, +// and periodically invoked at each group evaluation interval to +// evaluate the rules in the group at that point in time. +// DefaultEvalIterationFunc is the default implementation. +type GroupEvalIterationFunc func(ctx context.Context, g *Group, evalTimestamp time.Time) type GroupOptions struct { - Name, File string - Interval time.Duration - Limit int - Rules []Rule - ShouldRestore bool - Opts *ManagerOptions - done chan struct{} - RuleGroupPostProcessFunc RuleGroupPostProcessFunc + Name, File string + Interval time.Duration + Limit int + Rules []Rule + ShouldRestore bool + Opts *ManagerOptions + done chan struct{} + EvalIterationFunc GroupEvalIterationFunc } // NewGroup makes a new Group with the given name, options, and rules. @@ -302,21 +308,26 @@ func NewGroup(o GroupOptions) *Group { metrics.GroupSamples.WithLabelValues(key) metrics.GroupInterval.WithLabelValues(key).Set(o.Interval.Seconds()) + evalIterationFunc := o.EvalIterationFunc + if evalIterationFunc == nil { + evalIterationFunc = DefaultEvalIterationFunc + } + return &Group{ - name: o.Name, - file: o.File, - interval: o.Interval, - limit: o.Limit, - rules: o.Rules, - shouldRestore: o.ShouldRestore, - opts: o.Opts, - seriesInPreviousEval: make([]map[string]labels.Labels, len(o.Rules)), - done: make(chan struct{}), - managerDone: o.done, - terminated: make(chan struct{}), - logger: log.With(o.Opts.Logger, "file", o.File, "group", o.Name), - metrics: metrics, - ruleGroupPostProcessFunc: o.RuleGroupPostProcessFunc, + name: o.Name, + file: o.File, + interval: o.Interval, + limit: o.Limit, + rules: o.Rules, + shouldRestore: o.ShouldRestore, + opts: o.Opts, + seriesInPreviousEval: make([]map[string]labels.Labels, len(o.Rules)), + done: make(chan struct{}), + managerDone: o.done, + terminated: make(chan struct{}), + logger: log.With(o.Opts.Logger, "file", o.File, "group", o.Name), + metrics: metrics, + evalIterationFunc: evalIterationFunc, } } @@ -341,6 +352,8 @@ func (g *Group) Interval() time.Duration { return g.interval } // Limit returns the group's limit. func (g *Group) Limit() int { return g.limit } +func (g *Group) Logger() log.Logger { return g.logger } + func (g *Group) run(ctx context.Context) { defer close(g.terminated) @@ -359,18 +372,6 @@ func (g *Group) run(ctx context.Context) { }, }) - iter := func() { - g.metrics.IterationsScheduled.WithLabelValues(GroupKey(g.file, g.name)).Inc() - - start := time.Now() - g.Eval(ctx, evalTimestamp) - timeSinceStart := time.Since(start) - - g.metrics.IterationDuration.Observe(timeSinceStart.Seconds()) - g.setEvaluationTime(timeSinceStart) - g.setLastEvaluation(start) - } - // The assumption here is that since the ticker was started after having // waited for `evalTimestamp` to pass, the ticks will trigger soon // after each `evalTimestamp + N * g.interval` occurrence. @@ -400,7 +401,7 @@ func (g *Group) run(ctx context.Context) { }(time.Now()) }() - iter() + g.evalIterationFunc(ctx, g, evalTimestamp) if g.shouldRestore { // If we have to restore, we wait for another Eval to finish. // The reason behind this is, during first eval (or before it) @@ -416,7 +417,7 @@ func (g *Group) run(ctx context.Context) { g.metrics.IterationsScheduled.WithLabelValues(GroupKey(g.file, g.name)).Add(float64(missed)) } evalTimestamp = evalTimestamp.Add((missed + 1) * g.interval) - iter() + g.evalIterationFunc(ctx, g, evalTimestamp) } g.RestoreForState(time.Now()) @@ -439,21 +440,29 @@ func (g *Group) run(ctx context.Context) { } evalTimestamp = evalTimestamp.Add((missed + 1) * g.interval) - useRuleGroupPostProcessFunc(g, evalTimestamp.Add(-(missed+1)*g.interval)) - - iter() + g.evalIterationFunc(ctx, g, evalTimestamp) } } } } -func useRuleGroupPostProcessFunc(g *Group, lastEvalTimestamp time.Time) { - if g.ruleGroupPostProcessFunc != nil { - err := g.ruleGroupPostProcessFunc(g, lastEvalTimestamp, g.logger) - if err != nil { - level.Warn(g.logger).Log("msg", "ruleGroupPostProcessFunc failed", "err", err) - } - } +// DefaultEvalIterationFunc is the default implementation of +// GroupEvalIterationFunc that is periodically invoked to evaluate the rules +// in a group at a given point in time and updates Group state and metrics +// accordingly. Custom GroupEvalIterationFunc implementations are recommended +// to invoke this function as well, to ensure correct Group state and metrics +// are maintained. +func DefaultEvalIterationFunc(ctx context.Context, g *Group, evalTimestamp time.Time) { + g.metrics.IterationsScheduled.WithLabelValues(GroupKey(g.file, g.name)).Inc() + + start := time.Now() + g.Eval(ctx, evalTimestamp) + timeSinceStart := time.Since(start) + + g.metrics.IterationDuration.Observe(timeSinceStart.Seconds()) + g.setEvaluationTime(timeSinceStart) + g.setLastEvaluation(start) + g.setLastEvalTimestamp(evalTimestamp) } func (g *Group) stop() { @@ -533,6 +542,20 @@ func (g *Group) setLastEvaluation(ts time.Time) { g.lastEvaluation = ts } +// GetLastEvalTimestamp returns the timestamp of the last evaluation. +func (g *Group) GetLastEvalTimestamp() time.Time { + g.mtx.Lock() + defer g.mtx.Unlock() + return g.lastEvalTimestamp +} + +// setLastEvalTimestamp updates lastEvalTimestamp to the timestamp of the last evaluation. +func (g *Group) setLastEvalTimestamp(ts time.Time) { + g.mtx.Lock() + defer g.mtx.Unlock() + g.lastEvalTimestamp = ts +} + // EvalTimestamp returns the immediately preceding consistently slotted evaluation time. func (g *Group) EvalTimestamp(startTime int64) time.Time { var ( @@ -996,11 +1019,11 @@ func (m *Manager) Stop() { // Update the rule manager's state as the config requires. If // loading the new rules failed the old rule set is restored. -func (m *Manager) Update(interval time.Duration, files []string, externalLabels labels.Labels, externalURL string, ruleGroupPostProcessFunc RuleGroupPostProcessFunc) error { +func (m *Manager) Update(interval time.Duration, files []string, externalLabels labels.Labels, externalURL string, groupEvalIterationFunc GroupEvalIterationFunc) error { m.mtx.Lock() defer m.mtx.Unlock() - groups, errs := m.LoadGroups(interval, externalLabels, externalURL, ruleGroupPostProcessFunc, files...) + groups, errs := m.LoadGroups(interval, externalLabels, externalURL, groupEvalIterationFunc, files...) if errs != nil { for _, e := range errs { @@ -1085,7 +1108,7 @@ func (FileLoader) Parse(query string) (parser.Expr, error) { return parser.Parse // LoadGroups reads groups from a list of files. func (m *Manager) LoadGroups( - interval time.Duration, externalLabels labels.Labels, externalURL string, ruleGroupPostProcessFunc RuleGroupPostProcessFunc, filenames ...string, + interval time.Duration, externalLabels labels.Labels, externalURL string, groupEvalIterationFunc GroupEvalIterationFunc, filenames ...string, ) (map[string]*Group, []error) { groups := make(map[string]*Group) @@ -1133,15 +1156,15 @@ func (m *Manager) LoadGroups( } groups[GroupKey(fn, rg.Name)] = NewGroup(GroupOptions{ - Name: rg.Name, - File: fn, - Interval: itv, - Limit: rg.Limit, - Rules: rules, - ShouldRestore: shouldRestore, - Opts: m.opts, - done: m.done, - RuleGroupPostProcessFunc: ruleGroupPostProcessFunc, + Name: rg.Name, + File: fn, + Interval: itv, + Limit: rg.Limit, + Rules: rules, + ShouldRestore: shouldRestore, + Opts: m.opts, + done: m.done, + EvalIterationFunc: groupEvalIterationFunc, }) } } diff --git a/rules/manager_test.go b/rules/manager_test.go index aed289c5b9..1faf730be3 100644 --- a/rules/manager_test.go +++ b/rules/manager_test.go @@ -1237,7 +1237,7 @@ func TestRuleHealthUpdates(t *testing.T) { require.Equal(t, HealthBad, rules.Health()) } -func TestUpdateMissedEvalMetrics(t *testing.T) { +func TestRuleGroupEvalIterationFunc(t *testing.T) { suite, err := promql.NewTest(t, ` load 5m http_requests{instance="0"} 75 85 50 0 0 25 0 0 40 0 120 @@ -1254,26 +1254,40 @@ func TestUpdateMissedEvalMetrics(t *testing.T) { testValue := 1 - overrideFunc := func(g *Group, lastEvalTimestamp time.Time, log log.Logger) error { + evalIterationFunc := func(ctx context.Context, g *Group, evalTimestamp time.Time) { testValue = 2 - return nil + DefaultEvalIterationFunc(ctx, g, evalTimestamp) + testValue = 3 + } + + skipEvalIterationFunc := func(ctx context.Context, g *Group, evalTimestamp time.Time) { + testValue = 4 } type testInput struct { - overrideFunc func(g *Group, lastEvalTimestamp time.Time, logger log.Logger) error - expectedValue int + evalIterationFunc GroupEvalIterationFunc + expectedValue int + lastEvalTimestampIsZero bool } tests := []testInput{ - // testValue should still have value of 1 since overrideFunc is nil. + // testValue should still have value of 1 since the default iteration function will be called. { - overrideFunc: nil, - expectedValue: 1, + evalIterationFunc: nil, + expectedValue: 1, + lastEvalTimestampIsZero: false, }, - // testValue should be incremented to 2 since overrideFunc is called. + // testValue should be incremented to 3 since evalIterationFunc is called. { - overrideFunc: overrideFunc, - expectedValue: 2, + evalIterationFunc: evalIterationFunc, + expectedValue: 3, + lastEvalTimestampIsZero: false, + }, + // testValue should be incremented to 4 since skipEvalIterationFunc is called. + { + evalIterationFunc: skipEvalIterationFunc, + expectedValue: 4, + lastEvalTimestampIsZero: true, }, } @@ -1315,12 +1329,12 @@ func TestUpdateMissedEvalMetrics(t *testing.T) { } group := NewGroup(GroupOptions{ - Name: "default", - Interval: time.Second, - Rules: []Rule{rule}, - ShouldRestore: true, - Opts: opts, - RuleGroupPostProcessFunc: tst.overrideFunc, + Name: "default", + Interval: time.Second, + Rules: []Rule{rule}, + ShouldRestore: true, + Opts: opts, + EvalIterationFunc: tst.evalIterationFunc, }) go func() { @@ -1329,10 +1343,18 @@ func TestUpdateMissedEvalMetrics(t *testing.T) { time.Sleep(3 * time.Second) group.stop() + require.Equal(t, tst.expectedValue, testValue) + if tst.lastEvalTimestampIsZero { + require.Zero(t, group.GetLastEvalTimestamp()) + } else { + oneMinute, _ := time.ParseDuration("1m") + require.WithinDuration(t, time.Now(), group.GetLastEvalTimestamp(), oneMinute) + } } - for _, tst := range tests { + for i, tst := range tests { + t.Logf("case %d", i) testFunc(tst) } } From 391473141d86f3a923a45421e5fc0f2ec4d541dc Mon Sep 17 00:00:00 2001 From: Julien Pivotto Date: Wed, 5 Apr 2023 09:45:39 +0200 Subject: [PATCH 070/231] Check health & ready: move to flags (#12223) This makes it more consistent with other command like import rules. We don't have stricts rules and uniformity accross promtool unfortunately, but I think it's better to only have the http config on relevant check commands to avoid thinking Prometheus can e.g. check the config over the wire. Signed-off-by: Julien Pivotto --- cmd/promtool/main.go | 23 ++++++++--------------- docs/command-line/promtool.md | 19 ++++++++++--------- 2 files changed, 18 insertions(+), 24 deletions(-) diff --git a/cmd/promtool/main.go b/cmd/promtool/main.go index 064e7a04f7..de002a0b28 100644 --- a/cmd/promtool/main.go +++ b/cmd/promtool/main.go @@ -89,7 +89,6 @@ func main() { app.HelpFlag.Short('h') checkCmd := app.Command("check", "Check the resources for validity.") - checkCmd.Flag("http.config.file", "HTTP client configuration file for promtool to connect to Prometheus.").PlaceHolder("").ExistingFileVar(&httpConfigFilePath) sdCheckCmd := checkCmd.Command("service-discovery", "Perform service discovery for the given job name and report the results, including relabeling.") sdConfigFile := sdCheckCmd.Arg("config-file", "The prometheus config file.").Required().ExistingFile() @@ -117,16 +116,12 @@ func main() { ).Required().ExistingFiles() checkServerHealthCmd := checkCmd.Command("healthy", "Check if the Prometheus server is healthy.") - serverHealthURLArg := checkServerHealthCmd.Arg( - "server", - "The URL of the Prometheus server to check (e.g. http://localhost:9090)", - ).URL() + checkServerHealthCmd.Flag("http.config.file", "HTTP client configuration file for promtool to connect to Prometheus.").PlaceHolder("").ExistingFileVar(&httpConfigFilePath) + checkServerHealthCmd.Flag("url", "The URL for the Prometheus server.").Default("http://localhost:9090").URLVar(&serverURL) checkServerReadyCmd := checkCmd.Command("ready", "Check if the Prometheus server is ready.") - serverReadyURLArg := checkServerReadyCmd.Arg( - "server", - "The URL of the Prometheus server to check (e.g. http://localhost:9090)", - ).URL() + checkServerReadyCmd.Flag("http.config.file", "HTTP client configuration file for promtool to connect to Prometheus.").PlaceHolder("").ExistingFileVar(&httpConfigFilePath) + checkServerReadyCmd.Flag("url", "The URL for the Prometheus server.").Default("http://localhost:9090").URLVar(&serverURL) checkRulesCmd := checkCmd.Command("rules", "Check if the rule files are valid or not.") ruleFiles := checkRulesCmd.Arg( @@ -292,10 +287,10 @@ func main() { os.Exit(CheckConfig(*agentMode, *checkConfigSyntaxOnly, newLintConfig(*checkConfigLint, *checkConfigLintFatal), *configFiles...)) case checkServerHealthCmd.FullCommand(): - os.Exit(checkErr(CheckServerStatus(*serverHealthURLArg, checkHealth, httpRoundTripper))) + os.Exit(checkErr(CheckServerStatus(serverURL, checkHealth, httpRoundTripper))) case checkServerReadyCmd.FullCommand(): - os.Exit(checkErr(CheckServerStatus(*serverReadyURLArg, checkReadiness, httpRoundTripper))) + os.Exit(checkErr(CheckServerStatus(serverURL, checkReadiness, httpRoundTripper))) case checkWebConfigCmd.FullCommand(): os.Exit(CheckWebConfig(*webConfigFiles...)) @@ -390,12 +385,10 @@ func (ls lintConfig) lintDuplicateRules() bool { return ls.all || ls.duplicateRules } -const promDefaultURL = "http://localhost:9090" - // Check server status - healthy & ready. func CheckServerStatus(serverURL *url.URL, checkEndpoint string, roundTripper http.RoundTripper) error { - if serverURL == nil { - serverURL, _ = url.Parse(promDefaultURL) + if serverURL.Scheme == "" { + serverURL.Scheme = "http" } config := api.Config{ diff --git a/docs/command-line/promtool.md b/docs/command-line/promtool.md index 543034f157..e149d374a0 100644 --- a/docs/command-line/promtool.md +++ b/docs/command-line/promtool.md @@ -58,7 +58,6 @@ Check the resources for validity. | Flag | Description | | --- | --- | -| --http.config.file | HTTP client configuration file for promtool to connect to Prometheus. | | --extended | Print extended information related to the cardinality of the metrics. | @@ -137,11 +136,12 @@ Check if the Prometheus server is healthy. -###### Arguments +###### Flags -| Argument | Description | -| --- | --- | -| server | The URL of the Prometheus server to check (e.g. http://localhost:9090) | +| Flag | Description | Default | +| --- | --- | --- | +| --http.config.file | HTTP client configuration file for promtool to connect to Prometheus. | | +| --url | The URL for the Prometheus server. | `http://localhost:9090` | @@ -152,11 +152,12 @@ Check if the Prometheus server is ready. -###### Arguments +###### Flags -| Argument | Description | -| --- | --- | -| server | The URL of the Prometheus server to check (e.g. http://localhost:9090) | +| Flag | Description | Default | +| --- | --- | --- | +| --http.config.file | HTTP client configuration file for promtool to connect to Prometheus. | | +| --url | The URL for the Prometheus server. | `http://localhost:9090` | From 034eb2b3f2b31589ce1ebd6402e1fb6465b0afd4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fran=C3=A7ois=20Gouteroux?= Date: Tue, 4 Apr 2023 13:28:02 +0200 Subject: [PATCH 071/231] promtool: read from stdin if no filenames are provided in check rules MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: François Gouteroux --- cmd/promtool/main.go | 21 +++++++++++++++++---- docs/command-line/promtool.md | 4 ++-- 2 files changed, 19 insertions(+), 6 deletions(-) diff --git a/cmd/promtool/main.go b/cmd/promtool/main.go index 064e7a04f7..abac38d5fa 100644 --- a/cmd/promtool/main.go +++ b/cmd/promtool/main.go @@ -131,8 +131,8 @@ func main() { checkRulesCmd := checkCmd.Command("rules", "Check if the rule files are valid or not.") ruleFiles := checkRulesCmd.Arg( "rule-files", - "The rule files to check.", - ).Required().ExistingFiles() + "The rule files to check, default is read from standard input (STDIN).", + ).Default("-").Strings() checkRulesLint := checkRulesCmd.Flag( "lint", "Linting checks to apply. Available options are: "+strings.Join(lintOptions, ", ")+". Use --lint=none to disable linting", @@ -715,9 +715,22 @@ func CheckRules(ls lintConfig, files ...string) int { } func checkRules(filename string, lintSettings lintConfig) (int, []error) { - fmt.Println("Checking", filename) + var rgs *rulefmt.RuleGroups + var errs []error + if filename == "-" || filename == "" { + var buf bytes.Buffer + tee := io.TeeReader(os.Stdin, &buf) + if _, err := buf.ReadFrom(tee); err != nil { + errs = append(errs, err) + return failureExitCode, errs + } + fmt.Println("Checking stdin") + rgs, errs = rulefmt.Parse(buf.Bytes()) + } else { + fmt.Println("Checking", filename) + rgs, errs = rulefmt.ParseFile(filename) + } - rgs, errs := rulefmt.ParseFile(filename) if errs != nil { return successExitCode, errs } diff --git a/docs/command-line/promtool.md b/docs/command-line/promtool.md index 543034f157..6b67408c9f 100644 --- a/docs/command-line/promtool.md +++ b/docs/command-line/promtool.md @@ -179,9 +179,9 @@ Check if the rule files are valid or not. ###### Arguments -| Argument | Description | Required | +| Argument | Description | Default | | --- | --- | --- | -| rule-files | The rule files to check. | Yes | +| rule-files | The rule files to check, default is read from standard input (STDIN). | `-` | From 8472596fd0673b972ca8d1e957ff997fe26db401 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fran=C3=A7ois=20Gouteroux?= Date: Wed, 5 Apr 2023 11:24:49 +0200 Subject: [PATCH 072/231] fix: apply suggested changes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: François Gouteroux --- cmd/promtool/main.go | 18 ++++++++++++------ docs/command-line/promtool.md | 6 +++--- 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/cmd/promtool/main.go b/cmd/promtool/main.go index abac38d5fa..b3d743e441 100644 --- a/cmd/promtool/main.go +++ b/cmd/promtool/main.go @@ -132,7 +132,7 @@ func main() { ruleFiles := checkRulesCmd.Arg( "rule-files", "The rule files to check, default is read from standard input (STDIN).", - ).Default("-").Strings() + ).ExistingFiles() checkRulesLint := checkRulesCmd.Flag( "lint", "Linting checks to apply. Available options are: "+strings.Join(lintOptions, ", ")+". Use --lint=none to disable linting", @@ -690,6 +690,11 @@ func CheckRules(ls lintConfig, files ...string) int { failed := false hasErrors := false + // add empty string to avoid matching filename + if len(files) == 0 { + files = append(files, "") + } + for _, f := range files { if n, errs := checkRules(f, ls); errs != nil { fmt.Fprintln(os.Stderr, " FAILED:") @@ -717,15 +722,16 @@ func CheckRules(ls lintConfig, files ...string) int { func checkRules(filename string, lintSettings lintConfig) (int, []error) { var rgs *rulefmt.RuleGroups var errs []error - if filename == "-" || filename == "" { - var buf bytes.Buffer - tee := io.TeeReader(os.Stdin, &buf) - if _, err := buf.ReadFrom(tee); err != nil { + + // if filename is an empty string it is a stdin + if filename == "" { + data, err := io.ReadAll(os.Stdin) + if err != nil { errs = append(errs, err) return failureExitCode, errs } fmt.Println("Checking stdin") - rgs, errs = rulefmt.Parse(buf.Bytes()) + rgs, errs = rulefmt.Parse(data) } else { fmt.Println("Checking", filename) rgs, errs = rulefmt.ParseFile(filename) diff --git a/docs/command-line/promtool.md b/docs/command-line/promtool.md index 6b67408c9f..59c46dd79b 100644 --- a/docs/command-line/promtool.md +++ b/docs/command-line/promtool.md @@ -179,9 +179,9 @@ Check if the rule files are valid or not. ###### Arguments -| Argument | Description | Default | -| --- | --- | --- | -| rule-files | The rule files to check, default is read from standard input (STDIN). | `-` | +| Argument | Description | +| --- | --- | +| rule-files | The rule files to check, default is read from standard input (STDIN). | From 0c75f76193b71f4860e95e5b9dd108361a124380 Mon Sep 17 00:00:00 2001 From: znley Date: Thu, 6 Apr 2023 11:07:10 +0800 Subject: [PATCH 073/231] fix: RLIM_INFINITY type is uint64 on loong64 Signed-off-by: znley --- util/runtime/limits_default.go | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/util/runtime/limits_default.go b/util/runtime/limits_default.go index c3e0b4701a..d49b3bc1ed 100644 --- a/util/runtime/limits_default.go +++ b/util/runtime/limits_default.go @@ -18,16 +18,18 @@ package runtime import ( "fmt" + "math" "syscall" ) -// syscall.RLIM_INFINITY is a constant and its default type is int. -// It needs to be converted to an int64 variable to be compared with uint64 values. -// See https://golang.org/ref/spec#Conversions -var unlimited int64 = syscall.RLIM_INFINITY +// syscall.RLIM_INFINITY is a constant. +// Its type is int on most architectures but there are exceptions such as loong64. +// Uniform it to uint accorind to the standard. +// https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_resource.h.html +var unlimited uint64 = syscall.RLIM_INFINITY & math.MaxUint64 func limitToString(v uint64, unit string) string { - if v == uint64(unlimited) { + if v == unlimited { return "unlimited" } return fmt.Sprintf("%d%s", v, unit) From 79db04eb122f9ff9d56574f8139bb797d56a8d45 Mon Sep 17 00:00:00 2001 From: Justin Lei Date: Fri, 3 Mar 2023 13:05:13 -0800 Subject: [PATCH 074/231] Adjust samplesPerChunk from 120 to 220 Signed-off-by: Justin Lei --- tsdb/head_append.go | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/tsdb/head_append.go b/tsdb/head_append.go index 14e343f74d..967c743591 100644 --- a/tsdb/head_append.go +++ b/tsdb/head_append.go @@ -1324,10 +1324,7 @@ func (s *memSeries) appendFloatHistogram(t int64, fh *histogram.FloatHistogram, func (s *memSeries) appendPreprocessor( t int64, e chunkenc.Encoding, chunkDiskMapper *chunks.ChunkDiskMapper, chunkRange int64, ) (c *memChunk, sampleInOrder, chunkCreated bool) { - // Based on Gorilla white papers this offers near-optimal compression ratio - // so anything bigger that this has diminishing returns and increases - // the time range within which we have to decompress all samples. - const samplesPerChunk = 120 + const samplesPerChunk = 220 c = s.head() From c770ba804762d30c4cf9b63b092a3dfd9fed77c4 Mon Sep 17 00:00:00 2001 From: Justin Lei Date: Fri, 3 Mar 2023 13:10:24 -0800 Subject: [PATCH 075/231] Add comment linking to PR Signed-off-by: Justin Lei --- tsdb/head_append.go | 1 + 1 file changed, 1 insertion(+) diff --git a/tsdb/head_append.go b/tsdb/head_append.go index 967c743591..46180051ee 100644 --- a/tsdb/head_append.go +++ b/tsdb/head_append.go @@ -1324,6 +1324,7 @@ func (s *memSeries) appendFloatHistogram(t int64, fh *histogram.FloatHistogram, func (s *memSeries) appendPreprocessor( t int64, e chunkenc.Encoding, chunkDiskMapper *chunks.ChunkDiskMapper, chunkRange int64, ) (c *memChunk, sampleInOrder, chunkCreated bool) { + // The basis for this number can be found here: https://github.com/prometheus/prometheus/pull/12055 const samplesPerChunk = 220 c = s.head() From 73ff91d182c991daff13ee945bf861c3d48a58c0 Mon Sep 17 00:00:00 2001 From: Justin Lei Date: Fri, 3 Mar 2023 14:27:13 -0800 Subject: [PATCH 076/231] Test fixes Signed-off-by: Justin Lei --- storage/remote/read_handler_test.go | 133 ++++++++++++++-------------- tsdb/db_test.go | 12 +-- tsdb/head_test.go | 24 ++--- 3 files changed, 86 insertions(+), 83 deletions(-) diff --git a/storage/remote/read_handler_test.go b/storage/remote/read_handler_test.go index 261c28e215..0c186097af 100644 --- a/storage/remote/read_handler_test.go +++ b/storage/remote/read_handler_test.go @@ -202,15 +202,18 @@ func BenchmarkStreamReadEndpoint(b *testing.B) { } func TestStreamReadEndpoint(t *testing.T) { - // First with 120 samples. We expect 1 frame with 1 chunk. - // Second with 121 samples, We expect 1 frame with 2 chunks. - // Third with 241 samples. We expect 1 frame with 2 chunks, and 1 frame with 1 chunk for the same series due to bytes limit. + // Note: samplesPerChunk is set to 220, but that isn't cleanly divisible by the chunkRange of 24 hrs and 1 min + // resolution used in this test so tsdb.computeChunkEndTime will put 240 samples in each chunk. + // + // First with 239 samples; we expect 1 frame with 1 full chunk. + // Second with 241 samples; we expect 1 frame with 2 chunks. + // Third with 481 samples; we expect 1 frame with 2 chunks, and 1 frame with 1 chunk for the same series due to bytes limit. // Fourth with 120 histogram samples. We expect 1 frame with 1 chunk. suite, err := promql.NewTest(t, ` load 1m - test_metric1{foo="bar1",baz="qux"} 0+100x119 - test_metric1{foo="bar2",baz="qux"} 0+100x120 - test_metric1{foo="bar3",baz="qux"} 0+100x240 + test_metric1{foo="bar1",baz="qux"} 0+100x239 + test_metric1{foo="bar2",baz="qux"} 0+100x240 + test_metric1{foo="bar3",baz="qux"} 0+100x480 `) require.NoError(t, err) defer suite.Close() @@ -228,8 +231,8 @@ func TestStreamReadEndpoint(t *testing.T) { } }, 1e6, 1, - // Labelset has 57 bytes. Full chunk in test data has roughly 240 bytes. This allows us to have at max 2 chunks in this test. - 57+480, + // Labelset has 57 bytes. Full chunk in test data has roughly 440 bytes. This allows us to have at max 2 chunks in this test. + 57+880, ) // Encode the request. @@ -245,19 +248,19 @@ func TestStreamReadEndpoint(t *testing.T) { matcher4, err := labels.NewMatcher(labels.MatchEqual, "__name__", "test_histogram_metric1") require.NoError(t, err) - query1, err := ToQuery(0, 14400001, []*labels.Matcher{matcher1, matcher2}, &storage.SelectHints{ + query1, err := ToQuery(0, 32460001, []*labels.Matcher{matcher1, matcher2}, &storage.SelectHints{ Step: 1, Func: "avg", Start: 0, - End: 14400001, + End: 32460001, }) require.NoError(t, err) - query2, err := ToQuery(0, 14400001, []*labels.Matcher{matcher1, matcher3}, &storage.SelectHints{ + query2, err := ToQuery(0, 32460001, []*labels.Matcher{matcher1, matcher3}, &storage.SelectHints{ Step: 1, Func: "avg", Start: 0, - End: 14400001, + End: 32460001, }) require.NoError(t, err) @@ -316,8 +319,8 @@ func TestStreamReadEndpoint(t *testing.T) { Chunks: []prompb.Chunk{ { Type: prompb.Chunk_XOR, - MaxTimeMs: 7140000, - Data: []byte("\000x\000\000\000\000\000\000\000\000\000\340\324\003\302|\005\224\000\301\254}\351z2\320O\355\264n[\007\316\224\243md\371\320\375\032Pm\nS\235\016Q\255\006P\275\250\277\312\201Z\003(3\240R\207\332\005(\017\240\322\201\332=(\023\2402\203Z\007(w\2402\201Z\017(\023\265\227\364P\033@\245\007\364\nP\033C\245\002t\036P+@e\036\364\016Pk@e\002t:P;A\245\001\364\nS\373@\245\006t\006P+C\345\002\364\006Pk@\345\036t\nP\033A\245\003\364:P\033@\245\006t\016ZJ\377\\\205\313\210\327\270\017\345+F[\310\347E)\355\024\241\366\342}(v\215(N\203)\326\207(\336\203(V\332W\362\202t4\240m\005(\377AJ\006\320\322\202t\374\240\255\003(oA\312:\3202"), + MaxTimeMs: 14340000, + Data: []byte("\x00\xf0\x00\x00\x00\x00\x00\x00\x00\x00\x00\xe0\xd4\x03\xc2|\x05\x94\x00\xc1\xac}\xe9z2\xd0O\xed\xb4n[\aΔ\xa3md\xf9\xd0\xfd\x1aPm\nS\x9d\x0eQ\xad\x06P\xbd\xa8\xbfʁZ\x03(3\xa0R\x87\xda\x05(\x0f\xa0ҁ\xda=(\x13\xa02\x83Z\a(w\xa02\x81Z\x0f(\x13\xb5\x97\xf4P\x1b@\xa5\a\xf4\nP\x1bC\xa5\x02t\x1eP+@e\x1e\xf4\x0ePk@e\x02t:P;A\xa5\x01\xf4\nS\xfb@\xa5\x06t\x06P+C\xe5\x02\xf4\x06Pk@\xe5\x1et\nP\x1bA\xa5\x03\xf4:P\x1b@\xa5\x06t\x0eZJ\xff\\\x85ˈ\u05f8\x0f\xe5+F[\xc8\xe7E)\xed\x14\xa1\xf6\xe2}(v\x8d(N\x83)և(ރ(V\xdaW\xf2\x82t4\xa0m\x05(\xffAJ\x06\xd0҂t\xfc\xa0\xad\x03(oA\xca:\xd02\x82t4\xa0\xed\xb0\xbfҀ\xfa\x05(=\xa0R\x87:\x03(\x15\xa0\xf2\x81z\x03(\xf5\xa0r\x83:\x05(\r\xa1ҁ\xfa\r(\r\xa0R\x9f:\a(5\xa02\x81z\x1f(\x15\xa02\x83:\x05(\xfd\xa0R\x80\xfa\r(\x1d\xa1ҁ:\x03(5\xa0r\xd6g\xf4\xb9\x8b\x97!\xabq\x1f\xcaN\xe0GJ\x1bE)\xfd\x14\xa1\xb6\xf2}(N\x8f(V\x83)އ(փ(N\xdcW҃\xb44\xa0}\x05(\xfbAJ\x19\xd02\x82\xb4\xfc\xa0\xbd\x03(kA\xca9\xd0R\x81\xb44\xa0\xfd\xb4\xbfҀ\xda\x05(3\xa0r\x87Z\x03(\x17\xa0\xf2\x81Z\x03(\xf3\xa0R\x83\xda\x05(\x0f\xa1ҁ\xda\r(\x13\xa02\x9fZ\a(7\xa02\x81Z\x1f(\x13\xa0Ҁ\xda\x05(\xff\xa0R\x80\xda\r "), }, }, }, @@ -336,61 +339,61 @@ func TestStreamReadEndpoint(t *testing.T) { Chunks: []prompb.Chunk{ { Type: prompb.Chunk_XOR, - MaxTimeMs: 7140000, - Data: []byte("\000x\000\000\000\000\000\000\000\000\000\340\324\003\302|\005\224\000\301\254}\351z2\320O\355\264n[\007\316\224\243md\371\320\375\032Pm\nS\235\016Q\255\006P\275\250\277\312\201Z\003(3\240R\207\332\005(\017\240\322\201\332=(\023\2402\203Z\007(w\2402\201Z\017(\023\265\227\364P\033@\245\007\364\nP\033C\245\002t\036P+@e\036\364\016Pk@e\002t:P;A\245\001\364\nS\373@\245\006t\006P+C\345\002\364\006Pk@\345\036t\nP\033A\245\003\364:P\033@\245\006t\016ZJ\377\\\205\313\210\327\270\017\345+F[\310\347E)\355\024\241\366\342}(v\215(N\203)\326\207(\336\203(V\332W\362\202t4\240m\005(\377AJ\006\320\322\202t\374\240\255\003(oA\312:\3202"), - }, - { - Type: prompb.Chunk_XOR, - MinTimeMs: 7200000, - MaxTimeMs: 7200000, - Data: []byte("\000\001\200\364\356\006@\307p\000\000\000\000\000\000"), - }, - }, - }, - }, - }, - { - ChunkedSeries: []*prompb.ChunkedSeries{ - { - Labels: []prompb.Label{ - {Name: "__name__", Value: "test_metric1"}, - {Name: "b", Value: "c"}, - {Name: "baz", Value: "qux"}, - {Name: "d", Value: "e"}, - {Name: "foo", Value: "bar3"}, - }, - Chunks: []prompb.Chunk{ - { - Type: prompb.Chunk_XOR, - MaxTimeMs: 7140000, - Data: []byte("\000x\000\000\000\000\000\000\000\000\000\340\324\003\302|\005\224\000\301\254}\351z2\320O\355\264n[\007\316\224\243md\371\320\375\032Pm\nS\235\016Q\255\006P\275\250\277\312\201Z\003(3\240R\207\332\005(\017\240\322\201\332=(\023\2402\203Z\007(w\2402\201Z\017(\023\265\227\364P\033@\245\007\364\nP\033C\245\002t\036P+@e\036\364\016Pk@e\002t:P;A\245\001\364\nS\373@\245\006t\006P+C\345\002\364\006Pk@\345\036t\nP\033A\245\003\364:P\033@\245\006t\016ZJ\377\\\205\313\210\327\270\017\345+F[\310\347E)\355\024\241\366\342}(v\215(N\203)\326\207(\336\203(V\332W\362\202t4\240m\005(\377AJ\006\320\322\202t\374\240\255\003(oA\312:\3202"), - }, - { - Type: prompb.Chunk_XOR, - MinTimeMs: 7200000, MaxTimeMs: 14340000, - Data: []byte("\000x\200\364\356\006@\307p\000\000\000\000\000\340\324\003\340>\224\355\260\277\322\200\372\005(=\240R\207:\003(\025\240\362\201z\003(\365\240r\203:\005(\r\241\322\201\372\r(\r\240R\237:\007(5\2402\201z\037(\025\2402\203:\005(\375\240R\200\372\r(\035\241\322\201:\003(5\240r\326g\364\271\213\227!\253q\037\312N\340GJ\033E)\375\024\241\266\362}(N\217(V\203)\336\207(\326\203(N\334W\322\203\2644\240}\005(\373AJ\031\3202\202\264\374\240\275\003(kA\3129\320R\201\2644\240\375\264\277\322\200\332\005(3\240r\207Z\003(\027\240\362\201Z\003(\363\240R\203\332\005(\017\241\322\201\332\r(\023\2402\237Z\007(7\2402\201Z\037(\023\240\322\200\332\005(\377\240R\200\332\r "), + Data: []byte("\x00\xf0\x00\x00\x00\x00\x00\x00\x00\x00\x00\xe0\xd4\x03\xc2|\x05\x94\x00\xc1\xac}\xe9z2\xd0O\xed\xb4n[\aΔ\xa3md\xf9\xd0\xfd\x1aPm\nS\x9d\x0eQ\xad\x06P\xbd\xa8\xbfʁZ\x03(3\xa0R\x87\xda\x05(\x0f\xa0ҁ\xda=(\x13\xa02\x83Z\a(w\xa02\x81Z\x0f(\x13\xb5\x97\xf4P\x1b@\xa5\a\xf4\nP\x1bC\xa5\x02t\x1eP+@e\x1e\xf4\x0ePk@e\x02t:P;A\xa5\x01\xf4\nS\xfb@\xa5\x06t\x06P+C\xe5\x02\xf4\x06Pk@\xe5\x1et\nP\x1bA\xa5\x03\xf4:P\x1b@\xa5\x06t\x0eZJ\xff\\\x85ˈ\u05f8\x0f\xe5+F[\xc8\xe7E)\xed\x14\xa1\xf6\xe2}(v\x8d(N\x83)և(ރ(V\xdaW\xf2\x82t4\xa0m\x05(\xffAJ\x06\xd0҂t\xfc\xa0\xad\x03(oA\xca:\xd02\x82t4\xa0\xed\xb0\xbfҀ\xfa\x05(=\xa0R\x87:\x03(\x15\xa0\xf2\x81z\x03(\xf5\xa0r\x83:\x05(\r\xa1ҁ\xfa\r(\r\xa0R\x9f:\a(5\xa02\x81z\x1f(\x15\xa02\x83:\x05(\xfd\xa0R\x80\xfa\r(\x1d\xa1ҁ:\x03(5\xa0r\xd6g\xf4\xb9\x8b\x97!\xabq\x1f\xcaN\xe0GJ\x1bE)\xfd\x14\xa1\xb6\xf2}(N\x8f(V\x83)އ(փ(N\xdcW҃\xb44\xa0}\x05(\xfbAJ\x19\xd02\x82\xb4\xfc\xa0\xbd\x03(kA\xca9\xd0R\x81\xb44\xa0\xfd\xb4\xbfҀ\xda\x05(3\xa0r\x87Z\x03(\x17\xa0\xf2\x81Z\x03(\xf3\xa0R\x83\xda\x05(\x0f\xa1ҁ\xda\r(\x13\xa02\x9fZ\a(7\xa02\x81Z\x1f(\x13\xa0Ҁ\xda\x05(\xff\xa0R\x80\xda\r "), }, - }, - }, - }, - }, - { - ChunkedSeries: []*prompb.ChunkedSeries{ - { - Labels: []prompb.Label{ - {Name: "__name__", Value: "test_metric1"}, - {Name: "b", Value: "c"}, - {Name: "baz", Value: "qux"}, - {Name: "d", Value: "e"}, - {Name: "foo", Value: "bar3"}, - }, - Chunks: []prompb.Chunk{ { Type: prompb.Chunk_XOR, MinTimeMs: 14400000, MaxTimeMs: 14400000, - Data: []byte("\000\001\200\350\335\r@\327p\000\000\000\000\000\000"), + Data: []byte("\x00\x01\x80\xe8\xdd\r@\xd7p\x00\x00\x00\x00\x00\x00"), + }, + }, + }, + }, + }, + { + ChunkedSeries: []*prompb.ChunkedSeries{ + { + Labels: []prompb.Label{ + {Name: "__name__", Value: "test_metric1"}, + {Name: "b", Value: "c"}, + {Name: "baz", Value: "qux"}, + {Name: "d", Value: "e"}, + {Name: "foo", Value: "bar3"}, + }, + Chunks: []prompb.Chunk{ + { + Type: prompb.Chunk_XOR, + MaxTimeMs: 14340000, + Data: []byte("\x00\xf0\x00\x00\x00\x00\x00\x00\x00\x00\x00\xe0\xd4\x03\xc2|\x05\x94\x00\xc1\xac}\xe9z2\xd0O\xed\xb4n[\aΔ\xa3md\xf9\xd0\xfd\x1aPm\nS\x9d\x0eQ\xad\x06P\xbd\xa8\xbfʁZ\x03(3\xa0R\x87\xda\x05(\x0f\xa0ҁ\xda=(\x13\xa02\x83Z\a(w\xa02\x81Z\x0f(\x13\xb5\x97\xf4P\x1b@\xa5\a\xf4\nP\x1bC\xa5\x02t\x1eP+@e\x1e\xf4\x0ePk@e\x02t:P;A\xa5\x01\xf4\nS\xfb@\xa5\x06t\x06P+C\xe5\x02\xf4\x06Pk@\xe5\x1et\nP\x1bA\xa5\x03\xf4:P\x1b@\xa5\x06t\x0eZJ\xff\\\x85ˈ\u05f8\x0f\xe5+F[\xc8\xe7E)\xed\x14\xa1\xf6\xe2}(v\x8d(N\x83)և(ރ(V\xdaW\xf2\x82t4\xa0m\x05(\xffAJ\x06\xd0҂t\xfc\xa0\xad\x03(oA\xca:\xd02\x82t4\xa0\xed\xb0\xbfҀ\xfa\x05(=\xa0R\x87:\x03(\x15\xa0\xf2\x81z\x03(\xf5\xa0r\x83:\x05(\r\xa1ҁ\xfa\r(\r\xa0R\x9f:\a(5\xa02\x81z\x1f(\x15\xa02\x83:\x05(\xfd\xa0R\x80\xfa\r(\x1d\xa1ҁ:\x03(5\xa0r\xd6g\xf4\xb9\x8b\x97!\xabq\x1f\xcaN\xe0GJ\x1bE)\xfd\x14\xa1\xb6\xf2}(N\x8f(V\x83)އ(փ(N\xdcW҃\xb44\xa0}\x05(\xfbAJ\x19\xd02\x82\xb4\xfc\xa0\xbd\x03(kA\xca9\xd0R\x81\xb44\xa0\xfd\xb4\xbfҀ\xda\x05(3\xa0r\x87Z\x03(\x17\xa0\xf2\x81Z\x03(\xf3\xa0R\x83\xda\x05(\x0f\xa1ҁ\xda\r(\x13\xa02\x9fZ\a(7\xa02\x81Z\x1f(\x13\xa0Ҁ\xda\x05(\xff\xa0R\x80\xda\r "), + }, + { + Type: prompb.Chunk_XOR, + MinTimeMs: 14400000, + MaxTimeMs: 28740000, + Data: []byte("\x00\xf0\x80\xe8\xdd\r@\xd7p\x00\x00\x00\x00\x00\xe0\xd4\x03\xe0G\xca+C)\xbd\x1c\xb6\x19\xfdh\x06P\x13\xa0i@v\x83\xa5\x00\xfa\x02\x94\x0fh\nP\xf3\xa0\x19@V\x81\xe5\x01z\x01\x94\x1dh\x0eP3\xa0)@6\x8f\xa5\x01\xfa\x06\x94\x03h\nPs\xa09@րe\x01z\x1f\x94\x05h\x06P3\xa0)A\xf6\x80\xa5\x00\xfa\x06\x94\ai\xfaP\x13\xa0\x19@ր\xe5\az\x01\x94\x05h\x1eP\x13\xa1\xe9@6\x80\xa5\x03\xfa\x02\x94\x03h:P\x13\xa0y@V\x80e\x1fz\x03\x94\rh\x06P\x13\xa0\xe9@v\x81\xa5\x00\xfa\x02\x94?h\nP3\xa0\x19@V\x83\xe5\x01z\x01\x94\rh\x0eZ\x8e\xff\xad\xccjSnC\xe9O\xdcH\xe9Ch\xa53\xa3\x97\x02}h2\x85\xe8\xf2\x85h2\x9c\xe8R\x8fhR\x83\xed\xe5}(;CJ\t\xd02\x8e\xb4\x1c\xa1\xbd\x03(+O\xca\t\xd0ҁ\xb4\x14\xa3\xfd\x05(\x1bCJ\tۋ\xff(\x15\xa02\x83z\a(u\xa02\x81:\r(\x1d\xa3Ҁ\xfa\x05(=\xa0R\x87:\x03(\x15\xa0\xf2\x81z\x03)\xf5\xa0r\x83:\x05(\r\xa1ҁ\xfa\r(\r\xa0R\x8f:\a(5\xa02\x81z\x1f(\x15\xa02\x83:\x05-\xa6\x7f\xda\x02\x94\x03\xe8\x1aP\x1d\xa0\xe9@N\x80e\x03Z\x03\x94=\xe8\x06P\x15\xa0y@N\x83\xa5\x00\xda\x02\x94\x0f\xe8\nP\r\xa3\xe9@N\x81\xe5\x01Z\x01\x94\x1d\xe8\x0eP5\xa0\x19@N\x87\xa5\x01\xda\x06\x94\x03\xe8\nP}\xa0)@\u0380e\x01Z\x7f\x94\x05\xe8\x06P5\xa09A\u0380\xa5\x00\xda\x06\x94\a\xe8zP\r\xa0)@\u0380\xe5\aZ\x01\x94\x05\xe8\x1eP\x15\xa0\x19G\u0380\xa5\x03\xda\x02\x94\x03\xe8:P\x1d\xa0i"), + }, + }, + }, + }, + }, + { + ChunkedSeries: []*prompb.ChunkedSeries{ + { + Labels: []prompb.Label{ + {Name: "__name__", Value: "test_metric1"}, + {Name: "b", Value: "c"}, + {Name: "baz", Value: "qux"}, + {Name: "d", Value: "e"}, + {Name: "foo", Value: "bar3"}, + }, + Chunks: []prompb.Chunk{ + { + Type: prompb.Chunk_XOR, + MinTimeMs: 28800000, + MaxTimeMs: 28800000, + Data: []byte("\x00\x01\x80л\x1b@\xe7p\x00\x00\x00\x00\x00\x00"), }, }, }, @@ -409,8 +412,8 @@ func TestStreamReadEndpoint(t *testing.T) { Chunks: []prompb.Chunk{ { Type: prompb.Chunk_XOR, - MaxTimeMs: 7140000, - Data: []byte("\000x\000\000\000\000\000\000\000\000\000\340\324\003\302|\005\224\000\301\254}\351z2\320O\355\264n[\007\316\224\243md\371\320\375\032Pm\nS\235\016Q\255\006P\275\250\277\312\201Z\003(3\240R\207\332\005(\017\240\322\201\332=(\023\2402\203Z\007(w\2402\201Z\017(\023\265\227\364P\033@\245\007\364\nP\033C\245\002t\036P+@e\036\364\016Pk@e\002t:P;A\245\001\364\nS\373@\245\006t\006P+C\345\002\364\006Pk@\345\036t\nP\033A\245\003\364:P\033@\245\006t\016ZJ\377\\\205\313\210\327\270\017\345+F[\310\347E)\355\024\241\366\342}(v\215(N\203)\326\207(\336\203(V\332W\362\202t4\240m\005(\377AJ\006\320\322\202t\374\240\255\003(oA\312:\3202"), + MaxTimeMs: 14340000, + Data: []byte("\x00\xf0\x00\x00\x00\x00\x00\x00\x00\x00\x00\xe0\xd4\x03\xc2|\x05\x94\x00\xc1\xac}\xe9z2\xd0O\xed\xb4n[\aΔ\xa3md\xf9\xd0\xfd\x1aPm\nS\x9d\x0eQ\xad\x06P\xbd\xa8\xbfʁZ\x03(3\xa0R\x87\xda\x05(\x0f\xa0ҁ\xda=(\x13\xa02\x83Z\a(w\xa02\x81Z\x0f(\x13\xb5\x97\xf4P\x1b@\xa5\a\xf4\nP\x1bC\xa5\x02t\x1eP+@e\x1e\xf4\x0ePk@e\x02t:P;A\xa5\x01\xf4\nS\xfb@\xa5\x06t\x06P+C\xe5\x02\xf4\x06Pk@\xe5\x1et\nP\x1bA\xa5\x03\xf4:P\x1b@\xa5\x06t\x0eZJ\xff\\\x85ˈ\u05f8\x0f\xe5+F[\xc8\xe7E)\xed\x14\xa1\xf6\xe2}(v\x8d(N\x83)և(ރ(V\xdaW\xf2\x82t4\xa0m\x05(\xffAJ\x06\xd0҂t\xfc\xa0\xad\x03(oA\xca:\xd02\x82t4\xa0\xed\xb0\xbfҀ\xfa\x05(=\xa0R\x87:\x03(\x15\xa0\xf2\x81z\x03(\xf5\xa0r\x83:\x05(\r\xa1ҁ\xfa\r(\r\xa0R\x9f:\a(5\xa02\x81z\x1f(\x15\xa02\x83:\x05(\xfd\xa0R\x80\xfa\r(\x1d\xa1ҁ:\x03(5\xa0r\xd6g\xf4\xb9\x8b\x97!\xabq\x1f\xcaN\xe0GJ\x1bE)\xfd\x14\xa1\xb6\xf2}(N\x8f(V\x83)އ(փ(N\xdcW҃\xb44\xa0}\x05(\xfbAJ\x19\xd02\x82\xb4\xfc\xa0\xbd\x03(kA\xca9\xd0R\x81\xb44\xa0\xfd\xb4\xbfҀ\xda\x05(3\xa0r\x87Z\x03(\x17\xa0\xf2\x81Z\x03(\xf3\xa0R\x83\xda\x05(\x0f\xa1ҁ\xda\r(\x13\xa02\x9fZ\a(7\xa02\x81Z\x1f(\x13\xa0Ҁ\xda\x05(\xff\xa0R\x80\xda\r "), }, }, }, diff --git a/tsdb/db_test.go b/tsdb/db_test.go index 70639085e4..14297c3dc3 100644 --- a/tsdb/db_test.go +++ b/tsdb/db_test.go @@ -247,8 +247,8 @@ func TestNoPanicAfterWALCorruption(t *testing.T) { var maxt int64 ctx := context.Background() { - // Appending 121 samples because on the 121st a new chunk will be created. - for i := 0; i < 121; i++ { + // Appending 221 samples because on the 221st a new chunk will be created. + for i := 0; i < 221; i++ { app := db.Appender(ctx) _, err := app.Append(0, labels.FromStrings("foo", "bar"), maxt, 0) expSamples = append(expSamples, sample{t: maxt, v: 0}) @@ -1089,9 +1089,9 @@ func TestWALReplayRaceOnSamplesLoggedBeforeSeries(t *testing.T) { numSamplesBeforeSeriesCreation = 1000 ) - // We test both with few and many samples appended after series creation. If samples are < 120 then there's no + // We test both with few and many samples appended after series creation. If samples are < 220 then there's no // mmap-ed chunk, otherwise there's at least 1 mmap-ed chunk when replaying the WAL. - for _, numSamplesAfterSeriesCreation := range []int{1, 1000} { + for _, numSamplesAfterSeriesCreation := range []int{1, 2000} { for run := 1; run <= numRuns; run++ { t.Run(fmt.Sprintf("samples after series creation = %d, run = %d", numSamplesAfterSeriesCreation, run), func(t *testing.T) { testWALReplayRaceOnSamplesLoggedBeforeSeries(t, numSamplesBeforeSeriesCreation, numSamplesAfterSeriesCreation) @@ -1160,8 +1160,8 @@ func testWALReplayRaceOnSamplesLoggedBeforeSeries(t *testing.T, numSamplesBefore } require.NoError(t, chunksIt.Err()) - // We expect 1 chunk every 120 samples after series creation. - require.Equalf(t, (numSamplesAfterSeriesCreation/120)+1, actualChunks, "series: %s", set.At().Labels().String()) + // We expect 1 chunk every 220 samples after series creation. + require.Equalf(t, (numSamplesAfterSeriesCreation/220)+1, actualChunks, "series: %s", set.At().Labels().String()) } require.NoError(t, set.Err()) diff --git a/tsdb/head_test.go b/tsdb/head_test.go index 0a2a2ee6f4..9a7220957a 100644 --- a/tsdb/head_test.go +++ b/tsdb/head_test.go @@ -808,7 +808,7 @@ func TestMemSeries_truncateChunks(t *testing.T) { s := newMemSeries(labels.FromStrings("a", "b"), 1, defaultIsolationDisabled) - for i := 0; i < 4000; i += 5 { + for i := 0; i < 8000; i += 5 { ok, _ := s.append(int64(i), float64(i), 0, chunkDiskMapper, chunkRange) require.True(t, ok, "sample append failed") } @@ -825,9 +825,9 @@ func TestMemSeries_truncateChunks(t *testing.T) { require.NotNil(t, chk) require.NoError(t, err) - s.truncateChunksBefore(2000, 0) + s.truncateChunksBefore(4000, 0) - require.Equal(t, int64(2000), s.mmappedChunks[0].minTime) + require.Equal(t, int64(4000), s.mmappedChunks[0].minTime) _, _, err = s.chunk(0, chunkDiskMapper, &memChunkPool) require.Equal(t, storage.ErrNotFound, err, "first chunks not gone") require.Equal(t, countBefore/2, len(s.mmappedChunks)+1) // +1 for the head chunk. @@ -1364,9 +1364,9 @@ func TestMemSeries_append(t *testing.T) { require.Equal(t, int64(1000), s.headChunk.minTime, "wrong chunk range") require.Equal(t, int64(1001), s.headChunk.maxTime, "wrong chunk range") - // Fill the range [1000,2000) with many samples. Intermediate chunks should be cut - // at approximately 120 samples per chunk. - for i := 1; i < 1000; i++ { + // Fill the range [1000,3000) with many samples. Intermediate chunks should be cut + // at approximately 220 samples per chunk. + for i := 1; i < 2000; i++ { ok, _ := s.append(1001+int64(i), float64(i), 0, chunkDiskMapper, chunkRange) require.True(t, ok, "append failed") } @@ -1437,7 +1437,7 @@ func TestMemSeries_appendHistogram(t *testing.T) { } func TestMemSeries_append_atVariableRate(t *testing.T) { - const samplesPerChunk = 120 + const samplesPerChunk = 220 dir := t.TempDir() // This is usually taken from the Head, but passing manually here. chunkDiskMapper, err := chunks.NewChunkDiskMapper(nil, dir, chunkenc.NewPool(), chunks.DefaultWriteBufferSize, chunks.DefaultWriteQueueSize) @@ -2983,7 +2983,7 @@ func TestAppendHistogram(t *testing.T) { } func TestHistogramInWALAndMmapChunk(t *testing.T) { - head, _ := newTestHead(t, 3000, false, false) + head, _ := newTestHead(t, 6000, false, false) t.Cleanup(func() { require.NoError(t, head.Close()) }) @@ -2992,7 +2992,7 @@ func TestHistogramInWALAndMmapChunk(t *testing.T) { // Series with only histograms. s1 := labels.FromStrings("a", "b1") k1 := s1.String() - numHistograms := 300 + numHistograms := 600 exp := map[string][]tsdbutil.Sample{} ts := int64(0) var app storage.Appender @@ -3728,7 +3728,7 @@ func TestHistogramCounterResetHeader(t *testing.T) { checkExpCounterResetHeader(chunkenc.CounterReset) // Add 2 non-counter reset histograms. - for i := 0; i < 250; i++ { + for i := 0; i < 500; i++ { appendHistogram(h) } checkExpCounterResetHeader(chunkenc.NotCounterReset, chunkenc.NotCounterReset) @@ -3756,7 +3756,7 @@ func TestHistogramCounterResetHeader(t *testing.T) { checkExpCounterResetHeader(chunkenc.CounterReset) // Add 2 non-counter reset histograms. Just to have some non-counter reset chunks in between. - for i := 0; i < 250; i++ { + for i := 0; i < 500; i++ { appendHistogram(h) } checkExpCounterResetHeader(chunkenc.NotCounterReset, chunkenc.NotCounterReset) @@ -4223,7 +4223,7 @@ func TestHeadInit_DiscardChunksWithUnsupportedEncoding(t *testing.T) { h.chunkDiskMapper.WriteChunk(chunks.HeadSeriesRef(seriesRef), 500, 600, uc, false, func(err error) { require.NoError(t, err) }) app = h.Appender(ctx) - for i := 700; i < 1200; i++ { + for i := 700; i < 1700; i++ { _, err := app.Append(0, seriesLabels, int64(i), float64(i)) require.NoError(t, err) } From 83f43982c93cc776d7ef4b74c782f8f1eae72a37 Mon Sep 17 00:00:00 2001 From: Justin Lei Date: Mon, 27 Mar 2023 17:02:20 -0700 Subject: [PATCH 077/231] Add support for native histograms to concreteSeriesIterator Signed-off-by: Justin Lei --- storage/remote/codec.go | 196 +++++++++++++++++++++------ storage/remote/codec_test.go | 176 ++++++++++++++++++++++-- storage/remote/write_handler.go | 2 +- storage/remote/write_handler_test.go | 2 +- tsdb/chunkenc/chunk.go | 2 +- 5 files changed, 326 insertions(+), 52 deletions(-) diff --git a/storage/remote/codec.go b/storage/remote/codec.go index ba802bccb3..78534ed931 100644 --- a/storage/remote/codec.go +++ b/storage/remote/codec.go @@ -17,6 +17,7 @@ import ( "errors" "fmt" "io" + "math" "net/http" "sort" "strings" @@ -173,7 +174,7 @@ func FromQueryResult(sortSeries bool, res *prompb.QueryResult) storage.SeriesSet return errSeriesSet{err: err} } lbls := labelProtosToLabels(ts.Labels) - series = append(series, &concreteSeries{labels: lbls, samples: ts.Samples}) + series = append(series, &concreteSeries{labels: lbls, floats: ts.Samples, histograms: ts.Histograms}) } if sortSeries { @@ -359,8 +360,9 @@ func (c *concreteSeriesSet) Warnings() storage.Warnings { return nil } // concreteSeries implements storage.Series. type concreteSeries struct { - labels labels.Labels - samples []prompb.Sample + labels labels.Labels + floats []prompb.Sample + histograms []prompb.Histogram } func (c *concreteSeries) Labels() labels.Labels { @@ -372,84 +374,168 @@ func (c *concreteSeries) Iterator(it chunkenc.Iterator) chunkenc.Iterator { csi.reset(c) return csi } - return newConcreteSeriersIterator(c) + return newConcreteSeriesIterator(c) } // concreteSeriesIterator implements storage.SeriesIterator. type concreteSeriesIterator struct { - cur int - series *concreteSeries + floatsCur int + histogramsCur int + curValType chunkenc.ValueType + series *concreteSeries } -func newConcreteSeriersIterator(series *concreteSeries) chunkenc.Iterator { +func newConcreteSeriesIterator(series *concreteSeries) chunkenc.Iterator { return &concreteSeriesIterator{ - cur: -1, - series: series, + floatsCur: -1, + histogramsCur: -1, + curValType: chunkenc.ValNone, + series: series, } } func (c *concreteSeriesIterator) reset(series *concreteSeries) { - c.cur = -1 + c.floatsCur = -1 + c.histogramsCur = -1 + c.curValType = chunkenc.ValNone c.series = series } // Seek implements storage.SeriesIterator. func (c *concreteSeriesIterator) Seek(t int64) chunkenc.ValueType { - if c.cur == -1 { - c.cur = 0 + if c.floatsCur == -1 { + c.floatsCur = 0 } - if c.cur >= len(c.series.samples) { + if c.histogramsCur == -1 { + c.histogramsCur = 0 + } + if c.floatsCur >= len(c.series.floats) && c.histogramsCur >= len(c.series.histograms) { return chunkenc.ValNone } + // No-op check. - if s := c.series.samples[c.cur]; s.Timestamp >= t { - return chunkenc.ValFloat + if (c.curValType == chunkenc.ValFloat && c.series.floats[c.floatsCur].Timestamp >= t) || + ((c.curValType == chunkenc.ValHistogram || c.curValType == chunkenc.ValFloatHistogram) && c.series.histograms[c.histogramsCur].Timestamp >= t) { + return c.curValType } - // Do binary search between current position and end. - c.cur += sort.Search(len(c.series.samples)-c.cur, func(n int) bool { - return c.series.samples[n+c.cur].Timestamp >= t + + c.curValType = chunkenc.ValNone + + // Binary search between current position and end for both float and histograms samples. + c.floatsCur += sort.Search(len(c.series.floats)-c.floatsCur, func(n int) bool { + return c.series.floats[n+c.floatsCur].Timestamp >= t }) - if c.cur < len(c.series.samples) { - return chunkenc.ValFloat + c.histogramsCur += sort.Search(len(c.series.histograms)-c.histogramsCur, func(n int) bool { + return c.series.histograms[n+c.histogramsCur].Timestamp >= t + }) + + if c.floatsCur < len(c.series.floats) && c.histogramsCur < len(c.series.histograms) { + // If float samples and histogram samples have overlapping timestamps prefer the float samples. + if c.series.floats[c.floatsCur].Timestamp <= c.series.histograms[c.histogramsCur].Timestamp { + c.curValType = chunkenc.ValFloat + } else { + c.curValType = getHistogramValType(&c.series.histograms[c.histogramsCur]) + } + // When the timestamps do not overlap the cursor for the non-selected sample type has advanced too + // far; we decrement it back down here. + if c.series.floats[c.floatsCur].Timestamp != c.series.histograms[c.histogramsCur].Timestamp { + if c.curValType == chunkenc.ValFloat { + c.histogramsCur-- + } else { + c.floatsCur-- + } + } + } else if c.floatsCur < len(c.series.floats) { + c.curValType = chunkenc.ValFloat + } else if c.histogramsCur < len(c.series.histograms) { + c.curValType = getHistogramValType(&c.series.histograms[c.histogramsCur]) } - return chunkenc.ValNone - // TODO(beorn7): Add histogram support. + + return c.curValType +} + +func getHistogramValType(h *prompb.Histogram) chunkenc.ValueType { + _, isInt := h.GetCount().(*prompb.Histogram_CountInt) + if isInt { + return chunkenc.ValHistogram + } + return chunkenc.ValFloatHistogram } // At implements chunkenc.Iterator. func (c *concreteSeriesIterator) At() (t int64, v float64) { - s := c.series.samples[c.cur] + if c.curValType != chunkenc.ValFloat { + panic("iterator is not on a float sample") + } + s := c.series.floats[c.floatsCur] return s.Timestamp, s.Value } -// AtHistogram always returns (0, nil) because there is no support for histogram -// values yet. -// TODO(beorn7): Fix that for histogram support in remote storage. +// AtHistogram implements chunkenc.Iterator func (c *concreteSeriesIterator) AtHistogram() (int64, *histogram.Histogram) { - return 0, nil + if c.curValType != chunkenc.ValHistogram { + panic("iterator is not on an integer histogram sample") + } + h := c.series.histograms[c.histogramsCur] + return h.Timestamp, HistogramProtoToHistogram(h) } -// AtFloatHistogram always returns (0, nil) because there is no support for histogram -// values yet. -// TODO(beorn7): Fix that for histogram support in remote storage. +// AtFloatHistogram implements chunkenc.Iterator func (c *concreteSeriesIterator) AtFloatHistogram() (int64, *histogram.FloatHistogram) { - return 0, nil + switch c.curValType { + case chunkenc.ValHistogram: + fh := c.series.histograms[c.histogramsCur] + return fh.Timestamp, HistogramProtoToFloatHistogram(fh) + case chunkenc.ValFloatHistogram: + fh := c.series.histograms[c.histogramsCur] + return fh.Timestamp, FloatHistogramProtoToFloatHistogram(fh) + default: + panic("iterator is not on a histogram sample") + } } // AtT implements chunkenc.Iterator. func (c *concreteSeriesIterator) AtT() int64 { - s := c.series.samples[c.cur] - return s.Timestamp + if c.curValType == chunkenc.ValHistogram || c.curValType == chunkenc.ValFloatHistogram { + return c.series.histograms[c.histogramsCur].Timestamp + } + return c.series.floats[c.floatsCur].Timestamp } +const noTS = int64(math.MaxInt64) + // Next implements chunkenc.Iterator. func (c *concreteSeriesIterator) Next() chunkenc.ValueType { - c.cur++ - if c.cur < len(c.series.samples) { - return chunkenc.ValFloat + peekFloatTS := noTS + if c.floatsCur+1 < len(c.series.floats) { + peekFloatTS = c.series.floats[c.floatsCur+1].Timestamp } - return chunkenc.ValNone - // TODO(beorn7): Add histogram support. + peekHistTS := noTS + if c.histogramsCur+1 < len(c.series.histograms) { + peekHistTS = c.series.histograms[c.histogramsCur+1].Timestamp + } + c.curValType = chunkenc.ValNone + + if peekFloatTS < peekHistTS { + c.floatsCur++ + c.curValType = chunkenc.ValFloat + } else if peekHistTS < peekFloatTS { + c.histogramsCur++ + c.curValType = chunkenc.ValHistogram + } else if peekFloatTS == noTS && peekHistTS == noTS { + // This only happens when the iterator is exhausted; we set the cursors off the end to prevent + // Seek() from returning anything afterwards. + c.floatsCur = len(c.series.floats) + c.histogramsCur = len(c.series.histograms) + } else { + // Prefer float samples to histogram samples if there's a conflict. We advance the cursor for histograms + // anyway otherwise the histogram sample will get selected on the next call to Next(). + c.floatsCur++ + c.histogramsCur++ + c.curValType = chunkenc.ValFloat + } + + return c.curValType } // Err implements chunkenc.Iterator. @@ -557,10 +643,10 @@ func HistogramProtoToHistogram(hp prompb.Histogram) *histogram.Histogram { } } -// HistogramProtoToFloatHistogram extracts a (normal integer) Histogram from the +// FloatHistogramProtoToFloatHistogram extracts a float Histogram from the // provided proto message to a Float Histogram. The caller has to make sure that -// the proto message represents an float histogram and not a integer histogram. -func HistogramProtoToFloatHistogram(hp prompb.Histogram) *histogram.FloatHistogram { +// the proto message represents a float histogram and not an integer histogram. +func FloatHistogramProtoToFloatHistogram(hp prompb.Histogram) *histogram.FloatHistogram { return &histogram.FloatHistogram{ CounterResetHint: histogram.CounterResetHint(hp.ResetHint), Schema: hp.Schema, @@ -575,6 +661,24 @@ func HistogramProtoToFloatHistogram(hp prompb.Histogram) *histogram.FloatHistogr } } +// HistogramProtoToFloatHistogram extracts and converts a (normal integer) histogram from the provided proto message +// to a float histogram. The caller has to make sure that the proto message represents an integer histogram and not a +// float histogram. +func HistogramProtoToFloatHistogram(hp prompb.Histogram) *histogram.FloatHistogram { + return &histogram.FloatHistogram{ + CounterResetHint: histogram.CounterResetHint(hp.ResetHint), + Schema: hp.Schema, + ZeroThreshold: hp.ZeroThreshold, + ZeroCount: float64(hp.GetZeroCountInt()), + Count: float64(hp.GetCountInt()), + Sum: hp.Sum, + PositiveSpans: spansProtoToSpans(hp.GetPositiveSpans()), + PositiveBuckets: deltasToCounts(hp.GetPositiveDeltas()), + NegativeSpans: spansProtoToSpans(hp.GetNegativeSpans()), + NegativeBuckets: deltasToCounts(hp.GetNegativeDeltas()), + } +} + func spansProtoToSpans(s []prompb.BucketSpan) []histogram.Span { spans := make([]histogram.Span, len(s)) for i := 0; i < len(s); i++ { @@ -584,6 +688,16 @@ func spansProtoToSpans(s []prompb.BucketSpan) []histogram.Span { return spans } +func deltasToCounts(deltas []int64) []float64 { + counts := make([]float64, len(deltas)) + var cur float64 + for i, d := range deltas { + cur += float64(d) + counts[i] = cur + } + return counts +} + func HistogramToHistogramProto(timestamp int64, h *histogram.Histogram) prompb.Histogram { return prompb.Histogram{ Count: &prompb.Histogram_CountInt{CountInt: h.Count}, diff --git a/storage/remote/codec_test.go b/storage/remote/codec_test.go index 4137f91aa3..36d0d7c31d 100644 --- a/storage/remote/codec_test.go +++ b/storage/remote/codec_test.go @@ -29,6 +29,7 @@ import ( "github.com/prometheus/prometheus/storage" "github.com/prometheus/prometheus/tsdb/chunkenc" "github.com/prometheus/prometheus/tsdb/chunks" + "github.com/prometheus/prometheus/tsdb/tsdbutil" ) var testHistogram = histogram.Histogram{ @@ -174,12 +175,12 @@ func TestValidateLabelsAndMetricName(t *testing.T) { func TestConcreteSeriesSet(t *testing.T) { series1 := &concreteSeries{ - labels: labels.FromStrings("foo", "bar"), - samples: []prompb.Sample{{Value: 1, Timestamp: 2}}, + labels: labels.FromStrings("foo", "bar"), + floats: []prompb.Sample{{Value: 1, Timestamp: 2}}, } series2 := &concreteSeries{ - labels: labels.FromStrings("foo", "baz"), - samples: []prompb.Sample{{Value: 3, Timestamp: 4}}, + labels: labels.FromStrings("foo", "baz"), + floats: []prompb.Sample{{Value: 3, Timestamp: 4}}, } c := &concreteSeriesSet{ series: []storage.Series{series1, series2}, @@ -206,10 +207,10 @@ func TestConcreteSeriesClonesLabels(t *testing.T) { require.Equal(t, lbls, gotLabels) } -func TestConcreteSeriesIterator(t *testing.T) { +func TestConcreteSeriesIterator_FloatSamples(t *testing.T) { series := &concreteSeries{ labels: labels.FromStrings("foo", "bar"), - samples: []prompb.Sample{ + floats: []prompb.Sample{ {Value: 1, Timestamp: 1}, {Value: 1.5, Timestamp: 1}, {Value: 2, Timestamp: 2}, @@ -255,6 +256,165 @@ func TestConcreteSeriesIterator(t *testing.T) { require.Equal(t, chunkenc.ValNone, it.Seek(2)) } +func TestConcreteSeriesIterator_HistogramSamples(t *testing.T) { + histograms := tsdbutil.GenerateTestHistograms(5) + histProtos := make([]prompb.Histogram, len(histograms)) + for i, h := range histograms { + // Results in ts sequence of 1, 1, 2, 3, 4. + var ts int64 + if i == 0 { + ts = 1 + } else { + ts = int64(i) + } + histProtos[i] = HistogramToHistogramProto(ts, h) + } + series := &concreteSeries{ + labels: labels.FromStrings("foo", "bar"), + histograms: histProtos, + } + it := series.Iterator(nil) + + // Seek to the first sample with ts=1. + require.Equal(t, chunkenc.ValHistogram, it.Seek(1)) + ts, v := it.AtHistogram() + require.Equal(t, int64(1), ts) + require.Equal(t, histograms[0], v) + + // Seek one further, next sample still has ts=1. + require.Equal(t, chunkenc.ValHistogram, it.Next()) + ts, v = it.AtHistogram() + require.Equal(t, int64(1), ts) + require.Equal(t, histograms[1], v) + + // Seek again to 1 and make sure we stay where we are. + require.Equal(t, chunkenc.ValHistogram, it.Seek(1)) + ts, v = it.AtHistogram() + require.Equal(t, int64(1), ts) + require.Equal(t, histograms[1], v) + + // Another seek. + require.Equal(t, chunkenc.ValHistogram, it.Seek(3)) + ts, v = it.AtHistogram() + require.Equal(t, int64(3), ts) + require.Equal(t, histograms[3], v) + + // And we don't go back. + require.Equal(t, chunkenc.ValHistogram, it.Seek(2)) + ts, v = it.AtHistogram() + require.Equal(t, int64(3), ts) + require.Equal(t, histograms[3], v) + + // Seek beyond the end. + require.Equal(t, chunkenc.ValNone, it.Seek(5)) + // And we don't go back. (This exposes issue #10027.) + require.Equal(t, chunkenc.ValNone, it.Seek(2)) +} + +func TestConcreteSeriesIterator_FloatAndHistogramSamples(t *testing.T) { + // Series starts as histograms, then transitions to floats at ts=8 (with an overlap from ts=8 to ts=10), then + // transitions back to histograms at ts=16. + histograms := tsdbutil.GenerateTestHistograms(15) + histProtos := make([]prompb.Histogram, len(histograms)) + for i, h := range histograms { + if i < 10 { + histProtos[i] = HistogramToHistogramProto(int64(i+1), h) + } else { + histProtos[i] = HistogramToHistogramProto(int64(i+6), h) + } + } + series := &concreteSeries{ + labels: labels.FromStrings("foo", "bar"), + floats: []prompb.Sample{ + {Value: 1, Timestamp: 8}, + {Value: 2, Timestamp: 9}, + {Value: 3, Timestamp: 10}, + {Value: 4, Timestamp: 11}, + {Value: 5, Timestamp: 12}, + {Value: 6, Timestamp: 13}, + {Value: 7, Timestamp: 14}, + {Value: 8, Timestamp: 15}, + }, + histograms: histProtos, + } + it := series.Iterator(nil) + + var ( + ts int64 + v float64 + h *histogram.Histogram + fh *histogram.FloatHistogram + ) + require.Equal(t, chunkenc.ValHistogram, it.Next()) + ts, h = it.AtHistogram() + require.Equal(t, int64(1), ts) + require.Equal(t, histograms[0], h) + + require.Equal(t, chunkenc.ValHistogram, it.Next()) + ts, h = it.AtHistogram() + require.Equal(t, int64(2), ts) + require.Equal(t, histograms[1], h) + + // Seek to the first float/histogram sample overlap at ts=8 (should prefer the float sample). + require.Equal(t, chunkenc.ValFloat, it.Seek(8)) + ts, v = it.At() + require.Equal(t, int64(8), ts) + require.Equal(t, 1., v) + + // Attempting to seek backwards should do nothing. + require.Equal(t, chunkenc.ValFloat, it.Seek(1)) + ts, v = it.At() + require.Equal(t, int64(8), ts) + require.Equal(t, 1., v) + + // Seeking to 8 again should also do nothing. + require.Equal(t, chunkenc.ValFloat, it.Seek(8)) + ts, v = it.At() + require.Equal(t, int64(8), ts) + require.Equal(t, 1., v) + + // Again, should prefer the float sample. + require.Equal(t, chunkenc.ValFloat, it.Next()) + ts, v = it.At() + require.Equal(t, int64(9), ts) + require.Equal(t, 2., v) + + // Seek to ts=11 where there are only float samples. + require.Equal(t, chunkenc.ValFloat, it.Seek(11)) + ts, v = it.At() + require.Equal(t, int64(11), ts) + require.Equal(t, 4., v) + + // Seek to ts=15 right before the transition back to histogram samples. + require.Equal(t, chunkenc.ValFloat, it.Seek(15)) + ts, v = it.At() + require.Equal(t, int64(15), ts) + require.Equal(t, 8., v) + + require.Equal(t, chunkenc.ValHistogram, it.Next()) + ts, h = it.AtHistogram() + require.Equal(t, int64(16), ts) + require.Equal(t, histograms[10], h) + + // Getting a float histogram from an int histogram works. + require.Equal(t, chunkenc.ValHistogram, it.Next()) + ts, fh = it.AtFloatHistogram() + require.Equal(t, int64(17), ts) + expected := HistogramProtoToFloatHistogram(HistogramToHistogramProto(int64(17), histograms[11])) + require.Equal(t, expected, fh) + + // Keep calling Next() until the end. + for i := 0; i < 3; i++ { + require.Equal(t, chunkenc.ValHistogram, it.Next()) + } + + // The iterator is exhausted. + require.Equal(t, chunkenc.ValNone, it.Next()) + require.Equal(t, chunkenc.ValNone, it.Next()) + // Should also not be able to seek backwards again. + require.Equal(t, chunkenc.ValNone, it.Seek(1)) +} + func TestFromQueryResultWithDuplicates(t *testing.T) { ts1 := prompb.TimeSeries{ Labels: []prompb.Label{ @@ -368,7 +528,7 @@ func TestNilHistogramProto(t *testing.T) { // This function will panic if it impromperly handles nil // values, causing the test to fail. HistogramProtoToHistogram(prompb.Histogram{}) - HistogramProtoToFloatHistogram(prompb.Histogram{}) + FloatHistogramProtoToFloatHistogram(prompb.Histogram{}) } func exampleHistogram() histogram.Histogram { @@ -563,7 +723,7 @@ func TestFloatHistogramToProtoConvert(t *testing.T) { require.Equal(t, p, FloatHistogramToHistogramProto(1337, &h)) - require.Equal(t, h, *HistogramProtoToFloatHistogram(p)) + require.Equal(t, h, *FloatHistogramProtoToFloatHistogram(p)) } } diff --git a/storage/remote/write_handler.go b/storage/remote/write_handler.go index 45304c43c4..5aa1130888 100644 --- a/storage/remote/write_handler.go +++ b/storage/remote/write_handler.go @@ -126,7 +126,7 @@ func (h *writeHandler) write(ctx context.Context, req *prompb.WriteRequest) (err for _, hp := range ts.Histograms { if hp.GetCountFloat() > 0 || hp.GetZeroCountFloat() > 0 { // It is a float histogram. - fhs := HistogramProtoToFloatHistogram(hp) + fhs := FloatHistogramProtoToFloatHistogram(hp) _, err = app.AppendHistogram(0, labels, hp.Timestamp, nil, fhs) } else { hs := HistogramProtoToHistogram(hp) diff --git a/storage/remote/write_handler_test.go b/storage/remote/write_handler_test.go index 58c4439fa8..9c787f17e5 100644 --- a/storage/remote/write_handler_test.go +++ b/storage/remote/write_handler_test.go @@ -69,7 +69,7 @@ func TestRemoteWriteHandler(t *testing.T) { for _, hp := range ts.Histograms { if hp.GetCountFloat() > 0 || hp.GetZeroCountFloat() > 0 { // It is a float histogram. - fh := HistogramProtoToFloatHistogram(hp) + fh := FloatHistogramProtoToFloatHistogram(hp) require.Equal(t, mockHistogram{labels, hp.Timestamp, nil, fh}, appendable.histograms[k]) } else { h := HistogramProtoToHistogram(hp) diff --git a/tsdb/chunkenc/chunk.go b/tsdb/chunkenc/chunk.go index c550cbc78e..1ebef3eb10 100644 --- a/tsdb/chunkenc/chunk.go +++ b/tsdb/chunkenc/chunk.go @@ -96,7 +96,7 @@ type Iterator interface { // timestamp equal or greater than t. If the current sample found by a // previous `Next` or `Seek` operation already has this property, Seek // has no effect. If a sample has been found, Seek returns the type of - // its value. Otherwise, it returns ValNone, after with the iterator is + // its value. Otherwise, it returns ValNone, after which the iterator is // exhausted. Seek(t int64) ValueType // At returns the current timestamp/value pair if the value is a float. From f90013a5a0a0eb55ace4bd9f994769aceef2a0fc Mon Sep 17 00:00:00 2001 From: Justin Lei <97976793+leizor@users.noreply.github.com> Date: Wed, 5 Apr 2023 16:09:09 -0700 Subject: [PATCH 078/231] Update storage/remote/codec.go MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Björn Rabenstein Signed-off-by: Justin Lei <97976793+leizor@users.noreply.github.com> --- storage/remote/codec.go | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/storage/remote/codec.go b/storage/remote/codec.go index 78534ed931..bfbd08d24b 100644 --- a/storage/remote/codec.go +++ b/storage/remote/codec.go @@ -455,8 +455,7 @@ func (c *concreteSeriesIterator) Seek(t int64) chunkenc.ValueType { } func getHistogramValType(h *prompb.Histogram) chunkenc.ValueType { - _, isInt := h.GetCount().(*prompb.Histogram_CountInt) - if isInt { + if _, isInt := h.GetCount().(*prompb.Histogram_CountInt); isInt { return chunkenc.ValHistogram } return chunkenc.ValFloatHistogram From cca7178a120d40740a9957cc0567d081643ad1bc Mon Sep 17 00:00:00 2001 From: Arve Knudsen Date: Fri, 7 Apr 2023 18:03:19 +0200 Subject: [PATCH 079/231] tsdb: Improve a couple of histogram documentation comments Signed-off-by: Arve Knudsen --- tsdb/head_append.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tsdb/head_append.go b/tsdb/head_append.go index 14e343f74d..6bc91ae06d 100644 --- a/tsdb/head_append.go +++ b/tsdb/head_append.go @@ -430,7 +430,7 @@ func (s *memSeries) appendable(t int64, v float64, headMaxt, minValidTime, oooTi return false, headMaxt - t, storage.ErrOutOfOrderSample } -// appendableHistogram checks whether the given sample is valid for appending to the series. +// appendableHistogram checks whether the given histogram is valid for appending to the series. func (s *memSeries) appendableHistogram(t int64, h *histogram.Histogram) error { c := s.head() if c == nil { @@ -452,7 +452,7 @@ func (s *memSeries) appendableHistogram(t int64, h *histogram.Histogram) error { return nil } -// appendableFloatHistogram checks whether the given sample is valid for appending to the series. +// appendableFloatHistogram checks whether the given float histogram is valid for appending to the series. func (s *memSeries) appendableFloatHistogram(t int64, fh *histogram.FloatHistogram) error { c := s.head() if c == nil { From d5b33e1419b0fdc70719a45b747e155fa1a0781c Mon Sep 17 00:00:00 2001 From: Matthieu MOREL Date: Sat, 8 Apr 2023 10:26:06 +0200 Subject: [PATCH 080/231] =?UTF-8?q?yamllint:=20don=E2=80=99t=20check=20key?= =?UTF-8?q?s=20and=20add=20file=20extension=20to=20config=20file?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Matthieu MOREL Signed-off-by: Matthieu MOREL --- .yamllint => .yamllint.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) rename .yamllint => .yamllint.yml (90%) diff --git a/.yamllint b/.yamllint.yml similarity index 90% rename from .yamllint rename to .yamllint.yml index 19552574b2..955a5a6270 100644 --- a/.yamllint +++ b/.yamllint.yml @@ -20,5 +20,4 @@ rules: config/testdata/section_key_dup.bad.yml line-length: disable truthy: - ignore: | - .github/workflows/*.yml + check-keys: false From fb67d368a218d7279fe63d791f48a33e88113ad7 Mon Sep 17 00:00:00 2001 From: Ben Ye Date: Tue, 11 Apr 2023 13:45:34 -0700 Subject: [PATCH 081/231] use consistent error for instant and range query 400 Signed-off-by: Ben Ye --- web/api/v1/api.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/api/v1/api.go b/web/api/v1/api.go index 3b6ade562a..a210e00017 100644 --- a/web/api/v1/api.go +++ b/web/api/v1/api.go @@ -522,7 +522,7 @@ func (api *API) queryRange(r *http.Request) (result apiFuncResult) { } qry, err := api.QueryEngine.NewRangeQuery(api.Queryable, opts, r.FormValue("query"), start, end, step) if err != nil { - return apiFuncResult{nil, &apiError{errorBadData, err}, nil, nil} + return invalidParamError(err, "query") } // From now on, we must only return with a finalizer in the result (to // be called by the caller) or call qry.Close ourselves (which is From b6573353c1495da061c25e6a08859fd3ffcd47a4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Mierzwa?= Date: Wed, 12 Apr 2023 14:05:06 +0100 Subject: [PATCH 082/231] Add query_samples_total metric MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit query_samples_total is a counter that tracks the total number of samples loaded by all queries. The goal with this metric is to be able to see the amount of 'work' done by Prometheus to service queries. At the moment we have metrics with the number of queries, plus more detailed metrics showing how much time each step of a query takes. While those metrics do help they don't show us the whole picture. Queries that do load more samples are (in general) more expensive than queries that do load fewer samples. This means that looking only at the number of queries doesn't tell us how much 'work' Prometheus received. Adding a counter that tracks the total number of samples loaded allows us to see if there was a spike in the cost of queries, not just the number of them. Signed-off-by: Łukasz Mierzwa --- promql/engine.go | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/promql/engine.go b/promql/engine.go index 52278c7a24..60a575508a 100644 --- a/promql/engine.go +++ b/promql/engine.go @@ -70,6 +70,7 @@ type engineMetrics struct { queryPrepareTime prometheus.Observer queryInnerEval prometheus.Observer queryResultSort prometheus.Observer + querySamples prometheus.Counter } // convertibleToInt64 returns true if v does not over-/underflow an int64. @@ -332,6 +333,12 @@ func NewEngine(opts EngineOpts) *Engine { Name: "queries_concurrent_max", Help: "The max number of concurrent queries.", }), + querySamples: prometheus.NewCounter(prometheus.CounterOpts{ + Namespace: namespace, + Subsystem: subsystem, + Name: "query_samples_total", + Help: "The total number of samples loaded by all queries.", + }), queryQueueTime: queryResultSummary.WithLabelValues("queue_time"), queryPrepareTime: queryResultSummary.WithLabelValues("prepare_time"), queryInnerEval: queryResultSummary.WithLabelValues("inner_eval"), @@ -357,6 +364,7 @@ func NewEngine(opts EngineOpts) *Engine { metrics.maxConcurrentQueries, metrics.queryLogEnabled, metrics.queryLogFailures, + metrics.querySamples, queryResultSummary, ) } @@ -537,7 +545,10 @@ func (ng *Engine) newTestQuery(f func(context.Context) error) Query { // statements are not handled by the Engine. func (ng *Engine) exec(ctx context.Context, q *query) (v parser.Value, ws storage.Warnings, err error) { ng.metrics.currentQueries.Inc() - defer ng.metrics.currentQueries.Dec() + defer func() { + ng.metrics.currentQueries.Dec() + ng.metrics.querySamples.Add(float64(q.sampleStats.TotalSamples)) + }() ctx, cancel := context.WithTimeout(ctx, ng.timeout) q.cancel = cancel From 01d0dda4fc4c8e57636dbce7021307377340217f Mon Sep 17 00:00:00 2001 From: Alex Le Date: Wed, 12 Apr 2023 14:18:20 -0700 Subject: [PATCH 083/231] Rename PopulateBlockFunc to BlockPopulator Signed-off-by: Alex Le --- tsdb/compact.go | 18 +++++++++--------- tsdb/compact_test.go | 6 +++--- 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/tsdb/compact.go b/tsdb/compact.go index b2d4123754..fef1c646d4 100644 --- a/tsdb/compact.go +++ b/tsdb/compact.go @@ -392,10 +392,10 @@ func CompactBlockMetas(uid ulid.ULID, blocks ...*BlockMeta) *BlockMeta { // Compact creates a new block in the compactor's directory from the blocks in the // provided directories. func (c *LeveledCompactor) Compact(dest string, dirs []string, open []*Block) (uid ulid.ULID, err error) { - return c.CompactWithPopulateBlockFunc(dest, dirs, open, DefaultPopulateBlockFunc{}) + return c.CompactWithBlockPopulator(dest, dirs, open, DefaultBlockPopulator{}) } -func (c *LeveledCompactor) CompactWithPopulateBlockFunc(dest string, dirs []string, open []*Block, populateBlockFunc PopulateBlockFunc) (uid ulid.ULID, err error) { +func (c *LeveledCompactor) CompactWithBlockPopulator(dest string, dirs []string, open []*Block, blockPopulator BlockPopulator) (uid ulid.ULID, err error) { var ( blocks []BlockReader bs []*Block @@ -439,7 +439,7 @@ func (c *LeveledCompactor) CompactWithPopulateBlockFunc(dest string, dirs []stri uid = ulid.MustNew(ulid.Now(), rand.Reader) meta := CompactBlockMetas(uid, metas...) - err = c.write(dest, meta, populateBlockFunc, blocks...) + err = c.write(dest, meta, blockPopulator, blocks...) if err == nil { if meta.Stats.NumSamples == 0 { for _, b := range bs { @@ -505,7 +505,7 @@ func (c *LeveledCompactor) Write(dest string, b BlockReader, mint, maxt int64, p } } - err := c.write(dest, meta, DefaultPopulateBlockFunc{}, b) + err := c.write(dest, meta, DefaultBlockPopulator{}, b) if err != nil { return uid, err } @@ -550,7 +550,7 @@ func (w *instrumentedChunkWriter) WriteChunks(chunks ...chunks.Meta) error { } // write creates a new block that is the union of the provided blocks into dir. -func (c *LeveledCompactor) write(dest string, meta *BlockMeta, populateBlockFunc PopulateBlockFunc, blocks ...BlockReader) (err error) { +func (c *LeveledCompactor) write(dest string, meta *BlockMeta, blockPopulator BlockPopulator, blocks ...BlockReader) (err error) { dir := filepath.Join(dest, meta.ULID.String()) tmp := dir + tmpForCreationBlockDirSuffix var closers []io.Closer @@ -598,7 +598,7 @@ func (c *LeveledCompactor) write(dest string, meta *BlockMeta, populateBlockFunc } closers = append(closers, indexw) - if err := populateBlockFunc.PopulateBlock(c.ctx, c.metrics, c.logger, c.chunkPool, c.mergeFunc, blocks, meta, indexw, chunkw); err != nil { + if err := blockPopulator.PopulateBlock(c.ctx, c.metrics, c.logger, c.chunkPool, c.mergeFunc, blocks, meta, indexw, chunkw); err != nil { return errors.Wrap(err, "populate block") } @@ -663,16 +663,16 @@ func (c *LeveledCompactor) write(dest string, meta *BlockMeta, populateBlockFunc return nil } -type PopulateBlockFunc interface { +type BlockPopulator interface { PopulateBlock(ctx context.Context, metrics *CompactorMetrics, logger log.Logger, chunkPool chunkenc.Pool, mergeFunc storage.VerticalChunkSeriesMergeFunc, blocks []BlockReader, meta *BlockMeta, indexw IndexWriter, chunkw ChunkWriter) error } -type DefaultPopulateBlockFunc struct{} +type DefaultBlockPopulator struct{} // PopulateBlock fills the index and chunk writers with new data gathered as the union // of the provided blocks. It returns meta information for the new block. // It expects sorted blocks input by mint. -func (c DefaultPopulateBlockFunc) PopulateBlock(ctx context.Context, metrics *CompactorMetrics, logger log.Logger, chunkPool chunkenc.Pool, mergeFunc storage.VerticalChunkSeriesMergeFunc, blocks []BlockReader, meta *BlockMeta, indexw IndexWriter, chunkw ChunkWriter) (err error) { +func (c DefaultBlockPopulator) PopulateBlock(ctx context.Context, metrics *CompactorMetrics, logger log.Logger, chunkPool chunkenc.Pool, mergeFunc storage.VerticalChunkSeriesMergeFunc, blocks []BlockReader, meta *BlockMeta, indexw IndexWriter, chunkw ChunkWriter) (err error) { if len(blocks) == 0 { return errors.New("cannot populate block from no readers") } diff --git a/tsdb/compact_test.go b/tsdb/compact_test.go index 1b06845215..20ebf40df0 100644 --- a/tsdb/compact_test.go +++ b/tsdb/compact_test.go @@ -441,7 +441,7 @@ func TestCompactionFailWillCleanUpTempDir(t *testing.T) { tmpdir := t.TempDir() - require.Error(t, compactor.write(tmpdir, &BlockMeta{}, DefaultPopulateBlockFunc{}, erringBReader{})) + require.Error(t, compactor.write(tmpdir, &BlockMeta{}, DefaultBlockPopulator{}, erringBReader{})) _, err = os.Stat(filepath.Join(tmpdir, BlockMeta{}.ULID.String()) + tmpForCreationBlockDirSuffix) require.True(t, os.IsNotExist(err), "directory is not cleaned up") } @@ -953,8 +953,8 @@ func TestCompaction_populateBlock(t *testing.T) { } iw := &mockIndexWriter{} - populateBlockFunc := DefaultPopulateBlockFunc{} - err = populateBlockFunc.PopulateBlock(c.ctx, c.metrics, c.logger, c.chunkPool, c.mergeFunc, blocks, meta, iw, nopChunkWriter{}) + blockPopulator := DefaultBlockPopulator{} + err = blockPopulator.PopulateBlock(c.ctx, c.metrics, c.logger, c.chunkPool, c.mergeFunc, blocks, meta, iw, nopChunkWriter{}) if tc.expErr != nil { require.Error(t, err) require.Equal(t, tc.expErr.Error(), err.Error()) From 10cc60af013c5cfc33f4cc377c423e8120030276 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Thu, 13 Apr 2023 11:07:54 +0000 Subject: [PATCH 084/231] labels: add ScratchBuilder.Overwrite for slice implementation This is a method used by some downstream projects; it was created to optimize the implementation in `labels_string.go` but we should have one for both implementations so the same code works with either. Signed-off-by: Bryan Boreham --- model/labels/labels.go | 6 ++++++ model/labels/labels_string.go | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/model/labels/labels.go b/model/labels/labels.go index e3a478a25d..b7398d17f9 100644 --- a/model/labels/labels.go +++ b/model/labels/labels.go @@ -631,3 +631,9 @@ func (b *ScratchBuilder) Labels() Labels { // Copy the slice, so the next use of ScratchBuilder doesn't overwrite. return append([]Label{}, b.add...) } + +// Write the newly-built Labels out to ls. +// Callers must ensure that there are no other references to ls, or any strings fetched from it. +func (b *ScratchBuilder) Overwrite(ls *Labels) { + *ls = append((*ls)[:0], b.add...) +} diff --git a/model/labels/labels_string.go b/model/labels/labels_string.go index d830d5a435..f293c91675 100644 --- a/model/labels/labels_string.go +++ b/model/labels/labels_string.go @@ -811,7 +811,7 @@ func (b *ScratchBuilder) Labels() Labels { } // Write the newly-built Labels out to ls, reusing an internal buffer. -// Callers must ensure that there are no other references to ls. +// Callers must ensure that there are no other references to ls, or any strings fetched from it. func (b *ScratchBuilder) Overwrite(ls *Labels) { size := labelsSize(b.add) if size <= cap(b.overwriteBuffer) { From c0879d64cf66d7902b838fd73376c30b344055c1 Mon Sep 17 00:00:00 2001 From: beorn7 Date: Fri, 28 Oct 2022 16:58:40 +0200 Subject: [PATCH 085/231] promql: Separate `Point` into `FPoint` and `HPoint` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit In other words: Instead of having a “polymorphous” `Point` that can either contain a float value or a histogram value, use an `FPoint` for floats and an `HPoint` for histograms. This seemingly small change has a _lot_ of repercussions throughout the codebase. The idea here is to avoid the increase in size of `Point` arrays that happened after native histograms had been added. The higher-level data structures (`Sample`, `Series`, etc.) are still “polymorphous”. The same idea could be applied to them, but at each step the trade-offs needed to be evaluated. The idea with this change is to do the minimum necessary to get back to pre-histogram performance for functions that do not touch histograms. Here are comparisons for the `changes` function. The test data doesn't include histograms yet. Ideally, there would be no change in the benchmark result at all. First runtime v2.39 compared to directly prior to this commit: ``` name old time/op new time/op delta RangeQuery/expr=changes(a_one[1d]),steps=1-16 391µs ± 2% 542µs ± 1% +38.58% (p=0.000 n=9+8) RangeQuery/expr=changes(a_one[1d]),steps=10-16 452µs ± 2% 617µs ± 2% +36.48% (p=0.000 n=10+10) RangeQuery/expr=changes(a_one[1d]),steps=100-16 1.12ms ± 1% 1.36ms ± 2% +21.58% (p=0.000 n=8+10) RangeQuery/expr=changes(a_one[1d]),steps=1000-16 7.83ms ± 1% 8.94ms ± 1% +14.21% (p=0.000 n=10+10) RangeQuery/expr=changes(a_ten[1d]),steps=1-16 2.98ms ± 0% 3.30ms ± 1% +10.67% (p=0.000 n=9+10) RangeQuery/expr=changes(a_ten[1d]),steps=10-16 3.66ms ± 1% 4.10ms ± 1% +11.82% (p=0.000 n=10+10) RangeQuery/expr=changes(a_ten[1d]),steps=100-16 10.5ms ± 0% 11.8ms ± 1% +12.50% (p=0.000 n=8+10) RangeQuery/expr=changes(a_ten[1d]),steps=1000-16 77.6ms ± 1% 87.4ms ± 1% +12.63% (p=0.000 n=9+9) RangeQuery/expr=changes(a_hundred[1d]),steps=1-16 30.4ms ± 2% 32.8ms ± 1% +8.01% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=10-16 37.1ms ± 2% 40.6ms ± 2% +9.64% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=100-16 105ms ± 1% 117ms ± 1% +11.69% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=1000-16 783ms ± 3% 876ms ± 1% +11.83% (p=0.000 n=9+10) ``` And then runtime v2.39 compared to after this commit: ``` name old time/op new time/op delta RangeQuery/expr=changes(a_one[1d]),steps=1-16 391µs ± 2% 547µs ± 1% +39.84% (p=0.000 n=9+8) RangeQuery/expr=changes(a_one[1d]),steps=10-16 452µs ± 2% 616µs ± 2% +36.15% (p=0.000 n=10+10) RangeQuery/expr=changes(a_one[1d]),steps=100-16 1.12ms ± 1% 1.26ms ± 1% +12.20% (p=0.000 n=8+10) RangeQuery/expr=changes(a_one[1d]),steps=1000-16 7.83ms ± 1% 7.95ms ± 1% +1.59% (p=0.000 n=10+8) RangeQuery/expr=changes(a_ten[1d]),steps=1-16 2.98ms ± 0% 3.38ms ± 2% +13.49% (p=0.000 n=9+10) RangeQuery/expr=changes(a_ten[1d]),steps=10-16 3.66ms ± 1% 4.02ms ± 1% +9.80% (p=0.000 n=10+9) RangeQuery/expr=changes(a_ten[1d]),steps=100-16 10.5ms ± 0% 10.8ms ± 1% +3.08% (p=0.000 n=8+10) RangeQuery/expr=changes(a_ten[1d]),steps=1000-16 77.6ms ± 1% 78.1ms ± 1% +0.58% (p=0.035 n=9+10) RangeQuery/expr=changes(a_hundred[1d]),steps=1-16 30.4ms ± 2% 33.5ms ± 4% +10.18% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=10-16 37.1ms ± 2% 40.0ms ± 1% +7.98% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=100-16 105ms ± 1% 107ms ± 1% +1.92% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=1000-16 783ms ± 3% 775ms ± 1% -1.02% (p=0.019 n=9+9) ``` In summary, the runtime doesn't really improve with this change for queries with just a few steps. For queries with many steps, this commit essentially reinstates the old performance. This is good because the many-step queries are the one that matter most (longest absolute runtime). In terms of allocations, though, this commit doesn't make a dent at all (numbers not shown). The reason is that most of the allocations happen in the sampleRingIterator (in the storage package), which has to be addressed in a separate commit. Signed-off-by: beorn7 --- cmd/promtool/unittest.go | 5 +- docs/querying/functions.md | 35 ++- promql/engine.go | 392 +++++++++++++++++++----------- promql/engine_test.go | 141 ++++++----- promql/functions.go | 475 ++++++++++++++++++++++--------------- promql/functions_test.go | 2 +- promql/quantile.go | 2 +- promql/test.go | 26 +- promql/test_test.go | 22 +- promql/value.go | 243 +++++++++++-------- rules/alerting.go | 10 +- rules/alerting_test.go | 46 ++-- rules/manager.go | 5 +- rules/manager_test.go | 40 ++-- rules/recording_test.go | 24 +- template/template.go | 2 +- template/template_test.go | 26 +- util/jsonutil/marshal.go | 79 +++++- web/api/v1/api.go | 163 ++++--------- web/api/v1/api_test.go | 48 ++-- web/federate.go | 6 +- web/federate_test.go | 29 ++- 22 files changed, 1062 insertions(+), 759 deletions(-) diff --git a/cmd/promtool/unittest.go b/cmd/promtool/unittest.go index cc40ac9d02..b3e6f67f97 100644 --- a/cmd/promtool/unittest.go +++ b/cmd/promtool/unittest.go @@ -347,7 +347,7 @@ Outer: for _, s := range got { gotSamples = append(gotSamples, parsedSample{ Labels: s.Metric.Copy(), - Value: s.V, + Value: s.F, }) } @@ -447,7 +447,8 @@ func query(ctx context.Context, qs string, t time.Time, engine *promql.Engine, q return v, nil case promql.Scalar: return promql.Vector{promql.Sample{ - Point: promql.Point{T: v.T, V: v.V}, + T: v.T, + F: v.V, Metric: labels.Labels{}, }}, nil default: diff --git a/docs/querying/functions.md b/docs/querying/functions.md index b4bc0a7433..e92a58937d 100644 --- a/docs/querying/functions.md +++ b/docs/querying/functions.md @@ -17,9 +17,7 @@ _Notes about the experimental native histograms:_ flag](../feature_flags/#native-histograms). As long as no native histograms have been ingested into the TSDB, all functions will behave as usual. * Functions that do not explicitly mention native histograms in their - documentation (see below) effectively treat a native histogram as a float - sample of value 0. (This is confusing and will change before native - histograms become a stable feature.) + documentation (see below) will ignore histogram samples. * Functions that do already act on native histograms might still change their behavior in the future. * If a function requires the same bucket layout between multiple native @@ -404,6 +402,8 @@ For each timeseries in `v`, `label_join(v instant-vector, dst_label string, sepa using `separator` and returns the timeseries with the label `dst_label` containing the joined value. There can be any number of `src_labels` in this function. +`label_join` acts on float and histogram samples in the same way. + This example will return a vector with each time series having a `foo` label with the value `a,b,c` added to it: ``` @@ -419,6 +419,8 @@ of `replacement`, together with the original labels in the input. Capturing grou regular expression can be referenced with `$1`, `$2`, etc. If the regular expression doesn't match then the timeseries is returned unchanged. +`label_replace` acts on float and histogram samples in the same way. + This example will return timeseries with the values `a:c` at label `service` and `a` at label `foo`: ``` @@ -501,10 +503,21 @@ counter resets when your target restarts. For each input time series, `resets(v range-vector)` returns the number of counter resets within the provided time range as an instant vector. Any -decrease in the value between two consecutive samples is interpreted as a -counter reset. +decrease in the value between two consecutive float samples is interpreted as a +counter reset. A reset in a native histogram is detected in a more complex way: +Any decrease in any bucket, including the zero bucket, or in the count of +observation constitutes a counter reset, but also the disappearance of any +previously populated bucket, an increase in bucket resolution, or a decrease of +the zero-bucket width. -`resets` should only be used with counters. +`resets` should only be used with counters and counter-like native +histograms. + +If the range vector contains a mix of float and histogram samples for the same +series, counter resets are detected separately and their numbers added up. The +change from a float to a histogram sample is _not_ considered a counter +reset. Each float sample is compared to the next float sample, and each +histogram is comprared to the next histogram. ## `round()` @@ -526,7 +539,7 @@ have exactly one element, `scalar` will return `NaN`. ## `sort()` `sort(v instant-vector)` returns vector elements sorted by their sample values, -in ascending order. +in ascending order. Native histograms are sorted by their sum of observations. ## `sort_desc()` @@ -545,7 +558,8 @@ expression is to be evaluated. ## `timestamp()` `timestamp(v instant-vector)` returns the timestamp of each of the samples of -the given vector as the number of seconds since January 1, 1970 UTC. +the given vector as the number of seconds since January 1, 1970 UTC. It also +works with histogram samples. ## `vector()` @@ -569,12 +583,15 @@ over time and return an instant vector with per-series aggregation results: * `quantile_over_time(scalar, range-vector)`: the φ-quantile (0 ≤ φ ≤ 1) of the values in the specified interval. * `stddev_over_time(range-vector)`: the population standard deviation of the values in the specified interval. * `stdvar_over_time(range-vector)`: the population standard variance of the values in the specified interval. -* `last_over_time(range-vector)`: the most recent point value in specified interval. +* `last_over_time(range-vector)`: the most recent point value in the specified interval. * `present_over_time(range-vector)`: the value 1 for any series in the specified interval. Note that all values in the specified interval have the same weight in the aggregation even if the values are not equally spaced throughout the interval. +`count_over_time`, `last_over_time`, and `present_over_time` handle native +histograms as expected. All other functions ignore histogram samples. + ## Trigonometric Functions The trigonometric functions work in radians: diff --git a/promql/engine.go b/promql/engine.go index 52278c7a24..4c6751088f 100644 --- a/promql/engine.go +++ b/promql/engine.go @@ -189,7 +189,8 @@ func (q *query) Cancel() { // Close implements the Query interface. func (q *query) Close() { for _, s := range q.matrix { - putPointSlice(s.Points) + putFPointSlice(s.Floats) + putHPointSlice(s.Histograms) } } @@ -680,11 +681,15 @@ func (ng *Engine) execEvalStmt(ctx context.Context, query *query, s *parser.Eval for i, s := range mat { // Point might have a different timestamp, force it to the evaluation // timestamp as that is when we ran the evaluation. - vector[i] = Sample{Metric: s.Metric, Point: Point{V: s.Points[0].V, H: s.Points[0].H, T: start}} + if len(s.Histograms) > 0 { + vector[i] = Sample{Metric: s.Metric, H: s.Histograms[0].H, T: start} + } else { + vector[i] = Sample{Metric: s.Metric, F: s.Floats[0].F, T: start} + } } return vector, warnings, nil case parser.ValueTypeScalar: - return Scalar{V: mat[0].Points[0].V, T: start}, warnings, nil + return Scalar{V: mat[0].Floats[0].F, T: start}, warnings, nil case parser.ValueTypeMatrix: return mat, warnings, nil default: @@ -940,9 +945,10 @@ type errWithWarnings struct { func (e errWithWarnings) Error() string { return e.err.Error() } -// An evaluator evaluates given expressions over given fixed timestamps. It -// is attached to an engine through which it connects to a querier and reports -// errors. On timeout or cancellation of its context it terminates. +// An evaluator evaluates the given expressions over the given fixed +// timestamps. It is attached to an engine through which it connects to a +// querier and reports errors. On timeout or cancellation of its context it +// terminates. type evaluator struct { ctx context.Context @@ -1137,17 +1143,35 @@ func (ev *evaluator) rangeEval(prepSeries func(labels.Labels, *EvalSeriesHelper) } for si, series := range matrixes[i] { - for _, point := range series.Points { + for _, point := range series.Floats { if point.T == ts { if ev.currentSamples < ev.maxSamples { - vectors[i] = append(vectors[i], Sample{Metric: series.Metric, Point: point}) + vectors[i] = append(vectors[i], Sample{Metric: series.Metric, F: point.F, T: ts}) if prepSeries != nil { bufHelpers[i] = append(bufHelpers[i], seriesHelpers[i][si]) } // Move input vectors forward so we don't have to re-scan the same // past points at the next step. - matrixes[i][si].Points = series.Points[1:] + matrixes[i][si].Floats = series.Floats[1:] + ev.currentSamples++ + } else { + ev.error(ErrTooManySamples(env)) + } + } + break + } + for _, point := range series.Histograms { + if point.T == ts { + if ev.currentSamples < ev.maxSamples { + vectors[i] = append(vectors[i], Sample{Metric: series.Metric, H: point.H, T: ts}) + if prepSeries != nil { + bufHelpers[i] = append(bufHelpers[i], seriesHelpers[i][si]) + } + + // Move input vectors forward so we don't have to re-scan the same + // past points at the next step. + matrixes[i][si].Histograms = series.Histograms[1:] ev.currentSamples++ } else { ev.error(ErrTooManySamples(env)) @@ -1184,8 +1208,11 @@ func (ev *evaluator) rangeEval(prepSeries func(labels.Labels, *EvalSeriesHelper) if ev.endTimestamp == ev.startTimestamp { mat := make(Matrix, len(result)) for i, s := range result { - s.Point.T = ts - mat[i] = Series{Metric: s.Metric, Points: []Point{s.Point}} + if s.H == nil { + mat[i] = Series{Metric: s.Metric, Floats: []FPoint{{T: ts, F: s.F}}} + } else { + mat[i] = Series{Metric: s.Metric, Histograms: []HPoint{{T: ts, H: s.H}}} + } } ev.currentSamples = originalNumSamples + mat.TotalSamples() ev.samplesStats.UpdatePeak(ev.currentSamples) @@ -1197,22 +1224,28 @@ func (ev *evaluator) rangeEval(prepSeries func(labels.Labels, *EvalSeriesHelper) h := sample.Metric.Hash() ss, ok := seriess[h] if !ok { - ss = Series{ - Metric: sample.Metric, - Points: getPointSlice(numSteps), - } + ss = Series{Metric: sample.Metric} + } + if sample.H == nil { + if ss.Floats == nil { + ss.Floats = getFPointSlice(numSteps) + } + ss.Floats = append(ss.Floats, FPoint{T: ts, F: sample.F}) + } else { + if ss.Histograms == nil { + ss.Histograms = getHPointSlice(numSteps) + } + ss.Histograms = append(ss.Histograms, HPoint{T: ts, H: sample.H}) } - sample.Point.T = ts - ss.Points = append(ss.Points, sample.Point) seriess[h] = ss - } } // Reuse the original point slices. for _, m := range origMatrixes { for _, s := range m { - putPointSlice(s.Points) + putFPointSlice(s.Floats) + putHPointSlice(s.Histograms) } } // Assemble the output matrix. By the time we get here we know we don't have too many samples. @@ -1253,7 +1286,7 @@ func (ev *evaluator) evalSubquery(subq *parser.SubqueryExpr) (*parser.MatrixSele } totalSamples := 0 for _, s := range mat { - totalSamples += len(s.Points) + totalSamples += len(s.Floats) + len(s.Histograms) vs.Series = append(vs.Series, NewStorageSeries(s)) } return ms, totalSamples, ws @@ -1297,7 +1330,7 @@ func (ev *evaluator) eval(expr parser.Expr) (parser.Value, storage.Warnings) { return ev.rangeEval(initSeries, func(v []parser.Value, sh [][]EvalSeriesHelper, enh *EvalNodeHelper) (Vector, storage.Warnings) { var param float64 if e.Param != nil { - param = v[0].(Vector)[0].V + param = v[0].(Vector)[0].F } return ev.aggregation(e.Op, sortedGrouping, e.Without, param, v[1].(Vector), sh[1], enh), nil }, e.Param, e.Expr) @@ -1396,7 +1429,8 @@ func (ev *evaluator) eval(expr parser.Expr) (parser.Value, storage.Warnings) { stepRange = ev.interval } // Reuse objects across steps to save memory allocations. - points := getPointSlice(16) + var floats []FPoint + var histograms []HPoint inMatrix := make(Matrix, 1) inArgs[matrixArgIndex] = inMatrix enh := &EvalNodeHelper{Out: make(Vector, 0, 1)} @@ -1404,8 +1438,13 @@ func (ev *evaluator) eval(expr parser.Expr) (parser.Value, storage.Warnings) { it := storage.NewBuffer(selRange) var chkIter chunkenc.Iterator for i, s := range selVS.Series { - ev.currentSamples -= len(points) - points = points[:0] + ev.currentSamples -= len(floats) + len(histograms) + if floats != nil { + floats = floats[:0] + } + if histograms != nil { + histograms = histograms[:0] + } chkIter = s.Iterator(chkIter) it.Reset(chkIter) metric := selVS.Series[i].Labels() @@ -1418,7 +1457,6 @@ func (ev *evaluator) eval(expr parser.Expr) (parser.Value, storage.Warnings) { } ss := Series{ Metric: metric, - Points: getPointSlice(numSteps), } inMatrix[0].Metric = selVS.Series[i].Labels() for ts, step := ev.startTimestamp, -1; ts <= ev.endTimestamp; ts += ev.interval { @@ -1428,44 +1466,54 @@ func (ev *evaluator) eval(expr parser.Expr) (parser.Value, storage.Warnings) { // when looking up the argument, as there will be no gaps. for j := range e.Args { if j != matrixArgIndex { - otherInArgs[j][0].V = otherArgs[j][0].Points[step].V + otherInArgs[j][0].F = otherArgs[j][0].Floats[step].F } } maxt := ts - offset mint := maxt - selRange // Evaluate the matrix selector for this series for this step. - points = ev.matrixIterSlice(it, mint, maxt, points) - if len(points) == 0 { + floats, histograms = ev.matrixIterSlice(it, mint, maxt, floats, histograms) + if len(floats)+len(histograms) == 0 { continue } - inMatrix[0].Points = points + inMatrix[0].Floats = floats + inMatrix[0].Histograms = histograms enh.Ts = ts // Make the function call. outVec := call(inArgs, e.Args, enh) - ev.samplesStats.IncrementSamplesAtStep(step, int64(len(points))) + ev.samplesStats.IncrementSamplesAtStep(step, int64(len(floats)+len(histograms))) enh.Out = outVec[:0] if len(outVec) > 0 { - ss.Points = append(ss.Points, Point{V: outVec[0].Point.V, H: outVec[0].Point.H, T: ts}) + if outVec[0].H == nil { + if ss.Floats == nil { + ss.Floats = getFPointSlice(numSteps) + } + ss.Floats = append(ss.Floats, FPoint{F: outVec[0].F, T: ts}) + } else { + if ss.Histograms == nil { + ss.Histograms = getHPointSlice(numSteps) + } + ss.Histograms = append(ss.Histograms, HPoint{H: outVec[0].H, T: ts}) + } } // Only buffer stepRange milliseconds from the second step on. it.ReduceDelta(stepRange) } - if len(ss.Points) > 0 { - if ev.currentSamples+len(ss.Points) <= ev.maxSamples { + if len(ss.Floats)+len(ss.Histograms) > 0 { + if ev.currentSamples+len(ss.Floats)+len(ss.Histograms) <= ev.maxSamples { mat = append(mat, ss) - ev.currentSamples += len(ss.Points) + ev.currentSamples += len(ss.Floats) + len(ss.Histograms) } else { ev.error(ErrTooManySamples(env)) } - } else { - putPointSlice(ss.Points) } ev.samplesStats.UpdatePeak(ev.currentSamples) } ev.samplesStats.UpdatePeak(ev.currentSamples) - ev.currentSamples -= len(points) - putPointSlice(points) + ev.currentSamples -= len(floats) + len(histograms) + putFPointSlice(floats) + putHPointSlice(histograms) // The absent_over_time function returns 0 or 1 series. So far, the matrix // contains multiple series. The following code will create a new series @@ -1474,7 +1522,7 @@ func (ev *evaluator) eval(expr parser.Expr) (parser.Value, storage.Warnings) { steps := int(1 + (ev.endTimestamp-ev.startTimestamp)/ev.interval) // Iterate once to look for a complete series. for _, s := range mat { - if len(s.Points) == steps { + if len(s.Floats)+len(s.Histograms) == steps { return Matrix{}, warnings } } @@ -1482,7 +1530,10 @@ func (ev *evaluator) eval(expr parser.Expr) (parser.Value, storage.Warnings) { found := map[int64]struct{}{} for i, s := range mat { - for _, p := range s.Points { + for _, p := range s.Floats { + found[p.T] = struct{}{} + } + for _, p := range s.Histograms { found[p.T] = struct{}{} } if i > 0 && len(found) == steps { @@ -1490,17 +1541,17 @@ func (ev *evaluator) eval(expr parser.Expr) (parser.Value, storage.Warnings) { } } - newp := make([]Point, 0, steps-len(found)) + newp := make([]FPoint, 0, steps-len(found)) for ts := ev.startTimestamp; ts <= ev.endTimestamp; ts += ev.interval { if _, ok := found[ts]; !ok { - newp = append(newp, Point{T: ts, V: 1}) + newp = append(newp, FPoint{T: ts, F: 1}) } } return Matrix{ Series{ Metric: createLabelsForAbsentFunction(e.Args[0]), - Points: newp, + Floats: newp, }, }, warnings } @@ -1520,8 +1571,8 @@ func (ev *evaluator) eval(expr parser.Expr) (parser.Value, storage.Warnings) { if e.Op == parser.SUB { for i := range mat { mat[i].Metric = dropMetricName(mat[i].Metric) - for j := range mat[i].Points { - mat[i].Points[j].V = -mat[i].Points[j].V + for j := range mat[i].Floats { + mat[i].Floats[j].F = -mat[i].Floats[j].F } } if mat.ContainsSameLabelset() { @@ -1534,8 +1585,8 @@ func (ev *evaluator) eval(expr parser.Expr) (parser.Value, storage.Warnings) { switch lt, rt := e.LHS.Type(), e.RHS.Type(); { case lt == parser.ValueTypeScalar && rt == parser.ValueTypeScalar: return ev.rangeEval(nil, func(v []parser.Value, _ [][]EvalSeriesHelper, enh *EvalNodeHelper) (Vector, storage.Warnings) { - val := scalarBinop(e.Op, v[0].(Vector)[0].Point.V, v[1].(Vector)[0].Point.V) - return append(enh.Out, Sample{Point: Point{V: val}}), nil + val := scalarBinop(e.Op, v[0].(Vector)[0].F, v[1].(Vector)[0].F) + return append(enh.Out, Sample{F: val}), nil }, e.LHS, e.RHS) case lt == parser.ValueTypeVector && rt == parser.ValueTypeVector: // Function to compute the join signature for each series. @@ -1565,18 +1616,18 @@ func (ev *evaluator) eval(expr parser.Expr) (parser.Value, storage.Warnings) { case lt == parser.ValueTypeVector && rt == parser.ValueTypeScalar: return ev.rangeEval(nil, func(v []parser.Value, _ [][]EvalSeriesHelper, enh *EvalNodeHelper) (Vector, storage.Warnings) { - return ev.VectorscalarBinop(e.Op, v[0].(Vector), Scalar{V: v[1].(Vector)[0].Point.V}, false, e.ReturnBool, enh), nil + return ev.VectorscalarBinop(e.Op, v[0].(Vector), Scalar{V: v[1].(Vector)[0].F}, false, e.ReturnBool, enh), nil }, e.LHS, e.RHS) case lt == parser.ValueTypeScalar && rt == parser.ValueTypeVector: return ev.rangeEval(nil, func(v []parser.Value, _ [][]EvalSeriesHelper, enh *EvalNodeHelper) (Vector, storage.Warnings) { - return ev.VectorscalarBinop(e.Op, v[1].(Vector), Scalar{V: v[0].(Vector)[0].Point.V}, true, e.ReturnBool, enh), nil + return ev.VectorscalarBinop(e.Op, v[1].(Vector), Scalar{V: v[0].(Vector)[0].F}, true, e.ReturnBool, enh), nil }, e.LHS, e.RHS) } case *parser.NumberLiteral: return ev.rangeEval(nil, func(v []parser.Value, _ [][]EvalSeriesHelper, enh *EvalNodeHelper) (Vector, storage.Warnings) { - return append(enh.Out, Sample{Point: Point{V: e.Val}, Metric: labels.EmptyLabels()}), nil + return append(enh.Out, Sample{F: e.Val, Metric: labels.EmptyLabels()}), nil }) case *parser.StringLiteral: @@ -1595,15 +1646,24 @@ func (ev *evaluator) eval(expr parser.Expr) (parser.Value, storage.Warnings) { it.Reset(chkIter) ss := Series{ Metric: e.Series[i].Labels(), - Points: getPointSlice(numSteps), } for ts, step := ev.startTimestamp, -1; ts <= ev.endTimestamp; ts += ev.interval { step++ - _, v, h, ok := ev.vectorSelectorSingle(it, e, ts) + _, f, h, ok := ev.vectorSelectorSingle(it, e, ts) if ok { if ev.currentSamples < ev.maxSamples { - ss.Points = append(ss.Points, Point{V: v, H: h, T: ts}) + if h == nil { + if ss.Floats == nil { + ss.Floats = getFPointSlice(numSteps) + } + ss.Floats = append(ss.Floats, FPoint{F: f, T: ts}) + } else { + if ss.Histograms == nil { + ss.Histograms = getHPointSlice(numSteps) + } + ss.Histograms = append(ss.Histograms, HPoint{H: h, T: ts}) + } ev.samplesStats.IncrementSamplesAtStep(step, 1) ev.currentSamples++ } else { @@ -1612,10 +1672,8 @@ func (ev *evaluator) eval(expr parser.Expr) (parser.Value, storage.Warnings) { } } - if len(ss.Points) > 0 { + if len(ss.Floats)+len(ss.Histograms) > 0 { mat = append(mat, ss) - } else { - putPointSlice(ss.Points) } } ev.samplesStats.UpdatePeak(ev.currentSamples) @@ -1706,15 +1764,21 @@ func (ev *evaluator) eval(expr parser.Expr) (parser.Value, storage.Warnings) { panic(fmt.Errorf("unexpected result in StepInvariantExpr evaluation: %T", expr)) } for i := range mat { - if len(mat[i].Points) != 1 { + if len(mat[i].Floats)+len(mat[i].Histograms) != 1 { panic(fmt.Errorf("unexpected number of samples")) } for ts := ev.startTimestamp + ev.interval; ts <= ev.endTimestamp; ts = ts + ev.interval { - mat[i].Points = append(mat[i].Points, Point{ - T: ts, - V: mat[i].Points[0].V, - H: mat[i].Points[0].H, - }) + if len(mat[i].Floats) > 0 { + mat[i].Floats = append(mat[i].Floats, FPoint{ + T: ts, + F: mat[i].Floats[0].F, + }) + } else { + mat[i].Histograms = append(mat[i].Histograms, HPoint{ + T: ts, + H: mat[i].Histograms[0].H, + }) + } ev.currentSamples++ if ev.currentSamples > ev.maxSamples { ev.error(ErrTooManySamples(env)) @@ -1741,11 +1805,13 @@ func (ev *evaluator) vectorSelector(node *parser.VectorSelector, ts int64) (Vect chkIter = s.Iterator(chkIter) it.Reset(chkIter) - t, v, h, ok := ev.vectorSelectorSingle(it, node, ts) + t, f, h, ok := ev.vectorSelectorSingle(it, node, ts) if ok { vec = append(vec, Sample{ Metric: node.Series[i].Labels(), - Point: Point{V: v, H: h, T: t}, + T: t, + F: f, + H: h, }) ev.currentSamples++ @@ -1795,17 +1861,31 @@ func (ev *evaluator) vectorSelectorSingle(it *storage.MemoizedSeriesIterator, no return t, v, h, true } -var pointPool zeropool.Pool[[]Point] +var ( + fPointPool zeropool.Pool[[]FPoint] + hPointPool zeropool.Pool[[]HPoint] +) -func getPointSlice(sz int) []Point { - if p := pointPool.Get(); p != nil { +func getFPointSlice(sz int) []FPoint { + if p := fPointPool.Get(); p != nil { return p } - return make([]Point, 0, sz) + return make([]FPoint, 0, sz) } -func putPointSlice(p []Point) { - pointPool.Put(p[:0]) +func putFPointSlice(p []FPoint) { + fPointPool.Put(p[:0]) +} + +func getHPointSlice(sz int) []HPoint { + if p := hPointPool.Get(); p != nil { + return p + } + return make([]HPoint, 0, sz) +} + +func putHPointSlice(p []HPoint) { + hPointPool.Put(p[:0]) } // matrixSelector evaluates a *parser.MatrixSelector expression. @@ -1837,13 +1917,15 @@ func (ev *evaluator) matrixSelector(node *parser.MatrixSelector) (Matrix, storag Metric: series[i].Labels(), } - ss.Points = ev.matrixIterSlice(it, mint, maxt, getPointSlice(16)) - ev.samplesStats.IncrementSamplesAtTimestamp(ev.startTimestamp, int64(len(ss.Points))) + ss.Floats, ss.Histograms = ev.matrixIterSlice(it, mint, maxt, nil, nil) + totalLen := int64(len(ss.Floats)) + int64(len(ss.Histograms)) + ev.samplesStats.IncrementSamplesAtTimestamp(ev.startTimestamp, totalLen) - if len(ss.Points) > 0 { + if totalLen > 0 { matrix = append(matrix, ss) } else { - putPointSlice(ss.Points) + putFPointSlice(ss.Floats) + putHPointSlice(ss.Histograms) } } return matrix, ws @@ -1857,24 +1939,54 @@ func (ev *evaluator) matrixSelector(node *parser.MatrixSelector) (Matrix, storag // values). Any such points falling before mint are discarded; points that fall // into the [mint, maxt] range are retained; only points with later timestamps // are populated from the iterator. -func (ev *evaluator) matrixIterSlice(it *storage.BufferedSeriesIterator, mint, maxt int64, out []Point) []Point { - if len(out) > 0 && out[len(out)-1].T >= mint { +func (ev *evaluator) matrixIterSlice( + it *storage.BufferedSeriesIterator, mint, maxt int64, + floats []FPoint, histograms []HPoint, +) ([]FPoint, []HPoint) { + mintFloats, mintHistograms := mint, mint + + // First floats... + if len(floats) > 0 && floats[len(floats)-1].T >= mint { // There is an overlap between previous and current ranges, retain common // points. In most such cases: // (a) the overlap is significantly larger than the eval step; and/or // (b) the number of samples is relatively small. // so a linear search will be as fast as a binary search. var drop int - for drop = 0; out[drop].T < mint; drop++ { + for drop = 0; floats[drop].T < mint; drop++ { } ev.currentSamples -= drop - copy(out, out[drop:]) - out = out[:len(out)-drop] + copy(floats, floats[drop:]) + floats = floats[:len(floats)-drop] // Only append points with timestamps after the last timestamp we have. - mint = out[len(out)-1].T + 1 + mintFloats = floats[len(floats)-1].T + 1 } else { - ev.currentSamples -= len(out) - out = out[:0] + ev.currentSamples -= len(floats) + if floats != nil { + floats = floats[:0] + } + } + + // ...then the same for histograms. TODO(beorn7): Use generics? + if len(histograms) > 0 && histograms[len(histograms)-1].T >= mint { + // There is an overlap between previous and current ranges, retain common + // points. In most such cases: + // (a) the overlap is significantly larger than the eval step; and/or + // (b) the number of samples is relatively small. + // so a linear search will be as fast as a binary search. + var drop int + for drop = 0; histograms[drop].T < mint; drop++ { + } + ev.currentSamples -= drop + copy(histograms, histograms[drop:]) + histograms = histograms[:len(histograms)-drop] + // Only append points with timestamps after the last timestamp we have. + mintHistograms = histograms[len(histograms)-1].T + 1 + } else { + ev.currentSamples -= len(histograms) + if histograms != nil { + histograms = histograms[:0] + } } soughtValueType := it.Seek(maxt) @@ -1896,25 +2008,31 @@ loop: continue loop } // Values in the buffer are guaranteed to be smaller than maxt. - if t >= mint { + if t >= mintHistograms { if ev.currentSamples >= ev.maxSamples { ev.error(ErrTooManySamples(env)) } ev.currentSamples++ - out = append(out, Point{T: t, H: h}) + if histograms == nil { + histograms = getHPointSlice(16) + } + histograms = append(histograms, HPoint{T: t, H: h}) } case chunkenc.ValFloat: - t, v := buf.At() - if value.IsStaleNaN(v) { + t, f := buf.At() + if value.IsStaleNaN(f) { continue loop } // Values in the buffer are guaranteed to be smaller than maxt. - if t >= mint { + if t >= mintFloats { if ev.currentSamples >= ev.maxSamples { ev.error(ErrTooManySamples(env)) } ev.currentSamples++ - out = append(out, Point{T: t, V: v}) + if floats == nil { + floats = getFPointSlice(16) + } + floats = append(floats, FPoint{T: t, F: f}) } } } @@ -1926,21 +2044,27 @@ loop: if ev.currentSamples >= ev.maxSamples { ev.error(ErrTooManySamples(env)) } - out = append(out, Point{T: t, H: h}) + if histograms == nil { + histograms = getHPointSlice(16) + } + histograms = append(histograms, HPoint{T: t, H: h}) ev.currentSamples++ } case chunkenc.ValFloat: - t, v := it.At() - if t == maxt && !value.IsStaleNaN(v) { + t, f := it.At() + if t == maxt && !value.IsStaleNaN(f) { if ev.currentSamples >= ev.maxSamples { ev.error(ErrTooManySamples(env)) } - out = append(out, Point{T: t, V: v}) + if floats == nil { + floats = getFPointSlice(16) + } + floats = append(floats, FPoint{T: t, F: f}) ev.currentSamples++ } } ev.samplesStats.UpdatePeak(ev.currentSamples) - return out + return floats, histograms } func (ev *evaluator) VectorAnd(lhs, rhs Vector, matching *parser.VectorMatching, lhsh, rhsh []EvalSeriesHelper, enh *EvalNodeHelper) Vector { @@ -2086,18 +2210,18 @@ func (ev *evaluator) VectorBinop(op parser.ItemType, lhs, rhs Vector, matching * } // Account for potentially swapped sidedness. - vl, vr := ls.V, rs.V + fl, fr := ls.F, rs.F hl, hr := ls.H, rs.H if matching.Card == parser.CardOneToMany { - vl, vr = vr, vl + fl, fr = fr, fl hl, hr = hr, hl } - value, histogramValue, keep := vectorElemBinop(op, vl, vr, hl, hr) + floatValue, histogramValue, keep := vectorElemBinop(op, fl, fr, hl, hr) if returnBool { if keep { - value = 1.0 + floatValue = 1.0 } else { - value = 0.0 + floatValue = 0.0 } } else if !keep { continue @@ -2131,7 +2255,8 @@ func (ev *evaluator) VectorBinop(op parser.ItemType, lhs, rhs Vector, matching * // Both lhs and rhs are of same type. enh.Out = append(enh.Out, Sample{ Metric: metric, - Point: Point{V: value, H: histogramValue}, + F: floatValue, + H: histogramValue, }) } } @@ -2200,7 +2325,7 @@ func resultMetric(lhs, rhs labels.Labels, op parser.ItemType, matching *parser.V // VectorscalarBinop evaluates a binary operation between a Vector and a Scalar. func (ev *evaluator) VectorscalarBinop(op parser.ItemType, lhs Vector, rhs Scalar, swap, returnBool bool, enh *EvalNodeHelper) Vector { for _, lhsSample := range lhs { - lv, rv := lhsSample.V, rhs.V + lv, rv := lhsSample.F, rhs.V // lhs always contains the Vector. If the original position was different // swap for calculating the value. if swap { @@ -2221,7 +2346,7 @@ func (ev *evaluator) VectorscalarBinop(op parser.ItemType, lhs Vector, rhs Scala keep = true } if keep { - lhsSample.V = value + lhsSample.F = value if shouldDropMetricName(op) || returnBool { lhsSample.Metric = enh.DropMetricName(lhsSample.Metric) } @@ -2313,7 +2438,7 @@ type groupedAggregation struct { hasFloat bool // Has at least 1 float64 sample aggregated. hasHistogram bool // Has at least 1 histogram sample aggregated. labels labels.Labels - value float64 + floatValue float64 histogramValue *histogram.FloatHistogram mean float64 groupCount int @@ -2365,7 +2490,7 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without if op == parser.COUNT_VALUES { enh.resetBuilder(metric) - enh.lb.Set(valueLabel, strconv.FormatFloat(s.V, 'f', -1, 64)) + enh.lb.Set(valueLabel, strconv.FormatFloat(s.F, 'f', -1, 64)) metric = enh.lb.Labels() // We've changed the metric so we have to recompute the grouping key. @@ -2397,8 +2522,8 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without } newAgg := &groupedAggregation{ labels: m, - value: s.V, - mean: s.V, + floatValue: s.F, + mean: s.F, groupCount: 1, } if s.H == nil { @@ -2420,21 +2545,21 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without } switch op { case parser.STDVAR, parser.STDDEV: - result[groupingKey].value = 0 + result[groupingKey].floatValue = 0 case parser.TOPK, parser.QUANTILE: result[groupingKey].heap = make(vectorByValueHeap, 1, resultSize) result[groupingKey].heap[0] = Sample{ - Point: Point{V: s.V}, + F: s.F, Metric: s.Metric, } case parser.BOTTOMK: result[groupingKey].reverseHeap = make(vectorByReverseValueHeap, 1, resultSize) result[groupingKey].reverseHeap[0] = Sample{ - Point: Point{V: s.V}, + F: s.F, Metric: s.Metric, } case parser.GROUP: - result[groupingKey].value = 1 + result[groupingKey].floatValue = 1 } continue } @@ -2459,19 +2584,19 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without // point in copying the histogram in that case. } else { group.hasFloat = true - group.value += s.V + group.floatValue += s.F } case parser.AVG: group.groupCount++ if math.IsInf(group.mean, 0) { - if math.IsInf(s.V, 0) && (group.mean > 0) == (s.V > 0) { + if math.IsInf(s.F, 0) && (group.mean > 0) == (s.F > 0) { // The `mean` and `s.V` values are `Inf` of the same sign. They // can't be subtracted, but the value of `mean` is correct // already. break } - if !math.IsInf(s.V, 0) && !math.IsNaN(s.V) { + if !math.IsInf(s.F, 0) && !math.IsNaN(s.F) { // At this stage, the mean is an infinite. If the added // value is neither an Inf or a Nan, we can keep that mean // value. @@ -2482,19 +2607,19 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without } } // Divide each side of the `-` by `group.groupCount` to avoid float64 overflows. - group.mean += s.V/float64(group.groupCount) - group.mean/float64(group.groupCount) + group.mean += s.F/float64(group.groupCount) - group.mean/float64(group.groupCount) case parser.GROUP: // Do nothing. Required to avoid the panic in `default:` below. case parser.MAX: - if group.value < s.V || math.IsNaN(group.value) { - group.value = s.V + if group.floatValue < s.F || math.IsNaN(group.floatValue) { + group.floatValue = s.F } case parser.MIN: - if group.value > s.V || math.IsNaN(group.value) { - group.value = s.V + if group.floatValue > s.F || math.IsNaN(group.floatValue) { + group.floatValue = s.F } case parser.COUNT, parser.COUNT_VALUES: @@ -2502,21 +2627,21 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without case parser.STDVAR, parser.STDDEV: group.groupCount++ - delta := s.V - group.mean + delta := s.F - group.mean group.mean += delta / float64(group.groupCount) - group.value += delta * (s.V - group.mean) + group.floatValue += delta * (s.F - group.mean) case parser.TOPK: // We build a heap of up to k elements, with the smallest element at heap[0]. if int64(len(group.heap)) < k { heap.Push(&group.heap, &Sample{ - Point: Point{V: s.V}, + F: s.F, Metric: s.Metric, }) - } else if group.heap[0].V < s.V || (math.IsNaN(group.heap[0].V) && !math.IsNaN(s.V)) { + } else if group.heap[0].F < s.F || (math.IsNaN(group.heap[0].F) && !math.IsNaN(s.F)) { // This new element is bigger than the previous smallest element - overwrite that. group.heap[0] = Sample{ - Point: Point{V: s.V}, + F: s.F, Metric: s.Metric, } if k > 1 { @@ -2528,13 +2653,13 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without // We build a heap of up to k elements, with the biggest element at heap[0]. if int64(len(group.reverseHeap)) < k { heap.Push(&group.reverseHeap, &Sample{ - Point: Point{V: s.V}, + F: s.F, Metric: s.Metric, }) - } else if group.reverseHeap[0].V > s.V || (math.IsNaN(group.reverseHeap[0].V) && !math.IsNaN(s.V)) { + } else if group.reverseHeap[0].F > s.F || (math.IsNaN(group.reverseHeap[0].F) && !math.IsNaN(s.F)) { // This new element is smaller than the previous biggest element - overwrite that. group.reverseHeap[0] = Sample{ - Point: Point{V: s.V}, + F: s.F, Metric: s.Metric, } if k > 1 { @@ -2554,16 +2679,16 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without for _, aggr := range orderedResult { switch op { case parser.AVG: - aggr.value = aggr.mean + aggr.floatValue = aggr.mean case parser.COUNT, parser.COUNT_VALUES: - aggr.value = float64(aggr.groupCount) + aggr.floatValue = float64(aggr.groupCount) case parser.STDVAR: - aggr.value = aggr.value / float64(aggr.groupCount) + aggr.floatValue = aggr.floatValue / float64(aggr.groupCount) case parser.STDDEV: - aggr.value = math.Sqrt(aggr.value / float64(aggr.groupCount)) + aggr.floatValue = math.Sqrt(aggr.floatValue / float64(aggr.groupCount)) case parser.TOPK: // The heap keeps the lowest value on top, so reverse it. @@ -2573,7 +2698,7 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without for _, v := range aggr.heap { enh.Out = append(enh.Out, Sample{ Metric: v.Metric, - Point: Point{V: v.V}, + F: v.F, }) } continue // Bypass default append. @@ -2586,13 +2711,13 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without for _, v := range aggr.reverseHeap { enh.Out = append(enh.Out, Sample{ Metric: v.Metric, - Point: Point{V: v.V}, + F: v.F, }) } continue // Bypass default append. case parser.QUANTILE: - aggr.value = quantile(q, aggr.heap) + aggr.floatValue = quantile(q, aggr.heap) case parser.SUM: if aggr.hasFloat && aggr.hasHistogram { @@ -2605,7 +2730,8 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without enh.Out = append(enh.Out, Sample{ Metric: aggr.labels, - Point: Point{V: aggr.value, H: aggr.histogramValue}, + F: aggr.floatValue, + H: aggr.histogramValue, }) } return enh.Out diff --git a/promql/engine_test.go b/promql/engine_test.go index d1c4570ea2..b64e32ba46 100644 --- a/promql/engine_test.go +++ b/promql/engine_test.go @@ -662,7 +662,8 @@ load 10s Query: "metric", Result: Vector{ Sample{ - Point: Point{V: 1, T: 1000}, + F: 1, + T: 1000, Metric: labels.FromStrings("__name__", "metric"), }, }, @@ -672,7 +673,7 @@ load 10s Query: "metric[20s]", Result: Matrix{ Series{ - Points: []Point{{V: 1, T: 0}, {V: 2, T: 10000}}, + Floats: []FPoint{{F: 1, T: 0}, {F: 2, T: 10000}}, Metric: labels.FromStrings("__name__", "metric"), }, }, @@ -683,7 +684,7 @@ load 10s Query: "1", Result: Matrix{ Series{ - Points: []Point{{V: 1, T: 0}, {V: 1, T: 1000}, {V: 1, T: 2000}}, + Floats: []FPoint{{F: 1, T: 0}, {F: 1, T: 1000}, {F: 1, T: 2000}}, Metric: labels.EmptyLabels(), }, }, @@ -695,7 +696,7 @@ load 10s Query: "metric", Result: Matrix{ Series{ - Points: []Point{{V: 1, T: 0}, {V: 1, T: 1000}, {V: 1, T: 2000}}, + Floats: []FPoint{{F: 1, T: 0}, {F: 1, T: 1000}, {F: 1, T: 2000}}, Metric: labels.FromStrings("__name__", "metric"), }, }, @@ -707,7 +708,7 @@ load 10s Query: "metric", Result: Matrix{ Series{ - Points: []Point{{V: 1, T: 0}, {V: 1, T: 5000}, {V: 2, T: 10000}}, + Floats: []FPoint{{F: 1, T: 0}, {F: 1, T: 5000}, {F: 2, T: 10000}}, Metric: labels.FromStrings("__name__", "metric"), }, }, @@ -1462,20 +1463,20 @@ load 1ms query: `metric_neg @ 0`, start: 100, result: Vector{ - Sample{Point: Point{V: 1, T: 100000}, Metric: lblsneg}, + Sample{F: 1, T: 100000, Metric: lblsneg}, }, }, { query: `metric_neg @ -200`, start: 100, result: Vector{ - Sample{Point: Point{V: 201, T: 100000}, Metric: lblsneg}, + Sample{F: 201, T: 100000, Metric: lblsneg}, }, }, { query: `metric{job="2"} @ 50`, start: -2, end: 2, interval: 1, result: Matrix{ Series{ - Points: []Point{{V: 10, T: -2000}, {V: 10, T: -1000}, {V: 10, T: 0}, {V: 10, T: 1000}, {V: 10, T: 2000}}, + Floats: []FPoint{{F: 10, T: -2000}, {F: 10, T: -1000}, {F: 10, T: 0}, {F: 10, T: 1000}, {F: 10, T: 2000}}, Metric: lbls2, }, }, @@ -1484,11 +1485,11 @@ load 1ms start: 10, result: Matrix{ Series{ - Points: []Point{{V: 28, T: 280000}, {V: 29, T: 290000}, {V: 30, T: 300000}}, + Floats: []FPoint{{F: 28, T: 280000}, {F: 29, T: 290000}, {F: 30, T: 300000}}, Metric: lbls1, }, Series{ - Points: []Point{{V: 56, T: 280000}, {V: 58, T: 290000}, {V: 60, T: 300000}}, + Floats: []FPoint{{F: 56, T: 280000}, {F: 58, T: 290000}, {F: 60, T: 300000}}, Metric: lbls2, }, }, @@ -1497,7 +1498,7 @@ load 1ms start: 100, result: Matrix{ Series{ - Points: []Point{{V: 3, T: -2000}, {V: 2, T: -1000}, {V: 1, T: 0}}, + Floats: []FPoint{{F: 3, T: -2000}, {F: 2, T: -1000}, {F: 1, T: 0}}, Metric: lblsneg, }, }, @@ -1506,7 +1507,7 @@ load 1ms start: 100, result: Matrix{ Series{ - Points: []Point{{V: 504, T: -503000}, {V: 503, T: -502000}, {V: 502, T: -501000}, {V: 501, T: -500000}}, + Floats: []FPoint{{F: 504, T: -503000}, {F: 503, T: -502000}, {F: 502, T: -501000}, {F: 501, T: -500000}}, Metric: lblsneg, }, }, @@ -1515,7 +1516,7 @@ load 1ms start: 100, result: Matrix{ Series{ - Points: []Point{{V: 2342, T: 2342}, {V: 2343, T: 2343}, {V: 2344, T: 2344}, {V: 2345, T: 2345}}, + Floats: []FPoint{{F: 2342, T: 2342}, {F: 2343, T: 2343}, {F: 2344, T: 2344}, {F: 2345, T: 2345}}, Metric: lblsms, }, }, @@ -1524,11 +1525,11 @@ load 1ms start: 100, result: Matrix{ Series{ - Points: []Point{{V: 20, T: 200000}, {V: 22, T: 225000}, {V: 25, T: 250000}, {V: 27, T: 275000}, {V: 30, T: 300000}}, + Floats: []FPoint{{F: 20, T: 200000}, {F: 22, T: 225000}, {F: 25, T: 250000}, {F: 27, T: 275000}, {F: 30, T: 300000}}, Metric: lbls1, }, Series{ - Points: []Point{{V: 40, T: 200000}, {V: 44, T: 225000}, {V: 50, T: 250000}, {V: 54, T: 275000}, {V: 60, T: 300000}}, + Floats: []FPoint{{F: 40, T: 200000}, {F: 44, T: 225000}, {F: 50, T: 250000}, {F: 54, T: 275000}, {F: 60, T: 300000}}, Metric: lbls2, }, }, @@ -1537,7 +1538,7 @@ load 1ms start: 100, result: Matrix{ Series{ - Points: []Point{{V: 51, T: -50000}, {V: 26, T: -25000}, {V: 1, T: 0}}, + Floats: []FPoint{{F: 51, T: -50000}, {F: 26, T: -25000}, {F: 1, T: 0}}, Metric: lblsneg, }, }, @@ -1546,7 +1547,7 @@ load 1ms start: 100, result: Matrix{ Series{ - Points: []Point{{V: 151, T: -150000}, {V: 126, T: -125000}, {V: 101, T: -100000}}, + Floats: []FPoint{{F: 151, T: -150000}, {F: 126, T: -125000}, {F: 101, T: -100000}}, Metric: lblsneg, }, }, @@ -1555,7 +1556,7 @@ load 1ms start: 100, result: Matrix{ Series{ - Points: []Point{{V: 2250, T: 2250}, {V: 2275, T: 2275}, {V: 2300, T: 2300}, {V: 2325, T: 2325}}, + Floats: []FPoint{{F: 2250, T: 2250}, {F: 2275, T: 2275}, {F: 2300, T: 2300}, {F: 2325, T: 2325}}, Metric: lblsms, }, }, @@ -1564,7 +1565,7 @@ load 1ms start: 50, end: 80, interval: 10, result: Matrix{ Series{ - Points: []Point{{V: 995, T: 50000}, {V: 994, T: 60000}, {V: 993, T: 70000}, {V: 992, T: 80000}}, + Floats: []FPoint{{F: 995, T: 50000}, {F: 994, T: 60000}, {F: 993, T: 70000}, {F: 992, T: 80000}}, Metric: lblstopk3, }, }, @@ -1573,7 +1574,7 @@ load 1ms start: 50, end: 80, interval: 10, result: Matrix{ Series{ - Points: []Point{{V: 10, T: 50000}, {V: 12, T: 60000}, {V: 14, T: 70000}, {V: 16, T: 80000}}, + Floats: []FPoint{{F: 10, T: 50000}, {F: 12, T: 60000}, {F: 14, T: 70000}, {F: 16, T: 80000}}, Metric: lblstopk2, }, }, @@ -1582,7 +1583,7 @@ load 1ms start: 70, end: 100, interval: 10, result: Matrix{ Series{ - Points: []Point{{V: 993, T: 70000}, {V: 992, T: 80000}, {V: 991, T: 90000}, {V: 990, T: 100000}}, + Floats: []FPoint{{F: 993, T: 70000}, {F: 992, T: 80000}, {F: 991, T: 90000}, {F: 990, T: 100000}}, Metric: lblstopk3, }, }, @@ -1591,7 +1592,7 @@ load 1ms start: 100, end: 130, interval: 10, result: Matrix{ Series{ - Points: []Point{{V: 990, T: 100000}, {V: 989, T: 110000}, {V: 988, T: 120000}, {V: 987, T: 130000}}, + Floats: []FPoint{{F: 990, T: 100000}, {F: 989, T: 110000}, {F: 988, T: 120000}, {F: 987, T: 130000}}, Metric: lblstopk3, }, }, @@ -1602,15 +1603,15 @@ load 1ms start: 0, end: 7 * 60, interval: 60, result: Matrix{ Series{ - Points: []Point{ - {V: 3600, T: 0}, - {V: 3600, T: 60 * 1000}, - {V: 3600, T: 2 * 60 * 1000}, - {V: 3600, T: 3 * 60 * 1000}, - {V: 3600, T: 4 * 60 * 1000}, - {V: 3600, T: 5 * 60 * 1000}, - {V: 3600, T: 6 * 60 * 1000}, - {V: 3600, T: 7 * 60 * 1000}, + Floats: []FPoint{ + {F: 3600, T: 0}, + {F: 3600, T: 60 * 1000}, + {F: 3600, T: 2 * 60 * 1000}, + {F: 3600, T: 3 * 60 * 1000}, + {F: 3600, T: 4 * 60 * 1000}, + {F: 3600, T: 5 * 60 * 1000}, + {F: 3600, T: 6 * 60 * 1000}, + {F: 3600, T: 7 * 60 * 1000}, }, Metric: labels.EmptyLabels(), }, @@ -1723,7 +1724,7 @@ func TestSubquerySelector(t *testing.T) { nil, Matrix{ Series{ - Points: []Point{{V: 1, T: 0}, {V: 2, T: 10000}}, + Floats: []FPoint{{F: 1, T: 0}, {F: 2, T: 10000}}, Metric: labels.FromStrings("__name__", "metric"), }, }, @@ -1737,7 +1738,7 @@ func TestSubquerySelector(t *testing.T) { nil, Matrix{ Series{ - Points: []Point{{V: 1, T: 0}, {V: 1, T: 5000}, {V: 2, T: 10000}}, + Floats: []FPoint{{F: 1, T: 0}, {F: 1, T: 5000}, {F: 2, T: 10000}}, Metric: labels.FromStrings("__name__", "metric"), }, }, @@ -1751,7 +1752,7 @@ func TestSubquerySelector(t *testing.T) { nil, Matrix{ Series{ - Points: []Point{{V: 1, T: 0}, {V: 1, T: 5000}, {V: 2, T: 10000}}, + Floats: []FPoint{{F: 1, T: 0}, {F: 1, T: 5000}, {F: 2, T: 10000}}, Metric: labels.FromStrings("__name__", "metric"), }, }, @@ -1765,7 +1766,7 @@ func TestSubquerySelector(t *testing.T) { nil, Matrix{ Series{ - Points: []Point{{V: 1, T: 0}, {V: 1, T: 5000}, {V: 2, T: 10000}}, + Floats: []FPoint{{F: 1, T: 0}, {F: 1, T: 5000}, {F: 2, T: 10000}}, Metric: labels.FromStrings("__name__", "metric"), }, }, @@ -1779,7 +1780,7 @@ func TestSubquerySelector(t *testing.T) { nil, Matrix{ Series{ - Points: []Point{{V: 2, T: 15000}, {V: 2, T: 20000}, {V: 2, T: 25000}, {V: 2, T: 30000}}, + Floats: []FPoint{{F: 2, T: 15000}, {F: 2, T: 20000}, {F: 2, T: 25000}, {F: 2, T: 30000}}, Metric: labels.FromStrings("__name__", "metric"), }, }, @@ -1793,7 +1794,7 @@ func TestSubquerySelector(t *testing.T) { nil, Matrix{ Series{ - Points: []Point{{V: 2, T: 10000}, {V: 2, T: 15000}, {V: 2, T: 20000}, {V: 2, T: 25000}, {V: 2, T: 30000}}, + Floats: []FPoint{{F: 2, T: 10000}, {F: 2, T: 15000}, {F: 2, T: 20000}, {F: 2, T: 25000}, {F: 2, T: 30000}}, Metric: labels.FromStrings("__name__", "metric"), }, }, @@ -1807,7 +1808,7 @@ func TestSubquerySelector(t *testing.T) { nil, Matrix{ Series{ - Points: []Point{{V: 2, T: 10000}, {V: 2, T: 15000}, {V: 2, T: 20000}, {V: 2, T: 25000}}, + Floats: []FPoint{{F: 2, T: 10000}, {F: 2, T: 15000}, {F: 2, T: 20000}, {F: 2, T: 25000}}, Metric: labels.FromStrings("__name__", "metric"), }, }, @@ -1821,7 +1822,7 @@ func TestSubquerySelector(t *testing.T) { nil, Matrix{ Series{ - Points: []Point{{V: 2, T: 10000}, {V: 2, T: 15000}, {V: 2, T: 20000}, {V: 2, T: 25000}}, + Floats: []FPoint{{F: 2, T: 10000}, {F: 2, T: 15000}, {F: 2, T: 20000}, {F: 2, T: 25000}}, Metric: labels.FromStrings("__name__", "metric"), }, }, @@ -1844,7 +1845,7 @@ func TestSubquerySelector(t *testing.T) { nil, Matrix{ Series{ - Points: []Point{{V: 9990, T: 9990000}, {V: 10000, T: 10000000}, {V: 100, T: 10010000}, {V: 130, T: 10020000}}, + Floats: []FPoint{{F: 9990, T: 9990000}, {F: 10000, T: 10000000}, {F: 100, T: 10010000}, {F: 130, T: 10020000}}, Metric: labels.FromStrings("__name__", "http_requests", "job", "api-server", "instance", "0", "group", "production"), }, }, @@ -1858,7 +1859,7 @@ func TestSubquerySelector(t *testing.T) { nil, Matrix{ Series{ - Points: []Point{{V: 9840, T: 9840000}, {V: 9900, T: 9900000}, {V: 9960, T: 9960000}, {V: 130, T: 10020000}, {V: 310, T: 10080000}}, + Floats: []FPoint{{F: 9840, T: 9840000}, {F: 9900, T: 9900000}, {F: 9960, T: 9960000}, {F: 130, T: 10020000}, {F: 310, T: 10080000}}, Metric: labels.FromStrings("__name__", "http_requests", "job", "api-server", "instance", "0", "group", "production"), }, }, @@ -1872,7 +1873,7 @@ func TestSubquerySelector(t *testing.T) { nil, Matrix{ Series{ - Points: []Point{{V: 8640, T: 8640000}, {V: 8700, T: 8700000}, {V: 8760, T: 8760000}, {V: 8820, T: 8820000}, {V: 8880, T: 8880000}}, + Floats: []FPoint{{F: 8640, T: 8640000}, {F: 8700, T: 8700000}, {F: 8760, T: 8760000}, {F: 8820, T: 8820000}, {F: 8880, T: 8880000}}, Metric: labels.FromStrings("__name__", "http_requests", "job", "api-server", "instance", "0", "group", "production"), }, }, @@ -1886,19 +1887,19 @@ func TestSubquerySelector(t *testing.T) { nil, Matrix{ Series{ - Points: []Point{{V: 3, T: 7985000}, {V: 3, T: 7990000}, {V: 3, T: 7995000}, {V: 3, T: 8000000}}, + Floats: []FPoint{{F: 3, T: 7985000}, {F: 3, T: 7990000}, {F: 3, T: 7995000}, {F: 3, T: 8000000}}, Metric: labels.FromStrings("job", "api-server", "instance", "0", "group", "canary"), }, Series{ - Points: []Point{{V: 4, T: 7985000}, {V: 4, T: 7990000}, {V: 4, T: 7995000}, {V: 4, T: 8000000}}, + Floats: []FPoint{{F: 4, T: 7985000}, {F: 4, T: 7990000}, {F: 4, T: 7995000}, {F: 4, T: 8000000}}, Metric: labels.FromStrings("job", "api-server", "instance", "1", "group", "canary"), }, Series{ - Points: []Point{{V: 1, T: 7985000}, {V: 1, T: 7990000}, {V: 1, T: 7995000}, {V: 1, T: 8000000}}, + Floats: []FPoint{{F: 1, T: 7985000}, {F: 1, T: 7990000}, {F: 1, T: 7995000}, {F: 1, T: 8000000}}, Metric: labels.FromStrings("job", "api-server", "instance", "0", "group", "production"), }, Series{ - Points: []Point{{V: 2, T: 7985000}, {V: 2, T: 7990000}, {V: 2, T: 7995000}, {V: 2, T: 8000000}}, + Floats: []FPoint{{F: 2, T: 7985000}, {F: 2, T: 7990000}, {F: 2, T: 7995000}, {F: 2, T: 8000000}}, Metric: labels.FromStrings("job", "api-server", "instance", "1", "group", "production"), }, }, @@ -1912,7 +1913,7 @@ func TestSubquerySelector(t *testing.T) { nil, Matrix{ Series{ - Points: []Point{{V: 270, T: 90000}, {V: 300, T: 100000}, {V: 330, T: 110000}, {V: 360, T: 120000}}, + Floats: []FPoint{{F: 270, T: 90000}, {F: 300, T: 100000}, {F: 330, T: 110000}, {F: 360, T: 120000}}, Metric: labels.EmptyLabels(), }, }, @@ -1926,7 +1927,7 @@ func TestSubquerySelector(t *testing.T) { nil, Matrix{ Series{ - Points: []Point{{V: 800, T: 80000}, {V: 900, T: 90000}, {V: 1000, T: 100000}, {V: 1100, T: 110000}, {V: 1200, T: 120000}}, + Floats: []FPoint{{F: 800, T: 80000}, {F: 900, T: 90000}, {F: 1000, T: 100000}, {F: 1100, T: 110000}, {F: 1200, T: 120000}}, Metric: labels.EmptyLabels(), }, }, @@ -1940,7 +1941,7 @@ func TestSubquerySelector(t *testing.T) { nil, Matrix{ Series{ - Points: []Point{{V: 1000, T: 100000}, {V: 1000, T: 105000}, {V: 1100, T: 110000}, {V: 1100, T: 115000}, {V: 1200, T: 120000}}, + Floats: []FPoint{{F: 1000, T: 100000}, {F: 1000, T: 105000}, {F: 1100, T: 110000}, {F: 1100, T: 115000}, {F: 1200, T: 120000}}, Metric: labels.EmptyLabels(), }, }, @@ -2996,7 +2997,7 @@ func TestRangeQuery(t *testing.T) { Query: "sum_over_time(bar[30s])", Result: Matrix{ Series{ - Points: []Point{{V: 0, T: 0}, {V: 11, T: 60000}, {V: 1100, T: 120000}}, + Floats: []FPoint{{F: 0, T: 0}, {F: 11, T: 60000}, {F: 1100, T: 120000}}, Metric: labels.EmptyLabels(), }, }, @@ -3011,7 +3012,7 @@ func TestRangeQuery(t *testing.T) { Query: "sum_over_time(bar[30s])", Result: Matrix{ Series{ - Points: []Point{{V: 0, T: 0}, {V: 11, T: 60000}, {V: 1100, T: 120000}}, + Floats: []FPoint{{F: 0, T: 0}, {F: 11, T: 60000}, {F: 1100, T: 120000}}, Metric: labels.EmptyLabels(), }, }, @@ -3026,7 +3027,7 @@ func TestRangeQuery(t *testing.T) { Query: "sum_over_time(bar[30s])", Result: Matrix{ Series{ - Points: []Point{{V: 0, T: 0}, {V: 11, T: 60000}, {V: 1100, T: 120000}, {V: 110000, T: 180000}, {V: 11000000, T: 240000}}, + Floats: []FPoint{{F: 0, T: 0}, {F: 11, T: 60000}, {F: 1100, T: 120000}, {F: 110000, T: 180000}, {F: 11000000, T: 240000}}, Metric: labels.EmptyLabels(), }, }, @@ -3041,7 +3042,7 @@ func TestRangeQuery(t *testing.T) { Query: "sum_over_time(bar[30s])", Result: Matrix{ Series{ - Points: []Point{{V: 5, T: 0}, {V: 59, T: 60000}, {V: 9, T: 120000}, {V: 956, T: 180000}}, + Floats: []FPoint{{F: 5, T: 0}, {F: 59, T: 60000}, {F: 9, T: 120000}, {F: 956, T: 180000}}, Metric: labels.EmptyLabels(), }, }, @@ -3056,7 +3057,7 @@ func TestRangeQuery(t *testing.T) { Query: "metric", Result: Matrix{ Series{ - Points: []Point{{V: 1, T: 0}, {V: 3, T: 60000}, {V: 5, T: 120000}}, + Floats: []FPoint{{F: 1, T: 0}, {F: 3, T: 60000}, {F: 5, T: 120000}}, Metric: labels.FromStrings("__name__", "metric"), }, }, @@ -3071,7 +3072,7 @@ func TestRangeQuery(t *testing.T) { Query: "metric", Result: Matrix{ Series{ - Points: []Point{{V: 1, T: 0}, {V: 3, T: 60000}, {V: 5, T: 120000}}, + Floats: []FPoint{{F: 1, T: 0}, {F: 3, T: 60000}, {F: 5, T: 120000}}, Metric: labels.FromStrings("__name__", "metric"), }, }, @@ -3087,14 +3088,14 @@ func TestRangeQuery(t *testing.T) { Query: `foo > 2 or bar`, Result: Matrix{ Series{ - Points: []Point{{V: 1, T: 0}, {V: 3, T: 60000}, {V: 5, T: 120000}}, + Floats: []FPoint{{F: 1, T: 0}, {F: 3, T: 60000}, {F: 5, T: 120000}}, Metric: labels.FromStrings( "__name__", "bar", "job", "2", ), }, Series{ - Points: []Point{{V: 3, T: 60000}, {V: 5, T: 120000}}, + Floats: []FPoint{{F: 3, T: 60000}, {F: 5, T: 120000}}, Metric: labels.FromStrings( "__name__", "foo", "job", "1", @@ -3266,9 +3267,9 @@ func TestNativeHistogram_HistogramCountAndSum(t *testing.T) { require.Len(t, vector, 1) require.Nil(t, vector[0].H) if floatHisto { - require.Equal(t, float64(h.ToFloat().Count), vector[0].V) + require.Equal(t, float64(h.ToFloat().Count), vector[0].F) } else { - require.Equal(t, float64(h.Count), vector[0].V) + require.Equal(t, float64(h.Count), vector[0].F) } queryString = fmt.Sprintf("histogram_sum(%s)", seriesName) @@ -3284,9 +3285,9 @@ func TestNativeHistogram_HistogramCountAndSum(t *testing.T) { require.Len(t, vector, 1) require.Nil(t, vector[0].H) if floatHisto { - require.Equal(t, h.ToFloat().Sum, vector[0].V) + require.Equal(t, h.ToFloat().Sum, vector[0].F) } else { - require.Equal(t, h.Sum, vector[0].V) + require.Equal(t, h.Sum, vector[0].F) } }) } @@ -3519,7 +3520,7 @@ func TestNativeHistogram_HistogramQuantile(t *testing.T) { require.Len(t, vector, 1) require.Nil(t, vector[0].H) - require.True(t, almostEqual(sc.value, vector[0].V)) + require.True(t, almostEqual(sc.value, vector[0].F)) }) } idx++ @@ -3951,10 +3952,10 @@ func TestNativeHistogram_HistogramFraction(t *testing.T) { require.Len(t, vector, 1) require.Nil(t, vector[0].H) if math.IsNaN(sc.value) { - require.True(t, math.IsNaN(vector[0].V)) + require.True(t, math.IsNaN(vector[0].F)) return } - require.Equal(t, sc.value, vector[0].V) + require.Equal(t, sc.value, vector[0].F) }) } idx++ @@ -4090,24 +4091,18 @@ func TestNativeHistogram_Sum_Count_AddOperator(t *testing.T) { // sum(). queryString := fmt.Sprintf("sum(%s)", seriesName) - queryAndCheck(queryString, []Sample{ - {Point{T: ts, H: &c.expected}, labels.EmptyLabels()}, - }) + queryAndCheck(queryString, []Sample{{T: ts, H: &c.expected, Metric: labels.EmptyLabels()}}) // + operator. queryString = fmt.Sprintf(`%s{idx="0"}`, seriesName) for idx := 1; idx < len(c.histograms); idx++ { queryString += fmt.Sprintf(` + ignoring(idx) %s{idx="%d"}`, seriesName, idx) } - queryAndCheck(queryString, []Sample{ - {Point{T: ts, H: &c.expected}, labels.EmptyLabels()}, - }) + queryAndCheck(queryString, []Sample{{T: ts, H: &c.expected, Metric: labels.EmptyLabels()}}) // count(). queryString = fmt.Sprintf("count(%s)", seriesName) - queryAndCheck(queryString, []Sample{ - {Point{T: ts, V: 3}, labels.EmptyLabels()}, - }) + queryAndCheck(queryString, []Sample{{T: ts, F: 3, Metric: labels.EmptyLabels()}}) }) idx0++ } diff --git a/promql/functions.go b/promql/functions.go index e1a02ac8e3..8fc5e64a58 100644 --- a/promql/functions.go +++ b/promql/functions.go @@ -54,9 +54,9 @@ type FunctionCall func(vals []parser.Value, args parser.Expressions, enh *EvalNo // === time() float64 === func funcTime(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { - return Vector{Sample{Point: Point{ - V: float64(enh.Ts) / 1000, - }}} + return Vector{Sample{ + F: float64(enh.Ts) / 1000, + }} } // extrapolatedRate is a utility function for rate/increase/delta. @@ -67,65 +67,71 @@ func extrapolatedRate(vals []parser.Value, args parser.Expressions, enh *EvalNod ms := args[0].(*parser.MatrixSelector) vs := ms.VectorSelector.(*parser.VectorSelector) var ( - samples = vals[0].(Matrix)[0] - rangeStart = enh.Ts - durationMilliseconds(ms.Range+vs.Offset) - rangeEnd = enh.Ts - durationMilliseconds(vs.Offset) - resultValue float64 - resultHistogram *histogram.FloatHistogram + samples = vals[0].(Matrix)[0] + rangeStart = enh.Ts - durationMilliseconds(ms.Range+vs.Offset) + rangeEnd = enh.Ts - durationMilliseconds(vs.Offset) + resultValue float64 + resultHistogram *histogram.FloatHistogram + firstT, lastT int64 + numSamplesMinusOne int ) - // No sense in trying to compute a rate without at least two points. Drop - // this Vector element. - if len(samples.Points) < 2 { + // We need either at least two Histograms and no Floats, or at least two + // Floats and no Histograms to calculate a rate. Otherwise, drop this + // Vector element. + if len(samples.Histograms) > 0 && len(samples.Floats) > 0 { + // Mix of histograms and floats. TODO(beorn7): Communicate this failure reason. return enh.Out } - if samples.Points[0].H != nil { - resultHistogram = histogramRate(samples.Points, isCounter) + switch { + case len(samples.Histograms) > 1: + numSamplesMinusOne = len(samples.Histograms) - 1 + firstT = samples.Histograms[0].T + lastT = samples.Histograms[numSamplesMinusOne].T + resultHistogram = histogramRate(samples.Histograms, isCounter) if resultHistogram == nil { - // Points are a mix of floats and histograms, or the histograms - // are not compatible with each other. - // TODO(beorn7): find a way of communicating the exact reason + // The histograms are not compatible with each other. + // TODO(beorn7): Communicate this failure reason. return enh.Out } - } else { - resultValue = samples.Points[len(samples.Points)-1].V - samples.Points[0].V - prevValue := samples.Points[0].V - // We have to iterate through everything even in the non-counter - // case because we have to check that everything is a float. - // TODO(beorn7): Find a way to check that earlier, e.g. by - // handing in a []FloatPoint and a []HistogramPoint separately. - for _, currPoint := range samples.Points[1:] { - if currPoint.H != nil { - return nil // Range contains a mix of histograms and floats. - } - if !isCounter { - continue - } - if currPoint.V < prevValue { + case len(samples.Floats) > 1: + numSamplesMinusOne = len(samples.Floats) - 1 + firstT = samples.Floats[0].T + lastT = samples.Floats[numSamplesMinusOne].T + resultValue = samples.Floats[numSamplesMinusOne].F - samples.Floats[0].F + if !isCounter { + break + } + // Handle counter resets: + prevValue := samples.Floats[0].F + for _, currPoint := range samples.Floats[1:] { + if currPoint.F < prevValue { resultValue += prevValue } - prevValue = currPoint.V + prevValue = currPoint.F } + default: + // Not enough samples. TODO(beorn7): Communicate this failure reason. + return enh.Out } // Duration between first/last samples and boundary of range. - durationToStart := float64(samples.Points[0].T-rangeStart) / 1000 - durationToEnd := float64(rangeEnd-samples.Points[len(samples.Points)-1].T) / 1000 + durationToStart := float64(firstT-rangeStart) / 1000 + durationToEnd := float64(rangeEnd-lastT) / 1000 - sampledInterval := float64(samples.Points[len(samples.Points)-1].T-samples.Points[0].T) / 1000 - averageDurationBetweenSamples := sampledInterval / float64(len(samples.Points)-1) + sampledInterval := float64(lastT-firstT) / 1000 + averageDurationBetweenSamples := sampledInterval / float64(numSamplesMinusOne) // TODO(beorn7): Do this for histograms, too. - if isCounter && resultValue > 0 && samples.Points[0].V >= 0 { - // Counters cannot be negative. If we have any slope at - // all (i.e. resultValue went up), we can extrapolate - // the zero point of the counter. If the duration to the - // zero point is shorter than the durationToStart, we - // take the zero point as the start of the series, - // thereby avoiding extrapolation to negative counter - // values. - durationToZero := sampledInterval * (samples.Points[0].V / resultValue) + if isCounter && resultValue > 0 && len(samples.Floats) > 0 && samples.Floats[0].F >= 0 { + // Counters cannot be negative. If we have any slope at all + // (i.e. resultValue went up), we can extrapolate the zero point + // of the counter. If the duration to the zero point is shorter + // than the durationToStart, we take the zero point as the start + // of the series, thereby avoiding extrapolation to negative + // counter values. + durationToZero := sampledInterval * (samples.Floats[0].F / resultValue) if durationToZero < durationToStart { durationToStart = durationToZero } @@ -158,16 +164,14 @@ func extrapolatedRate(vals []parser.Value, args parser.Expressions, enh *EvalNod resultHistogram.Scale(factor) } - return append(enh.Out, Sample{ - Point: Point{V: resultValue, H: resultHistogram}, - }) + return append(enh.Out, Sample{F: resultValue, H: resultHistogram}) } // histogramRate is a helper function for extrapolatedRate. It requires // points[0] to be a histogram. It returns nil if any other Point in points is // not a histogram. -func histogramRate(points []Point, isCounter bool) *histogram.FloatHistogram { - prev := points[0].H // We already know that this is a histogram. +func histogramRate(points []HPoint, isCounter bool) *histogram.FloatHistogram { + prev := points[0].H last := points[len(points)-1].H if last == nil { return nil // Range contains a mix of histograms and floats. @@ -243,19 +247,19 @@ func instantValue(vals []parser.Value, out Vector, isRate bool) Vector { samples := vals[0].(Matrix)[0] // No sense in trying to compute a rate without at least two points. Drop // this Vector element. - if len(samples.Points) < 2 { + if len(samples.Floats) < 2 { return out } - lastSample := samples.Points[len(samples.Points)-1] - previousSample := samples.Points[len(samples.Points)-2] + lastSample := samples.Floats[len(samples.Floats)-1] + previousSample := samples.Floats[len(samples.Floats)-2] var resultValue float64 - if isRate && lastSample.V < previousSample.V { + if isRate && lastSample.F < previousSample.F { // Counter reset. - resultValue = lastSample.V + resultValue = lastSample.F } else { - resultValue = lastSample.V - previousSample.V + resultValue = lastSample.F - previousSample.F } sampledInterval := lastSample.T - previousSample.T @@ -269,9 +273,7 @@ func instantValue(vals []parser.Value, out Vector, isRate bool) Vector { resultValue /= float64(sampledInterval) / 1000 } - return append(out, Sample{ - Point: Point{V: resultValue}, - }) + return append(out, Sample{F: resultValue}) } // Calculate the trend value at the given index i in raw data d. @@ -300,10 +302,10 @@ func funcHoltWinters(vals []parser.Value, args parser.Expressions, enh *EvalNode samples := vals[0].(Matrix)[0] // The smoothing factor argument. - sf := vals[1].(Vector)[0].V + sf := vals[1].(Vector)[0].F // The trend factor argument. - tf := vals[2].(Vector)[0].V + tf := vals[2].(Vector)[0].F // Check that the input parameters are valid. if sf <= 0 || sf >= 1 { @@ -313,7 +315,7 @@ func funcHoltWinters(vals []parser.Value, args parser.Expressions, enh *EvalNode panic(fmt.Errorf("invalid trend factor. Expected: 0 < tf < 1, got: %f", tf)) } - l := len(samples.Points) + l := len(samples.Floats) // Can't do the smoothing operation with less than two points. if l < 2 { @@ -322,15 +324,15 @@ func funcHoltWinters(vals []parser.Value, args parser.Expressions, enh *EvalNode var s0, s1, b float64 // Set initial values. - s1 = samples.Points[0].V - b = samples.Points[1].V - samples.Points[0].V + s1 = samples.Floats[0].F + b = samples.Floats[1].F - samples.Floats[0].F // Run the smoothing operation. var x, y float64 for i := 1; i < l; i++ { // Scale the raw value against the smoothing factor. - x = sf * samples.Points[i].V + x = sf * samples.Floats[i].F // Scale the last smoothed value with the trend at this point. b = calcTrendValue(i-1, tf, s0, s1, b) @@ -339,9 +341,7 @@ func funcHoltWinters(vals []parser.Value, args parser.Expressions, enh *EvalNode s0, s1 = s1, x+y } - return append(enh.Out, Sample{ - Point: Point{V: s1}, - }) + return append(enh.Out, Sample{F: s1}) } // === sort(node parser.ValueTypeVector) Vector === @@ -365,15 +365,15 @@ func funcSortDesc(vals []parser.Value, args parser.Expressions, enh *EvalNodeHel // === clamp(Vector parser.ValueTypeVector, min, max Scalar) Vector === func funcClamp(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { vec := vals[0].(Vector) - min := vals[1].(Vector)[0].Point.V - max := vals[2].(Vector)[0].Point.V + min := vals[1].(Vector)[0].F + max := vals[2].(Vector)[0].F if max < min { return enh.Out } for _, el := range vec { enh.Out = append(enh.Out, Sample{ Metric: enh.DropMetricName(el.Metric), - Point: Point{V: math.Max(min, math.Min(max, el.V))}, + F: math.Max(min, math.Min(max, el.F)), }) } return enh.Out @@ -382,11 +382,11 @@ func funcClamp(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper // === clamp_max(Vector parser.ValueTypeVector, max Scalar) Vector === func funcClampMax(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { vec := vals[0].(Vector) - max := vals[1].(Vector)[0].Point.V + max := vals[1].(Vector)[0].F for _, el := range vec { enh.Out = append(enh.Out, Sample{ Metric: enh.DropMetricName(el.Metric), - Point: Point{V: math.Min(max, el.V)}, + F: math.Min(max, el.F), }) } return enh.Out @@ -395,11 +395,11 @@ func funcClampMax(vals []parser.Value, args parser.Expressions, enh *EvalNodeHel // === clamp_min(Vector parser.ValueTypeVector, min Scalar) Vector === func funcClampMin(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { vec := vals[0].(Vector) - min := vals[1].(Vector)[0].Point.V + min := vals[1].(Vector)[0].F for _, el := range vec { enh.Out = append(enh.Out, Sample{ Metric: enh.DropMetricName(el.Metric), - Point: Point{V: math.Max(min, el.V)}, + F: math.Max(min, el.F), }) } return enh.Out @@ -412,16 +412,16 @@ func funcRound(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper // Ties are solved by rounding up. toNearest := float64(1) if len(args) >= 2 { - toNearest = vals[1].(Vector)[0].Point.V + toNearest = vals[1].(Vector)[0].F } // Invert as it seems to cause fewer floating point accuracy issues. toNearestInverse := 1.0 / toNearest for _, el := range vec { - v := math.Floor(el.V*toNearestInverse+0.5) / toNearestInverse + v := math.Floor(el.F*toNearestInverse+0.5) / toNearestInverse enh.Out = append(enh.Out, Sample{ Metric: enh.DropMetricName(el.Metric), - Point: Point{V: v}, + F: v, }) } return enh.Out @@ -431,37 +431,38 @@ func funcRound(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper func funcScalar(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { v := vals[0].(Vector) if len(v) != 1 { - return append(enh.Out, Sample{ - Point: Point{V: math.NaN()}, - }) + return append(enh.Out, Sample{F: math.NaN()}) } - return append(enh.Out, Sample{ - Point: Point{V: v[0].V}, - }) + return append(enh.Out, Sample{F: v[0].F}) } -func aggrOverTime(vals []parser.Value, enh *EvalNodeHelper, aggrFn func([]Point) float64) Vector { +func aggrOverTime(vals []parser.Value, enh *EvalNodeHelper, aggrFn func(Series) float64) Vector { el := vals[0].(Matrix)[0] - return append(enh.Out, Sample{ - Point: Point{V: aggrFn(el.Points)}, - }) + return append(enh.Out, Sample{F: aggrFn(el)}) } // === avg_over_time(Matrix parser.ValueTypeMatrix) Vector === func funcAvgOverTime(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { - return aggrOverTime(vals, enh, func(values []Point) float64 { + if len(vals[0].(Matrix)[0].Floats) == 0 { + // TODO(beorn7): The passed values only contain + // histograms. avg_over_time ignores histograms for now. If + // there are only histograms, we have to return without adding + // anything to enh.Out. + return enh.Out + } + return aggrOverTime(vals, enh, func(s Series) float64 { var mean, count, c float64 - for _, v := range values { + for _, f := range s.Floats { count++ if math.IsInf(mean, 0) { - if math.IsInf(v.V, 0) && (mean > 0) == (v.V > 0) { - // The `mean` and `v.V` values are `Inf` of the same sign. They + if math.IsInf(f.F, 0) && (mean > 0) == (f.F > 0) { + // The `mean` and `f.F` values are `Inf` of the same sign. They // can't be subtracted, but the value of `mean` is correct // already. continue } - if !math.IsInf(v.V, 0) && !math.IsNaN(v.V) { + if !math.IsInf(f.F, 0) && !math.IsNaN(f.F) { // At this stage, the mean is an infinite. If the added // value is neither an Inf or a Nan, we can keep that mean // value. @@ -471,7 +472,7 @@ func funcAvgOverTime(vals []parser.Value, args parser.Expressions, enh *EvalNode continue } } - mean, c = kahanSumInc(v.V/count-mean/count, mean, c) + mean, c = kahanSumInc(f.F/count-mean/count, mean, c) } if math.IsInf(mean, 0) { @@ -483,8 +484,8 @@ func funcAvgOverTime(vals []parser.Value, args parser.Expressions, enh *EvalNode // === count_over_time(Matrix parser.ValueTypeMatrix) Vector === func funcCountOverTime(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { - return aggrOverTime(vals, enh, func(values []Point) float64 { - return float64(len(values)) + return aggrOverTime(vals, enh, func(s Series) float64 { + return float64(len(s.Floats) + len(s.Histograms)) }) } @@ -492,19 +493,42 @@ func funcCountOverTime(vals []parser.Value, args parser.Expressions, enh *EvalNo func funcLastOverTime(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { el := vals[0].(Matrix)[0] + var f FPoint + if len(el.Floats) > 0 { + f = el.Floats[len(el.Floats)-1] + } + + var h HPoint + if len(el.Histograms) > 0 { + h = el.Histograms[len(el.Histograms)-1] + } + + if h.H == nil || h.T < f.T { + return append(enh.Out, Sample{ + Metric: el.Metric, + F: f.F, + }) + } return append(enh.Out, Sample{ Metric: el.Metric, - Point: Point{V: el.Points[len(el.Points)-1].V}, + H: h.H, }) } // === max_over_time(Matrix parser.ValueTypeMatrix) Vector === func funcMaxOverTime(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { - return aggrOverTime(vals, enh, func(values []Point) float64 { - max := values[0].V - for _, v := range values { - if v.V > max || math.IsNaN(max) { - max = v.V + if len(vals[0].(Matrix)[0].Floats) == 0 { + // TODO(beorn7): The passed values only contain + // histograms. max_over_time ignores histograms for now. If + // there are only histograms, we have to return without adding + // anything to enh.Out. + return enh.Out + } + return aggrOverTime(vals, enh, func(s Series) float64 { + max := s.Floats[0].F + for _, f := range s.Floats { + if f.F > max || math.IsNaN(max) { + max = f.F } } return max @@ -513,11 +537,18 @@ func funcMaxOverTime(vals []parser.Value, args parser.Expressions, enh *EvalNode // === min_over_time(Matrix parser.ValueTypeMatrix) Vector === func funcMinOverTime(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { - return aggrOverTime(vals, enh, func(values []Point) float64 { - min := values[0].V - for _, v := range values { - if v.V < min || math.IsNaN(min) { - min = v.V + if len(vals[0].(Matrix)[0].Floats) == 0 { + // TODO(beorn7): The passed values only contain + // histograms. min_over_time ignores histograms for now. If + // there are only histograms, we have to return without adding + // anything to enh.Out. + return enh.Out + } + return aggrOverTime(vals, enh, func(s Series) float64 { + min := s.Floats[0].F + for _, f := range s.Floats { + if f.F < min || math.IsNaN(min) { + min = f.F } } return min @@ -526,10 +557,17 @@ func funcMinOverTime(vals []parser.Value, args parser.Expressions, enh *EvalNode // === sum_over_time(Matrix parser.ValueTypeMatrix) Vector === func funcSumOverTime(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { - return aggrOverTime(vals, enh, func(values []Point) float64 { + if len(vals[0].(Matrix)[0].Floats) == 0 { + // TODO(beorn7): The passed values only contain + // histograms. sum_over_time ignores histograms for now. If + // there are only histograms, we have to return without adding + // anything to enh.Out. + return enh.Out + } + return aggrOverTime(vals, enh, func(s Series) float64 { var sum, c float64 - for _, v := range values { - sum, c = kahanSumInc(v.V, sum, c) + for _, f := range s.Floats { + sum, c = kahanSumInc(f.F, sum, c) } if math.IsInf(sum, 0) { return sum @@ -540,29 +578,41 @@ func funcSumOverTime(vals []parser.Value, args parser.Expressions, enh *EvalNode // === quantile_over_time(Matrix parser.ValueTypeMatrix) Vector === func funcQuantileOverTime(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { - q := vals[0].(Vector)[0].V + q := vals[0].(Vector)[0].F el := vals[1].(Matrix)[0] - - values := make(vectorByValueHeap, 0, len(el.Points)) - for _, v := range el.Points { - values = append(values, Sample{Point: Point{V: v.V}}) + if len(el.Floats) == 0 { + // TODO(beorn7): The passed values only contain + // histograms. quantile_over_time ignores histograms for now. If + // there are only histograms, we have to return without adding + // anything to enh.Out. + return enh.Out } - return append(enh.Out, Sample{ - Point: Point{V: quantile(q, values)}, - }) + + values := make(vectorByValueHeap, 0, len(el.Floats)) + for _, f := range el.Floats { + values = append(values, Sample{F: f.F}) + } + return append(enh.Out, Sample{F: quantile(q, values)}) } // === stddev_over_time(Matrix parser.ValueTypeMatrix) Vector === func funcStddevOverTime(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { - return aggrOverTime(vals, enh, func(values []Point) float64 { + if len(vals[0].(Matrix)[0].Floats) == 0 { + // TODO(beorn7): The passed values only contain + // histograms. stddev_over_time ignores histograms for now. If + // there are only histograms, we have to return without adding + // anything to enh.Out. + return enh.Out + } + return aggrOverTime(vals, enh, func(s Series) float64 { var count float64 var mean, cMean float64 var aux, cAux float64 - for _, v := range values { + for _, f := range s.Floats { count++ - delta := v.V - (mean + cMean) + delta := f.F - (mean + cMean) mean, cMean = kahanSumInc(delta/count, mean, cMean) - aux, cAux = kahanSumInc(delta*(v.V-(mean+cMean)), aux, cAux) + aux, cAux = kahanSumInc(delta*(f.F-(mean+cMean)), aux, cAux) } return math.Sqrt((aux + cAux) / count) }) @@ -570,15 +620,22 @@ func funcStddevOverTime(vals []parser.Value, args parser.Expressions, enh *EvalN // === stdvar_over_time(Matrix parser.ValueTypeMatrix) Vector === func funcStdvarOverTime(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { - return aggrOverTime(vals, enh, func(values []Point) float64 { + if len(vals[0].(Matrix)[0].Floats) == 0 { + // TODO(beorn7): The passed values only contain + // histograms. stdvar_over_time ignores histograms for now. If + // there are only histograms, we have to return without adding + // anything to enh.Out. + return enh.Out + } + return aggrOverTime(vals, enh, func(s Series) float64 { var count float64 var mean, cMean float64 var aux, cAux float64 - for _, v := range values { + for _, f := range s.Floats { count++ - delta := v.V - (mean + cMean) + delta := f.F - (mean + cMean) mean, cMean = kahanSumInc(delta/count, mean, cMean) - aux, cAux = kahanSumInc(delta*(v.V-(mean+cMean)), aux, cAux) + aux, cAux = kahanSumInc(delta*(f.F-(mean+cMean)), aux, cAux) } return (aux + cAux) / count }) @@ -592,7 +649,7 @@ func funcAbsent(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelpe return append(enh.Out, Sample{ Metric: createLabelsForAbsentFunction(args[0]), - Point: Point{V: 1}, + F: 1, }) } @@ -602,25 +659,24 @@ func funcAbsent(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelpe // Due to engine optimization, this function is only called when this condition is true. // Then, the engine post-processes the results to get the expected output. func funcAbsentOverTime(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { - return append(enh.Out, - Sample{ - Point: Point{V: 1}, - }) + return append(enh.Out, Sample{F: 1}) } // === present_over_time(Vector parser.ValueTypeMatrix) Vector === func funcPresentOverTime(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { - return aggrOverTime(vals, enh, func(values []Point) float64 { + return aggrOverTime(vals, enh, func(s Series) float64 { return 1 }) } func simpleFunc(vals []parser.Value, enh *EvalNodeHelper, f func(float64) float64) Vector { for _, el := range vals[0].(Vector) { - enh.Out = append(enh.Out, Sample{ - Metric: enh.DropMetricName(el.Metric), - Point: Point{V: f(el.V)}, - }) + if el.H == nil { // Process only float samples. + enh.Out = append(enh.Out, Sample{ + Metric: enh.DropMetricName(el.Metric), + F: f(el.F), + }) + } } return enh.Out } @@ -741,9 +797,7 @@ func funcDeg(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) // === pi() Scalar === func funcPi(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { - return Vector{Sample{Point: Point{ - V: math.Pi, - }}} + return Vector{Sample{F: math.Pi}} } // === sgn(Vector parser.ValueTypeVector) Vector === @@ -764,7 +818,7 @@ func funcTimestamp(vals []parser.Value, args parser.Expressions, enh *EvalNodeHe for _, el := range vec { enh.Out = append(enh.Out, Sample{ Metric: enh.DropMetricName(el.Metric), - Point: Point{V: float64(el.T) / 1000}, + F: float64(el.T) / 1000, }) } return enh.Out @@ -793,7 +847,7 @@ func kahanSumInc(inc, sum, c float64) (newSum, newC float64) { // linearRegression performs a least-square linear regression analysis on the // provided SamplePairs. It returns the slope, and the intercept value at the // provided time. -func linearRegression(samples []Point, interceptTime int64) (slope, intercept float64) { +func linearRegression(samples []FPoint, interceptTime int64) (slope, intercept float64) { var ( n float64 sumX, cX float64 @@ -803,18 +857,18 @@ func linearRegression(samples []Point, interceptTime int64) (slope, intercept fl initY float64 constY bool ) - initY = samples[0].V + initY = samples[0].F constY = true for i, sample := range samples { // Set constY to false if any new y values are encountered. - if constY && i > 0 && sample.V != initY { + if constY && i > 0 && sample.F != initY { constY = false } n += 1.0 x := float64(sample.T-interceptTime) / 1e3 sumX, cX = kahanSumInc(x, sumX, cX) - sumY, cY = kahanSumInc(sample.V, sumY, cY) - sumXY, cXY = kahanSumInc(x*sample.V, sumXY, cXY) + sumY, cY = kahanSumInc(sample.F, sumY, cY) + sumXY, cXY = kahanSumInc(x*sample.F, sumXY, cXY) sumX2, cX2 = kahanSumInc(x*x, sumX2, cX2) } if constY { @@ -842,33 +896,29 @@ func funcDeriv(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper // No sense in trying to compute a derivative without at least two points. // Drop this Vector element. - if len(samples.Points) < 2 { + if len(samples.Floats) < 2 { return enh.Out } // We pass in an arbitrary timestamp that is near the values in use // to avoid floating point accuracy issues, see // https://github.com/prometheus/prometheus/issues/2674 - slope, _ := linearRegression(samples.Points, samples.Points[0].T) - return append(enh.Out, Sample{ - Point: Point{V: slope}, - }) + slope, _ := linearRegression(samples.Floats, samples.Floats[0].T) + return append(enh.Out, Sample{F: slope}) } // === predict_linear(node parser.ValueTypeMatrix, k parser.ValueTypeScalar) Vector === func funcPredictLinear(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { samples := vals[0].(Matrix)[0] - duration := vals[1].(Vector)[0].V + duration := vals[1].(Vector)[0].F // No sense in trying to predict anything without at least two points. // Drop this Vector element. - if len(samples.Points) < 2 { + if len(samples.Floats) < 2 { return enh.Out } - slope, intercept := linearRegression(samples.Points, enh.Ts) + slope, intercept := linearRegression(samples.Floats, enh.Ts) - return append(enh.Out, Sample{ - Point: Point{V: slope*duration + intercept}, - }) + return append(enh.Out, Sample{F: slope*duration + intercept}) } // === histogram_count(Vector parser.ValueTypeVector) Vector === @@ -882,7 +932,7 @@ func funcHistogramCount(vals []parser.Value, args parser.Expressions, enh *EvalN } enh.Out = append(enh.Out, Sample{ Metric: enh.DropMetricName(sample.Metric), - Point: Point{V: sample.H.Count}, + F: sample.H.Count, }) } return enh.Out @@ -899,7 +949,7 @@ func funcHistogramSum(vals []parser.Value, args parser.Expressions, enh *EvalNod } enh.Out = append(enh.Out, Sample{ Metric: enh.DropMetricName(sample.Metric), - Point: Point{V: sample.H.Sum}, + F: sample.H.Sum, }) } return enh.Out @@ -907,8 +957,8 @@ func funcHistogramSum(vals []parser.Value, args parser.Expressions, enh *EvalNod // === histogram_fraction(lower, upper parser.ValueTypeScalar, Vector parser.ValueTypeVector) Vector === func funcHistogramFraction(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { - lower := vals[0].(Vector)[0].V - upper := vals[1].(Vector)[0].V + lower := vals[0].(Vector)[0].F + upper := vals[1].(Vector)[0].F inVec := vals[2].(Vector) for _, sample := range inVec { @@ -918,7 +968,7 @@ func funcHistogramFraction(vals []parser.Value, args parser.Expressions, enh *Ev } enh.Out = append(enh.Out, Sample{ Metric: enh.DropMetricName(sample.Metric), - Point: Point{V: histogramFraction(lower, upper, sample.H)}, + F: histogramFraction(lower, upper, sample.H), }) } return enh.Out @@ -926,7 +976,7 @@ func funcHistogramFraction(vals []parser.Value, args parser.Expressions, enh *Ev // === histogram_quantile(k parser.ValueTypeScalar, Vector parser.ValueTypeVector) Vector === func funcHistogramQuantile(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { - q := vals[0].(Vector)[0].V + q := vals[0].(Vector)[0].F inVec := vals[1].(Vector) if enh.signatureToMetricWithBuckets == nil { @@ -965,7 +1015,7 @@ func funcHistogramQuantile(vals []parser.Value, args parser.Expressions, enh *Ev mb = &metricWithBuckets{sample.Metric, nil} enh.signatureToMetricWithBuckets[string(enh.lblBuf)] = mb } - mb.buckets = append(mb.buckets, bucket{upperBound, sample.V}) + mb.buckets = append(mb.buckets, bucket{upperBound, sample.F}) } @@ -985,7 +1035,7 @@ func funcHistogramQuantile(vals []parser.Value, args parser.Expressions, enh *Ev enh.Out = append(enh.Out, Sample{ Metric: enh.DropMetricName(sample.Metric), - Point: Point{V: histogramQuantile(q, sample.H)}, + F: histogramQuantile(q, sample.H), }) } @@ -993,7 +1043,7 @@ func funcHistogramQuantile(vals []parser.Value, args parser.Expressions, enh *Ev if len(mb.buckets) > 0 { enh.Out = append(enh.Out, Sample{ Metric: mb.metric, - Point: Point{V: bucketQuantile(q, mb.buckets)}, + F: bucketQuantile(q, mb.buckets), }) } } @@ -1003,40 +1053,55 @@ func funcHistogramQuantile(vals []parser.Value, args parser.Expressions, enh *Ev // === resets(Matrix parser.ValueTypeMatrix) Vector === func funcResets(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { - samples := vals[0].(Matrix)[0] - + floats := vals[0].(Matrix)[0].Floats + histograms := vals[0].(Matrix)[0].Histograms resets := 0 - prev := samples.Points[0].V - for _, sample := range samples.Points[1:] { - current := sample.V - if current < prev { - resets++ + + if len(floats) > 1 { + prev := floats[0].F + for _, sample := range floats[1:] { + current := sample.F + if current < prev { + resets++ + } + prev = current } - prev = current } - return append(enh.Out, Sample{ - Point: Point{V: float64(resets)}, - }) + if len(histograms) > 1 { + prev := histograms[0].H + for _, sample := range histograms[1:] { + current := sample.H + if current.DetectReset(prev) { + resets++ + } + prev = current + } + } + + return append(enh.Out, Sample{F: float64(resets)}) } // === changes(Matrix parser.ValueTypeMatrix) Vector === func funcChanges(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { - samples := vals[0].(Matrix)[0] - + floats := vals[0].(Matrix)[0].Floats changes := 0 - prev := samples.Points[0].V - for _, sample := range samples.Points[1:] { - current := sample.V + + if len(floats) == 0 { + // TODO(beorn7): Only histogram values, still need to add support. + return enh.Out + } + + prev := floats[0].F + for _, sample := range floats[1:] { + current := sample.F if current != prev && !(math.IsNaN(current) && math.IsNaN(prev)) { changes++ } prev = current } - return append(enh.Out, Sample{ - Point: Point{V: float64(changes)}, - }) + return append(enh.Out, Sample{F: float64(changes)}) } // === label_replace(Vector parser.ValueTypeVector, dst_label, replacement, src_labelname, regex parser.ValueTypeString) Vector === @@ -1087,7 +1152,8 @@ func funcLabelReplace(vals []parser.Value, args parser.Expressions, enh *EvalNod enh.Out = append(enh.Out, Sample{ Metric: outMetric, - Point: Point{V: el.Point.V}, + F: el.F, + H: el.H, }) } return enh.Out @@ -1098,7 +1164,7 @@ func funcVector(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelpe return append(enh.Out, Sample{ Metric: labels.Labels{}, - Point: Point{V: vals[0].(Vector)[0].V}, + F: vals[0].(Vector)[0].F, }) } @@ -1154,7 +1220,8 @@ func funcLabelJoin(vals []parser.Value, args parser.Expressions, enh *EvalNodeHe enh.Out = append(enh.Out, Sample{ Metric: outMetric, - Point: Point{V: el.Point.V}, + F: el.F, + H: el.H, }) } return enh.Out @@ -1166,15 +1233,15 @@ func dateWrapper(vals []parser.Value, enh *EvalNodeHelper, f func(time.Time) flo return append(enh.Out, Sample{ Metric: labels.Labels{}, - Point: Point{V: f(time.Unix(enh.Ts/1000, 0).UTC())}, + F: f(time.Unix(enh.Ts/1000, 0).UTC()), }) } for _, el := range vals[0].(Vector) { - t := time.Unix(int64(el.V), 0).UTC() + t := time.Unix(int64(el.F), 0).UTC() enh.Out = append(enh.Out, Sample{ Metric: enh.DropMetricName(el.Metric), - Point: Point{V: f(t)}, + F: f(t), }) } return enh.Out @@ -1332,10 +1399,20 @@ func (s vectorByValueHeap) Len() int { } func (s vectorByValueHeap) Less(i, j int) bool { - if math.IsNaN(s[i].V) { + // We compare histograms based on their sum of observations. + // TODO(beorn7): Is that what we want? + vi, vj := s[i].F, s[j].F + if s[i].H != nil { + vi = s[i].H.Sum + } + if s[j].H != nil { + vj = s[j].H.Sum + } + + if math.IsNaN(vi) { return true } - return s[i].V < s[j].V + return vi < vj } func (s vectorByValueHeap) Swap(i, j int) { @@ -1361,10 +1438,20 @@ func (s vectorByReverseValueHeap) Len() int { } func (s vectorByReverseValueHeap) Less(i, j int) bool { - if math.IsNaN(s[i].V) { + // We compare histograms based on their sum of observations. + // TODO(beorn7): Is that what we want? + vi, vj := s[i].F, s[j].F + if s[i].H != nil { + vi = s[i].H.Sum + } + if s[j].H != nil { + vj = s[j].H.Sum + } + + if math.IsNaN(vi) { return true } - return s[i].V > s[j].V + return vi > vj } func (s vectorByReverseValueHeap) Swap(i, j int) { diff --git a/promql/functions_test.go b/promql/functions_test.go index 1148b1339a..e552424b3d 100644 --- a/promql/functions_test.go +++ b/promql/functions_test.go @@ -64,7 +64,7 @@ func TestDeriv(t *testing.T) { vec, _ := result.Vector() require.Equal(t, 1, len(vec), "Expected 1 result, got %d", len(vec)) - require.Equal(t, 0.0, vec[0].V, "Expected 0.0 as value, got %f", vec[0].V) + require.Equal(t, 0.0, vec[0].F, "Expected 0.0 as value, got %f", vec[0].F) } func TestFunctionList(t *testing.T) { diff --git a/promql/quantile.go b/promql/quantile.go index 1561a2ce88..aaead671c7 100644 --- a/promql/quantile.go +++ b/promql/quantile.go @@ -382,5 +382,5 @@ func quantile(q float64, values vectorByValueHeap) float64 { upperIndex := math.Min(n-1, lowerIndex+1) weight := rank - math.Floor(rank) - return values[int(lowerIndex)].V*(1-weight) + values[int(upperIndex)].V*weight + return values[int(lowerIndex)].F*(1-weight) + values[int(upperIndex)].F*weight } diff --git a/promql/test.go b/promql/test.go index 78cc1e9fbb..0b3958393f 100644 --- a/promql/test.go +++ b/promql/test.go @@ -281,7 +281,7 @@ func (*evalCmd) testCmd() {} type loadCmd struct { gap time.Duration metrics map[uint64]labels.Labels - defs map[uint64][]Point + defs map[uint64][]FPoint exemplars map[uint64][]exemplar.Exemplar } @@ -289,7 +289,7 @@ func newLoadCmd(gap time.Duration) *loadCmd { return &loadCmd{ gap: gap, metrics: map[uint64]labels.Labels{}, - defs: map[uint64][]Point{}, + defs: map[uint64][]FPoint{}, exemplars: map[uint64][]exemplar.Exemplar{}, } } @@ -302,13 +302,13 @@ func (cmd loadCmd) String() string { func (cmd *loadCmd) set(m labels.Labels, vals ...parser.SequenceValue) { h := m.Hash() - samples := make([]Point, 0, len(vals)) + samples := make([]FPoint, 0, len(vals)) ts := testStartTime for _, v := range vals { if !v.Omitted { - samples = append(samples, Point{ + samples = append(samples, FPoint{ T: ts.UnixNano() / int64(time.Millisecond/time.Nanosecond), - V: v.Value, + F: v.Value, }) } ts = ts.Add(cmd.gap) @@ -323,7 +323,7 @@ func (cmd *loadCmd) append(a storage.Appender) error { m := cmd.metrics[h] for _, s := range smpls { - if _, err := a.Append(0, m, s.T, s.V); err != nil { + if _, err := a.Append(0, m, s.T, s.F); err != nil { return err } } @@ -399,8 +399,8 @@ func (ev *evalCmd) compareResult(result parser.Value) error { if ev.ordered && exp.pos != pos+1 { return fmt.Errorf("expected metric %s with %v at position %d but was at %d", v.Metric, exp.vals, exp.pos, pos+1) } - if !almostEqual(exp.vals[0].Value, v.V) { - return fmt.Errorf("expected %v for %s but got %v", exp.vals[0].Value, v.Metric, v.V) + if !almostEqual(exp.vals[0].Value, v.F) { + return fmt.Errorf("expected %v for %s but got %v", exp.vals[0].Value, v.Metric, v.F) } seen[fp] = true @@ -409,7 +409,7 @@ func (ev *evalCmd) compareResult(result parser.Value) error { if !seen[fp] { fmt.Println("vector result", len(val), ev.expr) for _, ss := range val { - fmt.Println(" ", ss.Metric, ss.Point) + fmt.Println(" ", ss.Metric, ss.T, ss.F) } return fmt.Errorf("expected metric %s with %v not found", ev.metrics[fp], expVals) } @@ -576,15 +576,15 @@ func (t *Test) exec(tc testCommand) error { mat := rangeRes.Value.(Matrix) vec := make(Vector, 0, len(mat)) for _, series := range mat { - for _, point := range series.Points { + for _, point := range series.Floats { if point.T == timeMilliseconds(iq.evalTime) { - vec = append(vec, Sample{Metric: series.Metric, Point: point}) + vec = append(vec, Sample{Metric: series.Metric, T: point.T, F: point.F}) break } } } if _, ok := res.Value.(Scalar); ok { - err = cmd.compareResult(Scalar{V: vec[0].Point.V}) + err = cmd.compareResult(Scalar{V: vec[0].F}) } else { err = cmd.compareResult(vec) } @@ -763,7 +763,7 @@ func (ll *LazyLoader) appendTill(ts int64) error { ll.loadCmd.defs[h] = smpls[i:] break } - if _, err := app.Append(0, m, s.T, s.V); err != nil { + if _, err := app.Append(0, m, s.T, s.F); err != nil { return err } if i == len(smpls)-1 { diff --git a/promql/test_test.go b/promql/test_test.go index 347f669161..cc1df62d09 100644 --- a/promql/test_test.go +++ b/promql/test_test.go @@ -47,8 +47,8 @@ func TestLazyLoader_WithSamplesTill(t *testing.T) { series: []Series{ { Metric: labels.FromStrings("__name__", "metric1"), - Points: []Point{ - {0, 1, nil}, {10000, 2, nil}, {20000, 3, nil}, {30000, 4, nil}, {40000, 5, nil}, + Floats: []FPoint{ + {0, 1}, {10000, 2}, {20000, 3}, {30000, 4}, {40000, 5}, }, }, }, @@ -58,8 +58,8 @@ func TestLazyLoader_WithSamplesTill(t *testing.T) { series: []Series{ { Metric: labels.FromStrings("__name__", "metric1"), - Points: []Point{ - {0, 1, nil}, {10000, 2, nil}, {20000, 3, nil}, {30000, 4, nil}, {40000, 5, nil}, + Floats: []FPoint{ + {0, 1}, {10000, 2}, {20000, 3}, {30000, 4}, {40000, 5}, }, }, }, @@ -69,8 +69,8 @@ func TestLazyLoader_WithSamplesTill(t *testing.T) { series: []Series{ { Metric: labels.FromStrings("__name__", "metric1"), - Points: []Point{ - {0, 1, nil}, {10000, 2, nil}, {20000, 3, nil}, {30000, 4, nil}, {40000, 5, nil}, {50000, 6, nil}, {60000, 7, nil}, + Floats: []FPoint{ + {0, 1}, {10000, 2}, {20000, 3}, {30000, 4}, {40000, 5}, {50000, 6}, {60000, 7}, }, }, }, @@ -89,14 +89,14 @@ func TestLazyLoader_WithSamplesTill(t *testing.T) { series: []Series{ { Metric: labels.FromStrings("__name__", "metric1"), - Points: []Point{ - {0, 1, nil}, {10000, 1, nil}, {20000, 1, nil}, {30000, 1, nil}, {40000, 1, nil}, {50000, 1, nil}, + Floats: []FPoint{ + {0, 1}, {10000, 1}, {20000, 1}, {30000, 1}, {40000, 1}, {50000, 1}, }, }, { Metric: labels.FromStrings("__name__", "metric2"), - Points: []Point{ - {0, 1, nil}, {10000, 2, nil}, {20000, 3, nil}, {30000, 4, nil}, {40000, 5, nil}, {50000, 6, nil}, {60000, 7, nil}, {70000, 8, nil}, + Floats: []FPoint{ + {0, 1}, {10000, 2}, {20000, 3}, {30000, 4}, {40000, 5}, {50000, 6}, {60000, 7}, {70000, 8}, }, }, }, @@ -146,7 +146,7 @@ func TestLazyLoader_WithSamplesTill(t *testing.T) { it := storageSeries.Iterator(nil) for it.Next() == chunkenc.ValFloat { t, v := it.At() - got.Points = append(got.Points, Point{T: t, V: v}) + got.Floats = append(got.Floats, FPoint{T: t, F: v}) } require.NoError(t, it.Err()) diff --git a/promql/value.go b/promql/value.go index 91904dda25..f59a25112b 100644 --- a/promql/value.go +++ b/promql/value.go @@ -17,6 +17,7 @@ import ( "encoding/json" "errors" "fmt" + "math" "strconv" "strings" @@ -64,76 +65,72 @@ func (s Scalar) MarshalJSON() ([]byte, error) { // Series is a stream of data points belonging to a metric. type Series struct { - Metric labels.Labels - Points []Point + Metric labels.Labels `json:"metric"` + Floats []FPoint `json:"values,omitempty"` + Histograms []HPoint `json:"histograms,omitempty"` } func (s Series) String() string { - vals := make([]string, len(s.Points)) - for i, v := range s.Points { - vals[i] = v.String() + // TODO(beorn7): This currently renders floats first and then + // histograms, each sorted by timestamp. Maybe, in mixed series, that's + // fine. Maybe, however, primary sorting by timestamp is preferred, in + // which case this has to be changed. + vals := make([]string, 0, len(s.Floats)+len(s.Histograms)) + for _, f := range s.Floats { + vals = append(vals, f.String()) + } + for _, h := range s.Histograms { + vals = append(vals, h.String()) } return fmt.Sprintf("%s =>\n%s", s.Metric, strings.Join(vals, "\n")) } -// MarshalJSON is mirrored in web/api/v1/api.go for efficiency reasons. -// This implementation is still provided for debug purposes and usage -// without jsoniter. -func (s Series) MarshalJSON() ([]byte, error) { - // Note that this is rather inefficient because it re-creates the whole - // series, just separated by Histogram Points and Value Points. For API - // purposes, there is a more efficient jsoniter implementation in - // web/api/v1/api.go. - series := struct { - M labels.Labels `json:"metric"` - V []Point `json:"values,omitempty"` - H []Point `json:"histograms,omitempty"` - }{ - M: s.Metric, - } - for _, p := range s.Points { - if p.H == nil { - series.V = append(series.V, p) - continue - } - series.H = append(series.H, p) - } - return json.Marshal(series) -} - -// Point represents a single data point for a given timestamp. -// If H is not nil, then this is a histogram point and only (T, H) is valid. -// If H is nil, then only (T, V) is valid. -type Point struct { +// FPoint represents a single float data point for a given timestamp. +type FPoint struct { T int64 - V float64 - H *histogram.FloatHistogram + F float64 } -func (p Point) String() string { - var s string - if p.H != nil { - s = p.H.String() - } else { - s = strconv.FormatFloat(p.V, 'f', -1, 64) - } +func (p FPoint) String() string { + s := strconv.FormatFloat(p.F, 'f', -1, 64) return fmt.Sprintf("%s @[%v]", s, p.T) } // MarshalJSON implements json.Marshaler. // -// JSON marshaling is only needed for the HTTP API. Since Point is such a +// JSON marshaling is only needed for the HTTP API. Since FPoint is such a // frequently marshaled type, it gets an optimized treatment directly in // web/api/v1/api.go. Therefore, this method is unused within Prometheus. It is // still provided here as convenience for debugging and for other users of this // code. Also note that the different marshaling implementations might lead to // slightly different results in terms of formatting and rounding of the // timestamp. -func (p Point) MarshalJSON() ([]byte, error) { - if p.H == nil { - v := strconv.FormatFloat(p.V, 'f', -1, 64) - return json.Marshal([...]interface{}{float64(p.T) / 1000, v}) - } +func (p FPoint) MarshalJSON() ([]byte, error) { + v := strconv.FormatFloat(p.F, 'f', -1, 64) + return json.Marshal([...]interface{}{float64(p.T) / 1000, v}) +} + +// HPoint represents a single histogram data point for a given timestamp. +// H must never be nil. +type HPoint struct { + T int64 + H *histogram.FloatHistogram +} + +func (p HPoint) String() string { + return fmt.Sprintf("%s @[%v]", p.H.String(), p.T) +} + +// MarshalJSON implements json.Marshaler. +// +// JSON marshaling is only needed for the HTTP API. Since HPoint is such a +// frequently marshaled type, it gets an optimized treatment directly in +// web/api/v1/api.go. Therefore, this method is unused within Prometheus. It is +// still provided here as convenience for debugging and for other users of this +// code. Also note that the different marshaling implementations might lead to +// slightly different results in terms of formatting and rounding of the +// timestamp. +func (p HPoint) MarshalJSON() ([]byte, error) { h := struct { Count string `json:"count"` Sum string `json:"sum"` @@ -171,42 +168,54 @@ func (p Point) MarshalJSON() ([]byte, error) { return json.Marshal([...]interface{}{float64(p.T) / 1000, h}) } -// Sample is a single sample belonging to a metric. +// Sample is a single sample belonging to a metric. It represents either a float +// sample or a histogram sample. If H is nil, it is a float sample. Otherwise, +// it is a histogram sample. type Sample struct { - Point + T int64 + F float64 + H *histogram.FloatHistogram Metric labels.Labels } func (s Sample) String() string { - return fmt.Sprintf("%s => %s", s.Metric, s.Point) + var str string + if s.H == nil { + p := FPoint{T: s.T, F: s.F} + str = p.String() + } else { + p := HPoint{T: s.T, H: s.H} + str = p.String() + } + return fmt.Sprintf("%s => %s", s.Metric, str) } -// MarshalJSON is mirrored in web/api/v1/api.go with jsoniter because Point -// wouldn't be marshaled with jsoniter in all cases otherwise. +// MarshalJSON is mirrored in web/api/v1/api.go with jsoniter because FPoint and +// HPoint wouldn't be marshaled with jsoniter otherwise. func (s Sample) MarshalJSON() ([]byte, error) { - if s.Point.H == nil { - v := struct { + if s.H == nil { + f := struct { M labels.Labels `json:"metric"` - V Point `json:"value"` + F FPoint `json:"value"` }{ M: s.Metric, - V: s.Point, + F: FPoint{T: s.T, F: s.F}, } - return json.Marshal(v) + return json.Marshal(f) } h := struct { M labels.Labels `json:"metric"` - H Point `json:"histogram"` + H HPoint `json:"histogram"` }{ M: s.Metric, - H: s.Point, + H: HPoint{T: s.T, H: s.H}, } return json.Marshal(h) } -// Vector is basically only an alias for model.Samples, but the -// contract is that in a Vector, all Samples have the same timestamp. +// Vector is basically only an an alias for []Sample, but the contract is that +// in a Vector, all Samples have the same timestamp. type Vector []Sample func (vec Vector) String() string { @@ -258,7 +267,7 @@ func (m Matrix) String() string { func (m Matrix) TotalSamples() int { numSamples := 0 for _, series := range m { - numSamples += len(series.Points) + numSamples += len(series.Floats) + len(series.Histograms) } return numSamples } @@ -362,7 +371,8 @@ func (ss *StorageSeries) Labels() labels.Labels { return ss.series.Metric } -// Iterator returns a new iterator of the data of the series. +// Iterator returns a new iterator of the data of the series. In case of +// multiple samples with the same timestamp, it returns the float samples first. func (ss *StorageSeries) Iterator(it chunkenc.Iterator) chunkenc.Iterator { if ssi, ok := it.(*storageSeriesIterator); ok { ssi.reset(ss.series) @@ -372,44 +382,51 @@ func (ss *StorageSeries) Iterator(it chunkenc.Iterator) chunkenc.Iterator { } type storageSeriesIterator struct { - points []Point - curr int + floats []FPoint + histograms []HPoint + iFloats, iHistograms int + currT int64 + currF float64 + currH *histogram.FloatHistogram } func newStorageSeriesIterator(series Series) *storageSeriesIterator { return &storageSeriesIterator{ - points: series.Points, - curr: -1, + floats: series.Floats, + histograms: series.Histograms, + iFloats: -1, + iHistograms: 0, + currT: math.MinInt64, } } func (ssi *storageSeriesIterator) reset(series Series) { - ssi.points = series.Points - ssi.curr = -1 + ssi.floats = series.Floats + ssi.histograms = series.Histograms + ssi.iFloats = -1 + ssi.iHistograms = 0 + ssi.currT = math.MinInt64 + ssi.currF = 0 + ssi.currH = nil } func (ssi *storageSeriesIterator) Seek(t int64) chunkenc.ValueType { - i := ssi.curr - if i < 0 { - i = 0 + if ssi.iFloats >= len(ssi.floats) && ssi.iHistograms >= len(ssi.histograms) { + return chunkenc.ValNone } - for ; i < len(ssi.points); i++ { - p := ssi.points[i] - if p.T >= t { - ssi.curr = i - if p.H != nil { - return chunkenc.ValFloatHistogram - } - return chunkenc.ValFloat + for ssi.currT < t { + if ssi.Next() == chunkenc.ValNone { + return chunkenc.ValNone } } - ssi.curr = len(ssi.points) - 1 - return chunkenc.ValNone + if ssi.currH != nil { + return chunkenc.ValFloatHistogram + } + return chunkenc.ValFloat } func (ssi *storageSeriesIterator) At() (t int64, v float64) { - p := ssi.points[ssi.curr] - return p.T, p.V + return ssi.currT, ssi.currF } func (ssi *storageSeriesIterator) AtHistogram() (int64, *histogram.Histogram) { @@ -417,25 +434,59 @@ func (ssi *storageSeriesIterator) AtHistogram() (int64, *histogram.Histogram) { } func (ssi *storageSeriesIterator) AtFloatHistogram() (int64, *histogram.FloatHistogram) { - p := ssi.points[ssi.curr] - return p.T, p.H + return ssi.currT, ssi.currH } func (ssi *storageSeriesIterator) AtT() int64 { - p := ssi.points[ssi.curr] - return p.T + return ssi.currT } func (ssi *storageSeriesIterator) Next() chunkenc.ValueType { - ssi.curr++ - if ssi.curr >= len(ssi.points) { - return chunkenc.ValNone + if ssi.currH != nil { + ssi.iHistograms++ + } else { + ssi.iFloats++ } - p := ssi.points[ssi.curr] - if p.H != nil { + var ( + pickH, pickF = false, false + floatsExhausted = ssi.iFloats >= len(ssi.floats) + histogramsExhausted = ssi.iHistograms >= len(ssi.histograms) + ) + + switch { + case floatsExhausted: + if histogramsExhausted { // Both exhausted! + return chunkenc.ValNone + } + pickH = true + case histogramsExhausted: // and floats not exhausted. + pickF = true + // From here on, we have to look at timestamps. + case ssi.histograms[ssi.iHistograms].T < ssi.floats[ssi.iFloats].T: + // Next histogram comes before next float. + pickH = true + default: + // In all other cases, we pick float so that we first iterate + // through floats if the timestamp is the same. + pickF = true + } + + switch { + case pickF: + p := ssi.floats[ssi.iFloats] + ssi.currT = p.T + ssi.currF = p.F + ssi.currH = nil + return chunkenc.ValFloat + case pickH: + p := ssi.histograms[ssi.iHistograms] + ssi.currT = p.T + ssi.currF = 0 + ssi.currH = p.H return chunkenc.ValFloatHistogram + default: + panic("storageSeriesIterater.Next failed to pick value type") } - return chunkenc.ValFloat } func (ssi *storageSeriesIterator) Err() error { diff --git a/rules/alerting.go b/rules/alerting.go index c793b90cb1..aed4e8f53e 100644 --- a/rules/alerting.go +++ b/rules/alerting.go @@ -235,7 +235,8 @@ func (r *AlertingRule) sample(alert *Alert, ts time.Time) promql.Sample { s := promql.Sample{ Metric: lb.Labels(), - Point: promql.Point{T: timestamp.FromTime(ts), V: 1}, + T: timestamp.FromTime(ts), + F: 1, } return s } @@ -253,7 +254,8 @@ func (r *AlertingRule) forStateSample(alert *Alert, ts time.Time, v float64) pro s := promql.Sample{ Metric: lb.Labels(), - Point: promql.Point{T: timestamp.FromTime(ts), V: v}, + T: timestamp.FromTime(ts), + F: v, } return s } @@ -339,7 +341,7 @@ func (r *AlertingRule) Eval(ctx context.Context, ts time.Time, query QueryFunc, // Provide the alert information to the template. l := smpl.Metric.Map() - tmplData := template.AlertTemplateData(l, r.externalLabels, r.externalURL, smpl.V) + tmplData := template.AlertTemplateData(l, r.externalLabels, r.externalURL, smpl.F) // Inject some convenience variables that are easier to remember for users // who are not used to Go's templating system. defs := []string{ @@ -394,7 +396,7 @@ func (r *AlertingRule) Eval(ctx context.Context, ts time.Time, query QueryFunc, Annotations: annotations, ActiveAt: ts, State: StatePending, - Value: smpl.V, + Value: smpl.F, } } diff --git a/rules/alerting_test.go b/rules/alerting_test.go index d6c0a9a154..13aab98048 100644 --- a/rules/alerting_test.go +++ b/rules/alerting_test.go @@ -109,7 +109,7 @@ func TestAlertingRuleLabelsUpdate(t *testing.T) { "job", "app-server", "severity", "critical", ), - Point: promql.Point{V: 1}, + F: 1, }, }, { @@ -122,7 +122,7 @@ func TestAlertingRuleLabelsUpdate(t *testing.T) { "job", "app-server", "severity", "warning", ), - Point: promql.Point{V: 1}, + F: 1, }, }, { @@ -135,7 +135,7 @@ func TestAlertingRuleLabelsUpdate(t *testing.T) { "job", "app-server", "severity", "critical", ), - Point: promql.Point{V: 1}, + F: 1, }, }, { @@ -148,7 +148,7 @@ func TestAlertingRuleLabelsUpdate(t *testing.T) { "job", "app-server", "severity", "critical", ), - Point: promql.Point{V: 1}, + F: 1, }, }, } @@ -157,7 +157,7 @@ func TestAlertingRuleLabelsUpdate(t *testing.T) { for i, result := range results { t.Logf("case %d", i) evalTime := baseTime.Add(time.Duration(i) * time.Minute) - result[0].Point.T = timestamp.FromTime(evalTime) + result[0].T = timestamp.FromTime(evalTime) res, err := rule.Eval(suite.Context(), evalTime, EngineQueryFunc(suite.QueryEngine(), suite.Storage()), nil, 0) require.NoError(t, err) @@ -225,7 +225,7 @@ func TestAlertingRuleExternalLabelsInTemplate(t *testing.T) { "job", "app-server", "templated_label", "There are 0 external Labels, of which foo is .", ), - Point: promql.Point{V: 1}, + F: 1, }, promql.Sample{ Metric: labels.FromStrings( @@ -236,13 +236,13 @@ func TestAlertingRuleExternalLabelsInTemplate(t *testing.T) { "job", "app-server", "templated_label", "There are 2 external Labels, of which foo is bar.", ), - Point: promql.Point{V: 1}, + F: 1, }, } evalTime := time.Unix(0, 0) - result[0].Point.T = timestamp.FromTime(evalTime) - result[1].Point.T = timestamp.FromTime(evalTime) + result[0].T = timestamp.FromTime(evalTime) + result[1].T = timestamp.FromTime(evalTime) var filteredRes promql.Vector // After removing 'ALERTS_FOR_STATE' samples. res, err := ruleWithoutExternalLabels.Eval( @@ -321,7 +321,7 @@ func TestAlertingRuleExternalURLInTemplate(t *testing.T) { "job", "app-server", "templated_label", "The external URL is .", ), - Point: promql.Point{V: 1}, + F: 1, }, promql.Sample{ Metric: labels.FromStrings( @@ -332,13 +332,13 @@ func TestAlertingRuleExternalURLInTemplate(t *testing.T) { "job", "app-server", "templated_label", "The external URL is http://localhost:1234.", ), - Point: promql.Point{V: 1}, + F: 1, }, } evalTime := time.Unix(0, 0) - result[0].Point.T = timestamp.FromTime(evalTime) - result[1].Point.T = timestamp.FromTime(evalTime) + result[0].T = timestamp.FromTime(evalTime) + result[1].T = timestamp.FromTime(evalTime) var filteredRes promql.Vector // After removing 'ALERTS_FOR_STATE' samples. res, err := ruleWithoutExternalURL.Eval( @@ -405,12 +405,12 @@ func TestAlertingRuleEmptyLabelFromTemplate(t *testing.T) { "instance", "0", "job", "app-server", ), - Point: promql.Point{V: 1}, + F: 1, }, } evalTime := time.Unix(0, 0) - result[0].Point.T = timestamp.FromTime(evalTime) + result[0].T = timestamp.FromTime(evalTime) var filteredRes promql.Vector // After removing 'ALERTS_FOR_STATE' samples. res, err := rule.Eval( @@ -760,7 +760,7 @@ func TestKeepFiringFor(t *testing.T) { "instance", "0", "job", "app-server", ), - Point: promql.Point{V: 1}, + F: 1, }, }, { @@ -772,7 +772,7 @@ func TestKeepFiringFor(t *testing.T) { "instance", "0", "job", "app-server", ), - Point: promql.Point{V: 1}, + F: 1, }, }, { @@ -784,7 +784,7 @@ func TestKeepFiringFor(t *testing.T) { "instance", "0", "job", "app-server", ), - Point: promql.Point{V: 1}, + F: 1, }, }, { @@ -796,7 +796,7 @@ func TestKeepFiringFor(t *testing.T) { "instance", "0", "job", "app-server", ), - Point: promql.Point{V: 1}, + F: 1, }, }, // From now on the alert should keep firing. @@ -809,7 +809,7 @@ func TestKeepFiringFor(t *testing.T) { "instance", "0", "job", "app-server", ), - Point: promql.Point{V: 1}, + F: 1, }, }, } @@ -818,7 +818,7 @@ func TestKeepFiringFor(t *testing.T) { for i, result := range results { t.Logf("case %d", i) evalTime := baseTime.Add(time.Duration(i) * time.Minute) - result[0].Point.T = timestamp.FromTime(evalTime) + result[0].T = timestamp.FromTime(evalTime) res, err := rule.Eval(suite.Context(), evalTime, EngineQueryFunc(suite.QueryEngine(), suite.Storage()), nil, 0) require.NoError(t, err) @@ -871,11 +871,11 @@ func TestPendingAndKeepFiringFor(t *testing.T) { "instance", "0", "job", "app-server", ), - Point: promql.Point{V: 1}, + F: 1, } baseTime := time.Unix(0, 0) - result.Point.T = timestamp.FromTime(baseTime) + result.T = timestamp.FromTime(baseTime) res, err := rule.Eval(suite.Context(), baseTime, EngineQueryFunc(suite.QueryEngine(), suite.Storage()), nil, 0) require.NoError(t, err) diff --git a/rules/manager.go b/rules/manager.go index 108b6a77a1..07d50be1b9 100644 --- a/rules/manager.go +++ b/rules/manager.go @@ -202,7 +202,8 @@ func EngineQueryFunc(engine *promql.Engine, q storage.Queryable) QueryFunc { return v, nil case promql.Scalar: return promql.Vector{promql.Sample{ - Point: promql.Point{T: v.T, V: v.V}, + T: v.T, + F: v.V, Metric: labels.Labels{}, }}, nil default: @@ -695,7 +696,7 @@ func (g *Group) Eval(ctx context.Context, ts time.Time) { if s.H != nil { _, err = app.AppendHistogram(0, s.Metric, s.T, nil, s.H) } else { - _, err = app.Append(0, s.Metric, s.T, s.V) + _, err = app.Append(0, s.Metric, s.T, s.F) } if err != nil { diff --git a/rules/manager_test.go b/rules/manager_test.go index 1faf730be3..440e06c9af 100644 --- a/rules/manager_test.go +++ b/rules/manager_test.go @@ -80,7 +80,7 @@ func TestAlertingRule(t *testing.T) { "job", "app-server", "severity", "critical", ), - Point: promql.Point{V: 1}, + F: 1, }, promql.Sample{ Metric: labels.FromStrings( @@ -92,7 +92,7 @@ func TestAlertingRule(t *testing.T) { "job", "app-server", "severity", "critical", ), - Point: promql.Point{V: 1}, + F: 1, }, promql.Sample{ Metric: labels.FromStrings( @@ -104,7 +104,7 @@ func TestAlertingRule(t *testing.T) { "job", "app-server", "severity", "critical", ), - Point: promql.Point{V: 1}, + F: 1, }, promql.Sample{ Metric: labels.FromStrings( @@ -116,7 +116,7 @@ func TestAlertingRule(t *testing.T) { "job", "app-server", "severity", "critical", ), - Point: promql.Point{V: 1}, + F: 1, }, } @@ -223,7 +223,7 @@ func TestForStateAddSamples(t *testing.T) { "job", "app-server", "severity", "critical", ), - Point: promql.Point{V: 1}, + F: 1, }, promql.Sample{ Metric: labels.FromStrings( @@ -234,7 +234,7 @@ func TestForStateAddSamples(t *testing.T) { "job", "app-server", "severity", "critical", ), - Point: promql.Point{V: 1}, + F: 1, }, promql.Sample{ Metric: labels.FromStrings( @@ -245,7 +245,7 @@ func TestForStateAddSamples(t *testing.T) { "job", "app-server", "severity", "critical", ), - Point: promql.Point{V: 1}, + F: 1, }, promql.Sample{ Metric: labels.FromStrings( @@ -256,7 +256,7 @@ func TestForStateAddSamples(t *testing.T) { "job", "app-server", "severity", "critical", ), - Point: promql.Point{V: 1}, + F: 1, }, } @@ -327,8 +327,8 @@ func TestForStateAddSamples(t *testing.T) { for i := range test.result { test.result[i].T = timestamp.FromTime(evalTime) // Updating the expected 'for' state. - if test.result[i].V >= 0 { - test.result[i].V = forState + if test.result[i].F >= 0 { + test.result[i].F = forState } } require.Equal(t, len(test.result), len(filteredRes), "%d. Number of samples in expected and actual output don't match (%d vs. %d)", i, len(test.result), len(res)) @@ -584,29 +584,29 @@ func TestStaleness(t *testing.T) { metricSample, ok := samples[metric] require.True(t, ok, "Series %s not returned.", metric) - require.True(t, value.IsStaleNaN(metricSample[2].V), "Appended second sample not as expected. Wanted: stale NaN Got: %x", math.Float64bits(metricSample[2].V)) - metricSample[2].V = 42 // require.Equal cannot handle NaN. + require.True(t, value.IsStaleNaN(metricSample[2].F), "Appended second sample not as expected. Wanted: stale NaN Got: %x", math.Float64bits(metricSample[2].F)) + metricSample[2].F = 42 // require.Equal cannot handle NaN. - want := map[string][]promql.Point{ - metric: {{T: 0, V: 2}, {T: 1000, V: 3}, {T: 2000, V: 42}}, + want := map[string][]promql.FPoint{ + metric: {{T: 0, F: 2}, {T: 1000, F: 3}, {T: 2000, F: 42}}, } require.Equal(t, want, samples) } // Convert a SeriesSet into a form usable with require.Equal. -func readSeriesSet(ss storage.SeriesSet) (map[string][]promql.Point, error) { - result := map[string][]promql.Point{} +func readSeriesSet(ss storage.SeriesSet) (map[string][]promql.FPoint, error) { + result := map[string][]promql.FPoint{} var it chunkenc.Iterator for ss.Next() { series := ss.At() - points := []promql.Point{} + points := []promql.FPoint{} it := series.Iterator(it) for it.Next() == chunkenc.ValFloat { t, v := it.At() - points = append(points, promql.Point{T: t, V: v}) + points = append(points, promql.FPoint{T: t, F: v}) } name := series.Labels().String() @@ -707,7 +707,7 @@ func TestDeletedRuleMarkedStale(t *testing.T) { metricSample, ok := samples[metric] require.True(t, ok, "Series %s not returned.", metric) - require.True(t, value.IsStaleNaN(metricSample[0].V), "Appended sample not as expected. Wanted: stale NaN Got: %x", math.Float64bits(metricSample[0].V)) + require.True(t, value.IsStaleNaN(metricSample[0].F), "Appended sample not as expected. Wanted: stale NaN Got: %x", math.Float64bits(metricSample[0].F)) } func TestUpdate(t *testing.T) { @@ -1134,7 +1134,7 @@ func countStaleNaN(t *testing.T, st storage.Storage) int { require.True(t, ok, "Series %s not returned.", metric) for _, s := range metricSample { - if value.IsStaleNaN(s.V) { + if value.IsStaleNaN(s.F) { c++ } } diff --git a/rules/recording_test.go b/rules/recording_test.go index 932b880b26..61f47e0487 100644 --- a/rules/recording_test.go +++ b/rules/recording_test.go @@ -46,11 +46,13 @@ var ruleEvalTestScenarios = []struct { expected: promql.Vector{ promql.Sample{ Metric: labels.FromStrings("__name__", "test_rule", "label_a", "1", "label_b", "3"), - Point: promql.Point{V: 1, T: timestamp.FromTime(ruleEvaluationTime)}, + F: 1, + T: timestamp.FromTime(ruleEvaluationTime), }, promql.Sample{ Metric: labels.FromStrings("__name__", "test_rule", "label_a", "2", "label_b", "4"), - Point: promql.Point{V: 10, T: timestamp.FromTime(ruleEvaluationTime)}, + F: 10, + T: timestamp.FromTime(ruleEvaluationTime), }, }, }, @@ -61,11 +63,13 @@ var ruleEvalTestScenarios = []struct { expected: promql.Vector{ promql.Sample{ Metric: labels.FromStrings("__name__", "test_rule", "label_a", "1", "label_b", "3", "extra_from_rule", "foo"), - Point: promql.Point{V: 1, T: timestamp.FromTime(ruleEvaluationTime)}, + F: 1, + T: timestamp.FromTime(ruleEvaluationTime), }, promql.Sample{ Metric: labels.FromStrings("__name__", "test_rule", "label_a", "2", "label_b", "4", "extra_from_rule", "foo"), - Point: promql.Point{V: 10, T: timestamp.FromTime(ruleEvaluationTime)}, + F: 10, + T: timestamp.FromTime(ruleEvaluationTime), }, }, }, @@ -76,11 +80,13 @@ var ruleEvalTestScenarios = []struct { expected: promql.Vector{ promql.Sample{ Metric: labels.FromStrings("__name__", "test_rule", "label_a", "from_rule", "label_b", "3"), - Point: promql.Point{V: 1, T: timestamp.FromTime(ruleEvaluationTime)}, + F: 1, + T: timestamp.FromTime(ruleEvaluationTime), }, promql.Sample{ Metric: labels.FromStrings("__name__", "test_rule", "label_a", "from_rule", "label_b", "4"), - Point: promql.Point{V: 10, T: timestamp.FromTime(ruleEvaluationTime)}, + F: 10, + T: timestamp.FromTime(ruleEvaluationTime), }, }, }, @@ -91,11 +97,13 @@ var ruleEvalTestScenarios = []struct { expected: promql.Vector{ promql.Sample{ Metric: labels.FromStrings("__name__", "test_rule", "label_a", "1", "label_b", "3"), - Point: promql.Point{V: 2, T: timestamp.FromTime(ruleEvaluationTime)}, + F: 2, + T: timestamp.FromTime(ruleEvaluationTime), }, promql.Sample{ Metric: labels.FromStrings("__name__", "test_rule", "label_a", "2", "label_b", "4"), - Point: promql.Point{V: 20, T: timestamp.FromTime(ruleEvaluationTime)}, + F: 20, + T: timestamp.FromTime(ruleEvaluationTime), }, }, }, diff --git a/template/template.go b/template/template.go index d5f80006fc..d61a880a22 100644 --- a/template/template.go +++ b/template/template.go @@ -93,7 +93,7 @@ func query(ctx context.Context, q string, ts time.Time, queryFn QueryFunc) (quer result := make(queryResult, len(vector)) for n, v := range vector { s := sample{ - Value: v.V, + Value: v.F, Labels: v.Metric.Map(), } result[n] = &s diff --git a/template/template_test.go b/template/template_test.go index 4d7524d6fd..e7bdcc3b80 100644 --- a/template/template_test.go +++ b/template/template_test.go @@ -70,7 +70,7 @@ func TestTemplateExpansion(t *testing.T) { { text: "{{ query \"1.5\" | first | value }}", output: "1.5", - queryResult: promql.Vector{{Point: promql.Point{T: 0, V: 1.5}}}, + queryResult: promql.Vector{{T: 0, F: 1.5}}, }, { // Get value from query. @@ -78,7 +78,8 @@ func TestTemplateExpansion(t *testing.T) { queryResult: promql.Vector{ { Metric: labels.FromStrings(labels.MetricName, "metric", "instance", "a"), - Point: promql.Point{T: 0, V: 11}, + T: 0, + F: 11, }, }, output: "11", @@ -90,7 +91,8 @@ func TestTemplateExpansion(t *testing.T) { queryResult: promql.Vector{ { Metric: labels.FromStrings(labels.MetricName, "metric", "instance", "a"), - Point: promql.Point{T: 0, V: 11}, + T: 0, + F: 11, }, }, output: "a", @@ -101,7 +103,8 @@ func TestTemplateExpansion(t *testing.T) { queryResult: promql.Vector{ { Metric: labels.FromStrings(labels.MetricName, "metric", "__value__", "a"), - Point: promql.Point{T: 0, V: 11}, + T: 0, + F: 11, }, }, output: "a", @@ -112,7 +115,8 @@ func TestTemplateExpansion(t *testing.T) { queryResult: promql.Vector{ { Metric: labels.FromStrings(labels.MetricName, "metric", "instance", "a"), - Point: promql.Point{T: 0, V: 11}, + T: 0, + F: 11, }, }, output: "", @@ -123,7 +127,8 @@ func TestTemplateExpansion(t *testing.T) { queryResult: promql.Vector{ { Metric: labels.FromStrings(labels.MetricName, "metric", "instance", "a"), - Point: promql.Point{T: 0, V: 11}, + T: 0, + F: 11, }, }, output: "", @@ -133,7 +138,8 @@ func TestTemplateExpansion(t *testing.T) { queryResult: promql.Vector{ { Metric: labels.FromStrings(labels.MetricName, "metric", "instance", "a"), - Point: promql.Point{T: 0, V: 11}, + T: 0, + F: 11, }, }, output: "", @@ -145,10 +151,12 @@ func TestTemplateExpansion(t *testing.T) { queryResult: promql.Vector{ { Metric: labels.FromStrings(labels.MetricName, "metric", "instance", "b"), - Point: promql.Point{T: 0, V: 21}, + T: 0, + F: 21, }, { Metric: labels.FromStrings(labels.MetricName, "metric", "instance", "a"), - Point: promql.Point{T: 0, V: 11}, + T: 0, + F: 11, }, }, output: "a:11: b:21: ", diff --git a/util/jsonutil/marshal.go b/util/jsonutil/marshal.go index a82ae100d7..4400385d0d 100644 --- a/util/jsonutil/marshal.go +++ b/util/jsonutil/marshal.go @@ -18,6 +18,8 @@ import ( "strconv" jsoniter "github.com/json-iterator/go" + + "github.com/prometheus/prometheus/model/histogram" ) // MarshalTimestamp marshals a point timestamp using the passed jsoniter stream. @@ -42,8 +44,8 @@ func MarshalTimestamp(t int64, stream *jsoniter.Stream) { } } -// MarshalValue marshals a point value using the passed jsoniter stream. -func MarshalValue(v float64, stream *jsoniter.Stream) { +// MarshalFloat marshals a float value using the passed jsoniter stream. +func MarshalFloat(v float64, stream *jsoniter.Stream) { stream.WriteRaw(`"`) // Taken from https://github.com/json-iterator/go/blob/master/stream_float.go#L71 as a workaround // to https://github.com/json-iterator/go/issues/365 (jsoniter, to follow json standard, doesn't allow inf/nan). @@ -60,3 +62,76 @@ func MarshalValue(v float64, stream *jsoniter.Stream) { stream.SetBuffer(buf) stream.WriteRaw(`"`) } + +// MarshalHistogram marshals a histogram value using the passed jsoniter stream. +// It writes something like: +// +// { +// "count": "42", +// "sum": "34593.34", +// "buckets": [ +// [ 3, "-0.25", "0.25", "3"], +// [ 0, "0.25", "0.5", "12"], +// [ 0, "0.5", "1", "21"], +// [ 0, "2", "4", "6"] +// ] +// } +// +// The 1st element in each bucket array determines if the boundaries are +// inclusive (AKA closed) or exclusive (AKA open): +// +// 0: lower exclusive, upper inclusive +// 1: lower inclusive, upper exclusive +// 2: both exclusive +// 3: both inclusive +// +// The 2nd and 3rd elements are the lower and upper boundary. The 4th element is +// the bucket count. +func MarshalHistogram(h *histogram.FloatHistogram, stream *jsoniter.Stream) { + stream.WriteObjectStart() + stream.WriteObjectField(`count`) + MarshalFloat(h.Count, stream) + stream.WriteMore() + stream.WriteObjectField(`sum`) + MarshalFloat(h.Sum, stream) + + bucketFound := false + it := h.AllBucketIterator() + for it.Next() { + bucket := it.At() + if bucket.Count == 0 { + continue // No need to expose empty buckets in JSON. + } + stream.WriteMore() + if !bucketFound { + stream.WriteObjectField(`buckets`) + stream.WriteArrayStart() + } + bucketFound = true + boundaries := 2 // Exclusive on both sides AKA open interval. + if bucket.LowerInclusive { + if bucket.UpperInclusive { + boundaries = 3 // Inclusive on both sides AKA closed interval. + } else { + boundaries = 1 // Inclusive only on lower end AKA right open. + } + } else { + if bucket.UpperInclusive { + boundaries = 0 // Inclusive only on upper end AKA left open. + } + } + stream.WriteArrayStart() + stream.WriteInt(boundaries) + stream.WriteMore() + MarshalFloat(bucket.Lower, stream) + stream.WriteMore() + MarshalFloat(bucket.Upper, stream) + stream.WriteMore() + MarshalFloat(bucket.Count, stream) + stream.WriteArrayEnd() + } + if bucketFound { + stream.WriteArrayEnd() + } + stream.WriteObjectEnd() +} diff --git a/web/api/v1/api.go b/web/api/v1/api.go index 3b6ade562a..aeea87ca70 100644 --- a/web/api/v1/api.go +++ b/web/api/v1/api.go @@ -41,7 +41,6 @@ import ( "github.com/prometheus/prometheus/config" "github.com/prometheus/prometheus/model/exemplar" - "github.com/prometheus/prometheus/model/histogram" "github.com/prometheus/prometheus/model/labels" "github.com/prometheus/prometheus/model/textparse" "github.com/prometheus/prometheus/model/timestamp" @@ -217,7 +216,8 @@ type API struct { func init() { jsoniter.RegisterTypeEncoderFunc("promql.Series", marshalSeriesJSON, marshalSeriesJSONIsEmpty) jsoniter.RegisterTypeEncoderFunc("promql.Sample", marshalSampleJSON, marshalSampleJSONIsEmpty) - jsoniter.RegisterTypeEncoderFunc("promql.Point", marshalPointJSON, marshalPointJSONIsEmpty) + jsoniter.RegisterTypeEncoderFunc("promql.FPoint", marshalFPointJSON, marshalPointJSONIsEmpty) + jsoniter.RegisterTypeEncoderFunc("promql.HPoint", marshalHPointJSON, marshalPointJSONIsEmpty) jsoniter.RegisterTypeEncoderFunc("exemplar.Exemplar", marshalExemplarJSON, marshalExemplarJSONEmpty) } @@ -1724,7 +1724,7 @@ OUTER: // < more values> // ], // "histograms": [ -// [ 1435781451.781, { < histogram, see below > } ], +// [ 1435781451.781, { < histogram, see jsonutil.MarshalHistogram > } ], // < more histograms > // ], // }, @@ -1739,41 +1739,26 @@ func marshalSeriesJSON(ptr unsafe.Pointer, stream *jsoniter.Stream) { } stream.SetBuffer(append(stream.Buffer(), m...)) - // We make two passes through the series here: In the first marshaling - // all value points, in the second marshaling all histogram - // points. That's probably cheaper than just one pass in which we copy - // out histogram Points into a newly allocated slice for separate - // marshaling. (Could be benchmarked, though.) - var foundValue, foundHistogram bool - for _, p := range s.Points { - if p.H == nil { - stream.WriteMore() - if !foundValue { - stream.WriteObjectField(`values`) - stream.WriteArrayStart() - } - foundValue = true - marshalPointJSON(unsafe.Pointer(&p), stream) - } else { - foundHistogram = true + for i, p := range s.Floats { + stream.WriteMore() + if i == 0 { + stream.WriteObjectField(`values`) + stream.WriteArrayStart() } + marshalFPointJSON(unsafe.Pointer(&p), stream) } - if foundValue { + if len(s.Floats) > 0 { stream.WriteArrayEnd() } - if foundHistogram { - firstHistogram := true - for _, p := range s.Points { - if p.H != nil { - stream.WriteMore() - if firstHistogram { - stream.WriteObjectField(`histograms`) - stream.WriteArrayStart() - } - firstHistogram = false - marshalPointJSON(unsafe.Pointer(&p), stream) - } + for i, p := range s.Histograms { + stream.WriteMore() + if i == 0 { + stream.WriteObjectField(`histograms`) + stream.WriteArrayStart() } + marshalHPointJSON(unsafe.Pointer(&p), stream) + } + if len(s.Histograms) > 0 { stream.WriteArrayEnd() } stream.WriteObjectEnd() @@ -1791,7 +1776,7 @@ func marshalSeriesJSONIsEmpty(ptr unsafe.Pointer) bool { // "job" : "prometheus", // "instance" : "localhost:9090" // }, -// "value": [ 1435781451.781, "1" ] +// "value": [ 1435781451.781, "1.234" ] // }, // // For histogram samples, it writes something like this: @@ -1802,7 +1787,7 @@ func marshalSeriesJSONIsEmpty(ptr unsafe.Pointer) bool { // "job" : "prometheus", // "instance" : "localhost:9090" // }, -// "histogram": [ 1435781451.781, { < histogram, see below > } ] +// "histogram": [ 1435781451.781, { < histogram, see jsonutil.MarshalHistogram > } ] // }, func marshalSampleJSON(ptr unsafe.Pointer, stream *jsoniter.Stream) { s := *((*promql.Sample)(ptr)) @@ -1815,12 +1800,20 @@ func marshalSampleJSON(ptr unsafe.Pointer, stream *jsoniter.Stream) { } stream.SetBuffer(append(stream.Buffer(), m...)) stream.WriteMore() - if s.Point.H == nil { + if s.H == nil { stream.WriteObjectField(`value`) } else { stream.WriteObjectField(`histogram`) } - marshalPointJSON(unsafe.Pointer(&s.Point), stream) + stream.WriteArrayStart() + jsonutil.MarshalTimestamp(s.T, stream) + stream.WriteMore() + if s.H == nil { + jsonutil.MarshalFloat(s.F, stream) + } else { + jsonutil.MarshalHistogram(s.H, stream) + } + stream.WriteArrayEnd() stream.WriteObjectEnd() } @@ -1828,17 +1821,23 @@ func marshalSampleJSONIsEmpty(ptr unsafe.Pointer) bool { return false } -// marshalPointJSON writes `[ts, "val"]`. -func marshalPointJSON(ptr unsafe.Pointer, stream *jsoniter.Stream) { - p := *((*promql.Point)(ptr)) +// marshalFPointJSON writes `[ts, "1.234"]`. +func marshalFPointJSON(ptr unsafe.Pointer, stream *jsoniter.Stream) { + p := *((*promql.FPoint)(ptr)) stream.WriteArrayStart() jsonutil.MarshalTimestamp(p.T, stream) stream.WriteMore() - if p.H == nil { - jsonutil.MarshalValue(p.V, stream) - } else { - marshalHistogram(p.H, stream) - } + jsonutil.MarshalFloat(p.F, stream) + stream.WriteArrayEnd() +} + +// marshalHPointJSON writes `[ts, { < histogram, see jsonutil.MarshalHistogram > } ]`. +func marshalHPointJSON(ptr unsafe.Pointer, stream *jsoniter.Stream) { + p := *((*promql.HPoint)(ptr)) + stream.WriteArrayStart() + jsonutil.MarshalTimestamp(p.T, stream) + stream.WriteMore() + jsonutil.MarshalHistogram(p.H, stream) stream.WriteArrayEnd() } @@ -1846,78 +1845,6 @@ func marshalPointJSONIsEmpty(ptr unsafe.Pointer) bool { return false } -// marshalHistogramJSON writes something like: -// -// { -// "count": "42", -// "sum": "34593.34", -// "buckets": [ -// [ 3, "-0.25", "0.25", "3"], -// [ 0, "0.25", "0.5", "12"], -// [ 0, "0.5", "1", "21"], -// [ 0, "2", "4", "6"] -// ] -// } -// -// The 1st element in each bucket array determines if the boundaries are -// inclusive (AKA closed) or exclusive (AKA open): -// -// 0: lower exclusive, upper inclusive -// 1: lower inclusive, upper exclusive -// 2: both exclusive -// 3: both inclusive -// -// The 2nd and 3rd elements are the lower and upper boundary. The 4th element is -// the bucket count. -func marshalHistogram(h *histogram.FloatHistogram, stream *jsoniter.Stream) { - stream.WriteObjectStart() - stream.WriteObjectField(`count`) - jsonutil.MarshalValue(h.Count, stream) - stream.WriteMore() - stream.WriteObjectField(`sum`) - jsonutil.MarshalValue(h.Sum, stream) - - bucketFound := false - it := h.AllBucketIterator() - for it.Next() { - bucket := it.At() - if bucket.Count == 0 { - continue // No need to expose empty buckets in JSON. - } - stream.WriteMore() - if !bucketFound { - stream.WriteObjectField(`buckets`) - stream.WriteArrayStart() - } - bucketFound = true - boundaries := 2 // Exclusive on both sides AKA open interval. - if bucket.LowerInclusive { - if bucket.UpperInclusive { - boundaries = 3 // Inclusive on both sides AKA closed interval. - } else { - boundaries = 1 // Inclusive only on lower end AKA right open. - } - } else { - if bucket.UpperInclusive { - boundaries = 0 // Inclusive only on upper end AKA left open. - } - } - stream.WriteArrayStart() - stream.WriteInt(boundaries) - stream.WriteMore() - jsonutil.MarshalValue(bucket.Lower, stream) - stream.WriteMore() - jsonutil.MarshalValue(bucket.Upper, stream) - stream.WriteMore() - jsonutil.MarshalValue(bucket.Count, stream) - stream.WriteArrayEnd() - } - if bucketFound { - stream.WriteArrayEnd() - } - stream.WriteObjectEnd() -} - // marshalExemplarJSON writes. // // { @@ -1941,7 +1868,7 @@ func marshalExemplarJSON(ptr unsafe.Pointer, stream *jsoniter.Stream) { // "value" key. stream.WriteMore() stream.WriteObjectField(`value`) - jsonutil.MarshalValue(p.Value, stream) + jsonutil.MarshalFloat(p.Value, stream) // "timestamp" key. stream.WriteMore() diff --git a/web/api/v1/api_test.go b/web/api/v1/api_test.go index 4a1e4d67db..efce042214 100644 --- a/web/api/v1/api_test.go +++ b/web/api/v1/api_test.go @@ -1102,10 +1102,10 @@ func testEndpoints(t *testing.T, api *API, tr *testTargetRetriever, es storage.E ResultType: parser.ValueTypeMatrix, Result: promql.Matrix{ promql.Series{ - Points: []promql.Point{ - {V: 0, T: timestamp.FromTime(start)}, - {V: 1, T: timestamp.FromTime(start.Add(1 * time.Second))}, - {V: 2, T: timestamp.FromTime(start.Add(2 * time.Second))}, + Floats: []promql.FPoint{ + {F: 0, T: timestamp.FromTime(start)}, + {F: 1, T: timestamp.FromTime(start.Add(1 * time.Second))}, + {F: 2, T: timestamp.FromTime(start.Add(2 * time.Second))}, }, // No Metric returned - use zero value for comparison. }, @@ -3059,7 +3059,7 @@ func TestRespond(t *testing.T) { ResultType: parser.ValueTypeMatrix, Result: promql.Matrix{ promql.Series{ - Points: []promql.Point{{V: 1, T: 1000}}, + Floats: []promql.FPoint{{F: 1, T: 1000}}, Metric: labels.FromStrings("__name__", "foo"), }, }, @@ -3071,7 +3071,7 @@ func TestRespond(t *testing.T) { ResultType: parser.ValueTypeMatrix, Result: promql.Matrix{ promql.Series{ - Points: []promql.Point{{H: &histogram.FloatHistogram{ + Histograms: []promql.HPoint{{H: &histogram.FloatHistogram{ Schema: 2, ZeroThreshold: 0.001, ZeroCount: 12, @@ -3094,63 +3094,63 @@ func TestRespond(t *testing.T) { expected: `{"status":"success","data":{"resultType":"matrix","result":[{"metric":{"__name__":"foo"},"histograms":[[1,{"count":"10","sum":"20","buckets":[[1,"-1.6817928305074288","-1.414213562373095","1"],[1,"-1.414213562373095","-1.189207115002721","2"],[3,"-0.001","0.001","12"],[0,"1.414213562373095","1.6817928305074288","1"],[0,"1.6817928305074288","2","2"],[0,"2.378414230005442","2.82842712474619","2"],[0,"2.82842712474619","3.3635856610148576","1"],[0,"3.3635856610148576","4","1"]]}]]}]}}`, }, { - response: promql.Point{V: 0, T: 0}, + response: promql.FPoint{F: 0, T: 0}, expected: `{"status":"success","data":[0,"0"]}`, }, { - response: promql.Point{V: 20, T: 1}, + response: promql.FPoint{F: 20, T: 1}, expected: `{"status":"success","data":[0.001,"20"]}`, }, { - response: promql.Point{V: 20, T: 10}, + response: promql.FPoint{F: 20, T: 10}, expected: `{"status":"success","data":[0.010,"20"]}`, }, { - response: promql.Point{V: 20, T: 100}, + response: promql.FPoint{F: 20, T: 100}, expected: `{"status":"success","data":[0.100,"20"]}`, }, { - response: promql.Point{V: 20, T: 1001}, + response: promql.FPoint{F: 20, T: 1001}, expected: `{"status":"success","data":[1.001,"20"]}`, }, { - response: promql.Point{V: 20, T: 1010}, + response: promql.FPoint{F: 20, T: 1010}, expected: `{"status":"success","data":[1.010,"20"]}`, }, { - response: promql.Point{V: 20, T: 1100}, + response: promql.FPoint{F: 20, T: 1100}, expected: `{"status":"success","data":[1.100,"20"]}`, }, { - response: promql.Point{V: 20, T: 12345678123456555}, + response: promql.FPoint{F: 20, T: 12345678123456555}, expected: `{"status":"success","data":[12345678123456.555,"20"]}`, }, { - response: promql.Point{V: 20, T: -1}, + response: promql.FPoint{F: 20, T: -1}, expected: `{"status":"success","data":[-0.001,"20"]}`, }, { - response: promql.Point{V: math.NaN(), T: 0}, + response: promql.FPoint{F: math.NaN(), T: 0}, expected: `{"status":"success","data":[0,"NaN"]}`, }, { - response: promql.Point{V: math.Inf(1), T: 0}, + response: promql.FPoint{F: math.Inf(1), T: 0}, expected: `{"status":"success","data":[0,"+Inf"]}`, }, { - response: promql.Point{V: math.Inf(-1), T: 0}, + response: promql.FPoint{F: math.Inf(-1), T: 0}, expected: `{"status":"success","data":[0,"-Inf"]}`, }, { - response: promql.Point{V: 1.2345678e6, T: 0}, + response: promql.FPoint{F: 1.2345678e6, T: 0}, expected: `{"status":"success","data":[0,"1234567.8"]}`, }, { - response: promql.Point{V: 1.2345678e-6, T: 0}, + response: promql.FPoint{F: 1.2345678e-6, T: 0}, expected: `{"status":"success","data":[0,"0.0000012345678"]}`, }, { - response: promql.Point{V: 1.2345678e-67, T: 0}, + response: promql.FPoint{F: 1.2345678e-67, T: 0}, expected: `{"status":"success","data":[0,"1.2345678e-67"]}`, }, { @@ -3283,15 +3283,15 @@ var testResponseWriter = httptest.ResponseRecorder{} func BenchmarkRespond(b *testing.B) { b.ReportAllocs() - points := []promql.Point{} + points := []promql.FPoint{} for i := 0; i < 10000; i++ { - points = append(points, promql.Point{V: float64(i * 1000000), T: int64(i)}) + points = append(points, promql.FPoint{F: float64(i * 1000000), T: int64(i)}) } response := &queryData{ ResultType: parser.ValueTypeMatrix, Result: promql.Matrix{ promql.Series{ - Points: points, + Floats: points, Metric: labels.EmptyLabels(), }, }, diff --git a/web/federate.go b/web/federate.go index 93526771b4..ad2574e80d 100644 --- a/web/federate.go +++ b/web/federate.go @@ -145,7 +145,9 @@ Loop: vec = append(vec, promql.Sample{ Metric: s.Labels(), - Point: promql.Point{T: t, V: v, H: fh}, + T: t, + F: v, + H: fh, }) } if ws := set.Warnings(); len(ws) > 0 { @@ -262,7 +264,7 @@ Loop: if !isHistogram { lastHistogramWasGauge = false protMetric.Untyped = &dto.Untyped{ - Value: proto.Float64(s.V), + Value: proto.Float64(s.F), } } else { lastHistogramWasGauge = s.H.CounterResetHint == histogram.GaugeType diff --git a/web/federate_test.go b/web/federate_test.go index 944b0f1ec0..76d4b9cf67 100644 --- a/web/federate_test.go +++ b/web/federate_test.go @@ -342,14 +342,16 @@ func TestFederationWithNativeHistograms(t *testing.T) { if i%3 == 0 { _, err = app.Append(0, l, 100*60*1000, float64(i*100)) expVec = append(expVec, promql.Sample{ - Point: promql.Point{T: 100 * 60 * 1000, V: float64(i * 100)}, + T: 100 * 60 * 1000, + F: float64(i * 100), Metric: expL, }) } else { hist.ZeroCount++ _, err = app.AppendHistogram(0, l, 100*60*1000, hist.Copy(), nil) expVec = append(expVec, promql.Sample{ - Point: promql.Point{T: 100 * 60 * 1000, H: hist.ToFloat()}, + T: 100 * 60 * 1000, + H: hist.ToFloat(), Metric: expL, }) } @@ -379,6 +381,7 @@ func TestFederationWithNativeHistograms(t *testing.T) { p := textparse.NewProtobufParser(body) var actVec promql.Vector metricFamilies := 0 + l := labels.Labels{} for { et, err := p.Next() if err == io.EOF { @@ -389,23 +392,23 @@ func TestFederationWithNativeHistograms(t *testing.T) { metricFamilies++ } if et == textparse.EntryHistogram || et == textparse.EntrySeries { - l := labels.Labels{} p.Metric(&l) - actVec = append(actVec, promql.Sample{Metric: l}) } if et == textparse.EntryHistogram { _, parsedTimestamp, h, fh := p.Histogram() require.Nil(t, h) - actVec[len(actVec)-1].Point = promql.Point{ - T: *parsedTimestamp, - H: fh, - } + actVec = append(actVec, promql.Sample{ + T: *parsedTimestamp, + H: fh, + Metric: l, + }) } else if et == textparse.EntrySeries { - _, parsedTimestamp, v := p.Series() - actVec[len(actVec)-1].Point = promql.Point{ - T: *parsedTimestamp, - V: v, - } + _, parsedTimestamp, f := p.Series() + actVec = append(actVec, promql.Sample{ + T: *parsedTimestamp, + F: f, + Metric: l, + }) } } From 630bcb494b1828e07a1b9744d5c9d938e42ee908 Mon Sep 17 00:00:00 2001 From: beorn7 Date: Thu, 8 Dec 2022 13:31:08 +0100 Subject: [PATCH 086/231] storage: Use separate sample types for histogram vs. float MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Previously, we had one “polymorphous” `sample` type in the `storage` package. This commit breaks it up into `fSample`, `hSample`, and `fhSample`, each still implementing the `tsdbutil.Sample` interface. This reduces allocations in `sampleRing.Add` but inflicts the penalty of the interface wrapper, which makes things worse in total. This commit therefore just demonstrates the step taken. The next commit will tackle the interface overhead problem. Signed-off-by: beorn7 --- storage/buffer.go | 142 +++++++----- storage/buffer_test.go | 46 ++-- storage/memoized_iterator_test.go | 16 +- storage/merge_test.go | 350 +++++++++++++++--------------- storage/series.go | 21 +- storage/series_test.go | 32 +-- tsdb/tsdbutil/chunks.go | 2 +- web/federate.go | 24 +- 8 files changed, 342 insertions(+), 291 deletions(-) diff --git a/storage/buffer.go b/storage/buffer.go index 92767cdd78..68210facc1 100644 --- a/storage/buffer.go +++ b/storage/buffer.go @@ -19,6 +19,7 @@ import ( "github.com/prometheus/prometheus/model/histogram" "github.com/prometheus/prometheus/tsdb/chunkenc" + "github.com/prometheus/prometheus/tsdb/tsdbutil" ) // BufferedSeriesIterator wraps an iterator with a look-back buffer. @@ -68,11 +69,8 @@ func (b *BufferedSeriesIterator) ReduceDelta(delta int64) bool { // PeekBack returns the nth previous element of the iterator. If there is none buffered, // ok is false. -func (b *BufferedSeriesIterator) PeekBack(n int) ( - t int64, v float64, h *histogram.Histogram, fh *histogram.FloatHistogram, ok bool, -) { - s, ok := b.buf.nthLast(n) - return s.t, s.v, s.h, s.fh, ok +func (b *BufferedSeriesIterator) PeekBack(n int) (sample tsdbutil.Sample, ok bool) { + return b.buf.nthLast(n) } // Buffer returns an iterator over the buffered data. Invalidates previously @@ -122,14 +120,14 @@ func (b *BufferedSeriesIterator) Next() chunkenc.ValueType { case chunkenc.ValNone: return chunkenc.ValNone case chunkenc.ValFloat: - t, v := b.it.At() - b.buf.add(sample{t: t, v: v}) + t, f := b.it.At() + b.buf.add(fSample{t: t, f: f}) case chunkenc.ValHistogram: t, h := b.it.AtHistogram() - b.buf.add(sample{t: t, h: h}) + b.buf.add(hSample{t: t, h: h}) case chunkenc.ValFloatHistogram: t, fh := b.it.AtFloatHistogram() - b.buf.add(sample{t: t, fh: fh}) + b.buf.add(fhSample{t: t, fh: fh}) default: panic(fmt.Errorf("BufferedSeriesIterator: unknown value type %v", b.valueType)) } @@ -166,54 +164,94 @@ func (b *BufferedSeriesIterator) Err() error { return b.it.Err() } -// TODO(beorn7): Consider having different sample types for different value types. -type sample struct { - t int64 - v float64 - h *histogram.Histogram - fh *histogram.FloatHistogram +type fSample struct { + t int64 + f float64 } -func (s sample) T() int64 { +func (s fSample) T() int64 { return s.t } -func (s sample) V() float64 { - return s.v +func (s fSample) V() float64 { + return s.f } -func (s sample) H() *histogram.Histogram { +func (s fSample) H() *histogram.Histogram { + panic("H() called for fSample") +} + +func (s fSample) FH() *histogram.FloatHistogram { + panic("FH() called for fSample") +} + +func (s fSample) Type() chunkenc.ValueType { + return chunkenc.ValFloat +} + +type hSample struct { + t int64 + h *histogram.Histogram +} + +func (s hSample) T() int64 { + return s.t +} + +func (s hSample) V() float64 { + panic("F() called for hSample") +} + +func (s hSample) H() *histogram.Histogram { return s.h } -func (s sample) FH() *histogram.FloatHistogram { +func (s hSample) FH() *histogram.FloatHistogram { + return s.h.ToFloat() +} + +func (s hSample) Type() chunkenc.ValueType { + return chunkenc.ValHistogram +} + +type fhSample struct { + t int64 + fh *histogram.FloatHistogram +} + +func (s fhSample) T() int64 { + return s.t +} + +func (s fhSample) V() float64 { + panic("F() called for fhSample") +} + +func (s fhSample) H() *histogram.Histogram { + panic("H() called for fhSample") +} + +func (s fhSample) FH() *histogram.FloatHistogram { return s.fh } -func (s sample) Type() chunkenc.ValueType { - switch { - case s.h != nil: - return chunkenc.ValHistogram - case s.fh != nil: - return chunkenc.ValFloatHistogram - default: - return chunkenc.ValFloat - } +func (s fhSample) Type() chunkenc.ValueType { + return chunkenc.ValFloatHistogram } type sampleRing struct { delta int64 - buf []sample // lookback buffer - i int // position of most recent element in ring buffer - f int // position of first element in ring buffer - l int // number of elements in buffer + buf []tsdbutil.Sample // lookback buffer + i int // position of most recent element in ring buffer + f int // position of first element in ring buffer + l int // number of elements in buffer it sampleRingIterator } func newSampleRing(delta int64, sz int) *sampleRing { - r := &sampleRing{delta: delta, buf: make([]sample, sz)} + r := &sampleRing{delta: delta, buf: make([]tsdbutil.Sample, sz)} r.reset() return r @@ -247,16 +285,16 @@ func (it *sampleRingIterator) Next() chunkenc.ValueType { return chunkenc.ValNone } s := it.r.at(it.i) - it.t = s.t - switch { - case s.h != nil: - it.h = s.h + it.t = s.T() + switch s.Type() { + case chunkenc.ValHistogram: + it.h = s.H() return chunkenc.ValHistogram - case s.fh != nil: - it.fh = s.fh + case chunkenc.ValFloatHistogram: + it.fh = s.FH() return chunkenc.ValFloatHistogram default: - it.v = s.v + it.v = s.V() return chunkenc.ValFloat } } @@ -288,18 +326,18 @@ func (it *sampleRingIterator) AtT() int64 { return it.t } -func (r *sampleRing) at(i int) sample { +func (r *sampleRing) at(i int) tsdbutil.Sample { j := (r.f + i) % len(r.buf) return r.buf[j] } // add adds a sample to the ring buffer and frees all samples that fall // out of the delta range. -func (r *sampleRing) add(s sample) { +func (r *sampleRing) add(s tsdbutil.Sample) { l := len(r.buf) // Grow the ring buffer if it fits no more elements. if l == r.l { - buf := make([]sample, 2*l) + buf := make([]tsdbutil.Sample, 2*l) copy(buf[l+r.f:], r.buf[r.f:]) copy(buf, r.buf[:r.f]) @@ -318,8 +356,8 @@ func (r *sampleRing) add(s sample) { r.l++ // Free head of the buffer of samples that just fell out of the range. - tmin := s.t - r.delta - for r.buf[r.f].t < tmin { + tmin := s.T() - r.delta + for r.buf[r.f].T() < tmin { r.f++ if r.f >= l { r.f -= l @@ -342,8 +380,8 @@ func (r *sampleRing) reduceDelta(delta int64) bool { // Free head of the buffer of samples that just fell out of the range. l := len(r.buf) - tmin := r.buf[r.i].t - delta - for r.buf[r.f].t < tmin { + tmin := r.buf[r.i].T() - delta + for r.buf[r.f].T() < tmin { r.f++ if r.f >= l { r.f -= l @@ -354,15 +392,15 @@ func (r *sampleRing) reduceDelta(delta int64) bool { } // nthLast returns the nth most recent element added to the ring. -func (r *sampleRing) nthLast(n int) (sample, bool) { +func (r *sampleRing) nthLast(n int) (tsdbutil.Sample, bool) { if n > r.l { - return sample{}, false + return fSample{}, false } return r.at(r.l - n), true } -func (r *sampleRing) samples() []sample { - res := make([]sample, r.l) +func (r *sampleRing) samples() []tsdbutil.Sample { + res := make([]tsdbutil.Sample, r.l) k := r.f + r.l var j int diff --git a/storage/buffer_test.go b/storage/buffer_test.go index 44d11f0eda..7cfda4d4cf 100644 --- a/storage/buffer_test.go +++ b/storage/buffer_test.go @@ -58,11 +58,11 @@ func TestSampleRing(t *testing.T) { for _, c := range cases { r := newSampleRing(c.delta, c.size) - input := []sample{} + input := []fSample{} for _, t := range c.input { - input = append(input, sample{ + input = append(input, fSample{ t: t, - v: float64(rand.Intn(100)), + f: float64(rand.Intn(100)), }) } @@ -73,7 +73,7 @@ func TestSampleRing(t *testing.T) { for _, sold := range input[:i] { found := false for _, bs := range buffered { - if bs.t == sold.t && bs.v == sold.v { + if bs.T() == sold.t && bs.V() == sold.f { found = true break } @@ -92,12 +92,12 @@ func TestSampleRing(t *testing.T) { func TestBufferedSeriesIterator(t *testing.T) { var it *BufferedSeriesIterator - bufferEq := func(exp []sample) { - var b []sample + bufferEq := func(exp []fSample) { + var b []fSample bit := it.Buffer() for bit.Next() == chunkenc.ValFloat { - t, v := bit.At() - b = append(b, sample{t: t, v: v}) + t, f := bit.At() + b = append(b, fSample{t: t, f: f}) } require.Equal(t, exp, b, "buffer mismatch") } @@ -107,21 +107,21 @@ func TestBufferedSeriesIterator(t *testing.T) { require.Equal(t, ev, v, "value mismatch") } prevSampleEq := func(ets int64, ev float64, eok bool) { - ts, v, _, _, ok := it.PeekBack(1) + s, ok := it.PeekBack(1) require.Equal(t, eok, ok, "exist mismatch") - require.Equal(t, ets, ts, "timestamp mismatch") - require.Equal(t, ev, v, "value mismatch") + require.Equal(t, ets, s.T(), "timestamp mismatch") + require.Equal(t, ev, s.V(), "value mismatch") } it = NewBufferIterator(NewListSeriesIterator(samples{ - sample{t: 1, v: 2}, - sample{t: 2, v: 3}, - sample{t: 3, v: 4}, - sample{t: 4, v: 5}, - sample{t: 5, v: 6}, - sample{t: 99, v: 8}, - sample{t: 100, v: 9}, - sample{t: 101, v: 10}, + fSample{t: 1, f: 2}, + fSample{t: 2, f: 3}, + fSample{t: 3, f: 4}, + fSample{t: 4, f: 5}, + fSample{t: 5, f: 6}, + fSample{t: 99, f: 8}, + fSample{t: 100, f: 9}, + fSample{t: 101, f: 10}, }), 2) require.Equal(t, chunkenc.ValFloat, it.Seek(-123), "seek failed") @@ -132,24 +132,24 @@ func TestBufferedSeriesIterator(t *testing.T) { require.Equal(t, chunkenc.ValFloat, it.Next(), "next failed") sampleEq(2, 3) prevSampleEq(1, 2, true) - bufferEq([]sample{{t: 1, v: 2}}) + bufferEq([]fSample{{t: 1, f: 2}}) require.Equal(t, chunkenc.ValFloat, it.Next(), "next failed") require.Equal(t, chunkenc.ValFloat, it.Next(), "next failed") require.Equal(t, chunkenc.ValFloat, it.Next(), "next failed") sampleEq(5, 6) prevSampleEq(4, 5, true) - bufferEq([]sample{{t: 2, v: 3}, {t: 3, v: 4}, {t: 4, v: 5}}) + bufferEq([]fSample{{t: 2, f: 3}, {t: 3, f: 4}, {t: 4, f: 5}}) require.Equal(t, chunkenc.ValFloat, it.Seek(5), "seek failed") sampleEq(5, 6) prevSampleEq(4, 5, true) - bufferEq([]sample{{t: 2, v: 3}, {t: 3, v: 4}, {t: 4, v: 5}}) + bufferEq([]fSample{{t: 2, f: 3}, {t: 3, f: 4}, {t: 4, f: 5}}) require.Equal(t, chunkenc.ValFloat, it.Seek(101), "seek failed") sampleEq(101, 10) prevSampleEq(100, 9, true) - bufferEq([]sample{{t: 99, v: 8}, {t: 100, v: 9}}) + bufferEq([]fSample{{t: 99, f: 8}, {t: 100, f: 9}}) require.Equal(t, chunkenc.ValNone, it.Next(), "next succeeded unexpectedly") require.Equal(t, chunkenc.ValNone, it.Seek(1024), "seek succeeded unexpectedly") diff --git a/storage/memoized_iterator_test.go b/storage/memoized_iterator_test.go index 6bdd395e09..382c84e63b 100644 --- a/storage/memoized_iterator_test.go +++ b/storage/memoized_iterator_test.go @@ -38,14 +38,14 @@ func TestMemoizedSeriesIterator(t *testing.T) { } it = NewMemoizedIterator(NewListSeriesIterator(samples{ - sample{t: 1, v: 2}, - sample{t: 2, v: 3}, - sample{t: 3, v: 4}, - sample{t: 4, v: 5}, - sample{t: 5, v: 6}, - sample{t: 99, v: 8}, - sample{t: 100, v: 9}, - sample{t: 101, v: 10}, + fSample{t: 1, f: 2}, + fSample{t: 2, f: 3}, + fSample{t: 3, f: 4}, + fSample{t: 4, f: 5}, + fSample{t: 5, f: 6}, + fSample{t: 99, f: 8}, + fSample{t: 100, f: 9}, + fSample{t: 101, f: 10}, }), 2) require.Equal(t, it.Seek(-123), chunkenc.ValFloat, "seek failed") diff --git a/storage/merge_test.go b/storage/merge_test.go index 6abf8723d1..b0544c2d81 100644 --- a/storage/merge_test.go +++ b/storage/merge_test.go @@ -62,116 +62,116 @@ func TestMergeQuerierWithChainMerger(t *testing.T) { { name: "one querier, two series", querierSeries: [][]Series{{ - NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}}), - NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}), + NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}, fSample{3, 3}}), + NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}, fSample{2, 2}}), }}, expected: NewMockSeriesSet( - NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}}), - NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}), + NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}, fSample{3, 3}}), + NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}, fSample{2, 2}}), ), }, { name: "two queriers, one different series each", querierSeries: [][]Series{{ - NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}}), + NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}, fSample{3, 3}}), }, { - NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}), + NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}, fSample{2, 2}}), }}, expected: NewMockSeriesSet( - NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}}), - NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}), + NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}, fSample{3, 3}}), + NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}, fSample{2, 2}}), ), }, { name: "two time unsorted queriers, two series each", querierSeries: [][]Series{{ - NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{5, 5, nil, nil}, sample{6, 6, nil, nil}}), - NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}), + NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{5, 5}, fSample{6, 6}}), + NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}, fSample{2, 2}}), }, { - NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}}), - NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{3, 3, nil, nil}, sample{4, 4, nil, nil}}), + NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}, fSample{3, 3}}), + NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{3, 3}, fSample{4, 4}}), }}, expected: NewMockSeriesSet( NewListSeries( labels.FromStrings("bar", "baz"), - []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{5, 5, nil, nil}, sample{6, 6, nil, nil}}, + []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}, fSample{3, 3}, fSample{5, 5}, fSample{6, 6}}, ), NewListSeries( labels.FromStrings("foo", "bar"), - []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{4, 4, nil, nil}}, + []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}, fSample{2, 2}, fSample{3, 3}, fSample{4, 4}}, ), ), }, { name: "five queriers, only two queriers have two time unsorted series each", querierSeries: [][]Series{{}, {}, { - NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{5, 5, nil, nil}, sample{6, 6, nil, nil}}), - NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}), + NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{5, 5}, fSample{6, 6}}), + NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}, fSample{2, 2}}), }, { - NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}}), - NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{3, 3, nil, nil}, sample{4, 4, nil, nil}}), + NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}, fSample{3, 3}}), + NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{3, 3}, fSample{4, 4}}), }, {}}, expected: NewMockSeriesSet( NewListSeries( labels.FromStrings("bar", "baz"), - []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{5, 5, nil, nil}, sample{6, 6, nil, nil}}, + []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}, fSample{3, 3}, fSample{5, 5}, fSample{6, 6}}, ), NewListSeries( labels.FromStrings("foo", "bar"), - []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{4, 4, nil, nil}}, + []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}, fSample{2, 2}, fSample{3, 3}, fSample{4, 4}}, ), ), }, { name: "two queriers, only two queriers have two time unsorted series each, with 3 noop and one nil querier together", querierSeries: [][]Series{{}, {}, { - NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{5, 5, nil, nil}, sample{6, 6, nil, nil}}), - NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}), + NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{5, 5}, fSample{6, 6}}), + NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}, fSample{2, 2}}), }, { - NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}}), - NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{3, 3, nil, nil}, sample{4, 4, nil, nil}}), + NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}, fSample{3, 3}}), + NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{3, 3}, fSample{4, 4}}), }, {}}, extraQueriers: []Querier{NoopQuerier(), NoopQuerier(), nil, NoopQuerier()}, expected: NewMockSeriesSet( NewListSeries( labels.FromStrings("bar", "baz"), - []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{5, 5, nil, nil}, sample{6, 6, nil, nil}}, + []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}, fSample{3, 3}, fSample{5, 5}, fSample{6, 6}}, ), NewListSeries( labels.FromStrings("foo", "bar"), - []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{4, 4, nil, nil}}, + []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}, fSample{2, 2}, fSample{3, 3}, fSample{4, 4}}, ), ), }, { name: "two queriers, with two series, one is overlapping", querierSeries: [][]Series{{}, {}, { - NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{2, 21, nil, nil}, sample{3, 31, nil, nil}, sample{5, 5, nil, nil}, sample{6, 6, nil, nil}}), - NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}), + NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{2, 21}, fSample{3, 31}, fSample{5, 5}, fSample{6, 6}}), + NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}, fSample{2, 2}}), }, { - NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 22, nil, nil}, sample{3, 32, nil, nil}}), - NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{3, 3, nil, nil}, sample{4, 4, nil, nil}}), + NewListSeries(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 22}, fSample{3, 32}}), + NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{3, 3}, fSample{4, 4}}), }, {}}, expected: NewMockSeriesSet( NewListSeries( labels.FromStrings("bar", "baz"), - []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 21, nil, nil}, sample{3, 31, nil, nil}, sample{5, 5, nil, nil}, sample{6, 6, nil, nil}}, + []tsdbutil.Sample{fSample{1, 1}, fSample{2, 21}, fSample{3, 31}, fSample{5, 5}, fSample{6, 6}}, ), NewListSeries( labels.FromStrings("foo", "bar"), - []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{4, 4, nil, nil}}, + []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}, fSample{2, 2}, fSample{3, 3}, fSample{4, 4}}, ), ), }, { name: "two queries, one with NaN samples series", querierSeries: [][]Series{{ - NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{0, math.NaN(), nil, nil}}), + NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{0, math.NaN()}}), }, { - NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{1, 1, nil, nil}}), + NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{1, 1}}), }}, expected: NewMockSeriesSet( - NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{0, math.NaN(), nil, nil}, sample{1, 1, nil, nil}}), + NewListSeries(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{0, math.NaN()}, fSample{1, 1}}), ), }, } { @@ -245,108 +245,108 @@ func TestMergeChunkQuerierWithNoVerticalChunkSeriesMerger(t *testing.T) { { name: "one querier, two series", chkQuerierSeries: [][]ChunkSeries{{ - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, []tsdbutil.Sample{sample{3, 3, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}}, []tsdbutil.Sample{sample{2, 2, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, []tsdbutil.Sample{fSample{3, 3}}), + NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}}, []tsdbutil.Sample{fSample{2, 2}}), }}, expected: NewMockChunkSeriesSet( - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, []tsdbutil.Sample{sample{3, 3, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}}, []tsdbutil.Sample{sample{2, 2, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, []tsdbutil.Sample{fSample{3, 3}}), + NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}}, []tsdbutil.Sample{fSample{2, 2}}), ), }, { name: "two secondaries, one different series each", chkQuerierSeries: [][]ChunkSeries{{ - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, []tsdbutil.Sample{sample{3, 3, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, []tsdbutil.Sample{fSample{3, 3}}), }, { - NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}}, []tsdbutil.Sample{sample{2, 2, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}}, []tsdbutil.Sample{fSample{2, 2}}), }}, expected: NewMockChunkSeriesSet( - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, []tsdbutil.Sample{sample{3, 3, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}}, []tsdbutil.Sample{sample{2, 2, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, []tsdbutil.Sample{fSample{3, 3}}), + NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}}, []tsdbutil.Sample{fSample{2, 2}}), ), }, { name: "two secondaries, two not in time order series each", chkQuerierSeries: [][]ChunkSeries{{ - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{5, 5, nil, nil}}, []tsdbutil.Sample{sample{6, 6, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}}, []tsdbutil.Sample{sample{2, 2, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{5, 5}}, []tsdbutil.Sample{fSample{6, 6}}), + NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}}, []tsdbutil.Sample{fSample{2, 2}}), }, { - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, []tsdbutil.Sample{sample{3, 3, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{3, 3, nil, nil}}, []tsdbutil.Sample{sample{4, 4, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, []tsdbutil.Sample{fSample{3, 3}}), + NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{3, 3}}, []tsdbutil.Sample{fSample{4, 4}}), }}, expected: NewMockChunkSeriesSet( NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), - []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, - []tsdbutil.Sample{sample{3, 3, nil, nil}}, - []tsdbutil.Sample{sample{5, 5, nil, nil}}, - []tsdbutil.Sample{sample{6, 6, nil, nil}}, + []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, + []tsdbutil.Sample{fSample{3, 3}}, + []tsdbutil.Sample{fSample{5, 5}}, + []tsdbutil.Sample{fSample{6, 6}}, ), NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), - []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}}, - []tsdbutil.Sample{sample{2, 2, nil, nil}}, - []tsdbutil.Sample{sample{3, 3, nil, nil}}, - []tsdbutil.Sample{sample{4, 4, nil, nil}}, + []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}}, + []tsdbutil.Sample{fSample{2, 2}}, + []tsdbutil.Sample{fSample{3, 3}}, + []tsdbutil.Sample{fSample{4, 4}}, ), ), }, { name: "five secondaries, only two have two not in time order series each", chkQuerierSeries: [][]ChunkSeries{{}, {}, { - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{5, 5, nil, nil}}, []tsdbutil.Sample{sample{6, 6, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}}, []tsdbutil.Sample{sample{2, 2, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{5, 5}}, []tsdbutil.Sample{fSample{6, 6}}), + NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}}, []tsdbutil.Sample{fSample{2, 2}}), }, { - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, []tsdbutil.Sample{sample{3, 3, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{3, 3, nil, nil}}, []tsdbutil.Sample{sample{4, 4, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, []tsdbutil.Sample{fSample{3, 3}}), + NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{3, 3}}, []tsdbutil.Sample{fSample{4, 4}}), }, {}}, expected: NewMockChunkSeriesSet( NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), - []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, - []tsdbutil.Sample{sample{3, 3, nil, nil}}, - []tsdbutil.Sample{sample{5, 5, nil, nil}}, - []tsdbutil.Sample{sample{6, 6, nil, nil}}, + []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, + []tsdbutil.Sample{fSample{3, 3}}, + []tsdbutil.Sample{fSample{5, 5}}, + []tsdbutil.Sample{fSample{6, 6}}, ), NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), - []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}}, - []tsdbutil.Sample{sample{2, 2, nil, nil}}, - []tsdbutil.Sample{sample{3, 3, nil, nil}}, - []tsdbutil.Sample{sample{4, 4, nil, nil}}, + []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}}, + []tsdbutil.Sample{fSample{2, 2}}, + []tsdbutil.Sample{fSample{3, 3}}, + []tsdbutil.Sample{fSample{4, 4}}, ), ), }, { name: "two secondaries, with two not in time order series each, with 3 noop queries and one nil together", chkQuerierSeries: [][]ChunkSeries{{ - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{5, 5, nil, nil}}, []tsdbutil.Sample{sample{6, 6, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}}, []tsdbutil.Sample{sample{2, 2, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{5, 5}}, []tsdbutil.Sample{fSample{6, 6}}), + NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}}, []tsdbutil.Sample{fSample{2, 2}}), }, { - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, []tsdbutil.Sample{sample{3, 3, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{3, 3, nil, nil}}, []tsdbutil.Sample{sample{4, 4, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, []tsdbutil.Sample{fSample{3, 3}}), + NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{3, 3}}, []tsdbutil.Sample{fSample{4, 4}}), }}, extraQueriers: []ChunkQuerier{NoopChunkedQuerier(), NoopChunkedQuerier(), nil, NoopChunkedQuerier()}, expected: NewMockChunkSeriesSet( NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), - []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, - []tsdbutil.Sample{sample{3, 3, nil, nil}}, - []tsdbutil.Sample{sample{5, 5, nil, nil}}, - []tsdbutil.Sample{sample{6, 6, nil, nil}}, + []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, + []tsdbutil.Sample{fSample{3, 3}}, + []tsdbutil.Sample{fSample{5, 5}}, + []tsdbutil.Sample{fSample{6, 6}}, ), NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), - []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}}, - []tsdbutil.Sample{sample{2, 2, nil, nil}}, - []tsdbutil.Sample{sample{3, 3, nil, nil}}, - []tsdbutil.Sample{sample{4, 4, nil, nil}}, + []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}}, + []tsdbutil.Sample{fSample{2, 2}}, + []tsdbutil.Sample{fSample{3, 3}}, + []tsdbutil.Sample{fSample{4, 4}}, ), ), }, { name: "two queries, one with NaN samples series", chkQuerierSeries: [][]ChunkSeries{{ - NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{0, math.NaN(), nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{0, math.NaN()}}), }, { - NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{1, 1, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{1, 1}}), }}, expected: NewMockChunkSeriesSet( - NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{sample{0, math.NaN(), nil, nil}}, []tsdbutil.Sample{sample{1, 1, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("foo", "bar"), []tsdbutil.Sample{fSample{0, math.NaN()}}, []tsdbutil.Sample{fSample{1, 1}}), ), }, } { @@ -385,12 +385,12 @@ func TestCompactingChunkSeriesMerger(t *testing.T) { m := NewCompactingChunkSeriesMerger(ChainedSeriesMerge) // histogramSample returns a histogram that is unique to the ts. - histogramSample := func(ts int64) sample { - return sample{t: ts, h: tsdbutil.GenerateTestHistogram(int(ts + 1))} + histogramSample := func(ts int64) hSample { + return hSample{t: ts, h: tsdbutil.GenerateTestHistogram(int(ts + 1))} } - floatHistogramSample := func(ts int64) sample { - return sample{t: ts, fh: tsdbutil.GenerateTestFloatHistogram(int(ts + 1))} + floatHistogramSample := func(ts int64) fhSample { + return fhSample{t: ts, fh: tsdbutil.GenerateTestFloatHistogram(int(ts + 1))} } for _, tc := range []struct { @@ -408,9 +408,9 @@ func TestCompactingChunkSeriesMerger(t *testing.T) { { name: "single series", input: []ChunkSeries{ - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, []tsdbutil.Sample{sample{3, 3, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, []tsdbutil.Sample{fSample{3, 3}}), }, - expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, []tsdbutil.Sample{sample{3, 3, nil, nil}}), + expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, []tsdbutil.Sample{fSample{3, 3}}), }, { name: "two empty series", @@ -423,55 +423,55 @@ func TestCompactingChunkSeriesMerger(t *testing.T) { { name: "two non overlapping", input: []ChunkSeries{ - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, []tsdbutil.Sample{sample{3, 3, nil, nil}, sample{5, 5, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{7, 7, nil, nil}, sample{9, 9, nil, nil}}, []tsdbutil.Sample{sample{10, 10, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, []tsdbutil.Sample{fSample{3, 3}, fSample{5, 5}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{7, 7}, fSample{9, 9}}, []tsdbutil.Sample{fSample{10, 10}}), }, - expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, []tsdbutil.Sample{sample{3, 3, nil, nil}, sample{5, 5, nil, nil}}, []tsdbutil.Sample{sample{7, 7, nil, nil}, sample{9, 9, nil, nil}}, []tsdbutil.Sample{sample{10, 10, nil, nil}}), + expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, []tsdbutil.Sample{fSample{3, 3}, fSample{5, 5}}, []tsdbutil.Sample{fSample{7, 7}, fSample{9, 9}}, []tsdbutil.Sample{fSample{10, 10}}), }, { name: "two overlapping", input: []ChunkSeries{ - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, []tsdbutil.Sample{sample{3, 3, nil, nil}, sample{8, 8, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{7, 7, nil, nil}, sample{9, 9, nil, nil}}, []tsdbutil.Sample{sample{10, 10, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, []tsdbutil.Sample{fSample{3, 3}, fSample{8, 8}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{7, 7}, fSample{9, 9}}, []tsdbutil.Sample{fSample{10, 10}}), }, - expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, []tsdbutil.Sample{sample{3, 3, nil, nil}, sample{7, 7, nil, nil}, sample{8, 8, nil, nil}, sample{9, 9, nil, nil}}, []tsdbutil.Sample{sample{10, 10, nil, nil}}), + expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, []tsdbutil.Sample{fSample{3, 3}, fSample{7, 7}, fSample{8, 8}, fSample{9, 9}}, []tsdbutil.Sample{fSample{10, 10}}), }, { name: "two duplicated", input: []ChunkSeries{ - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{5, 5, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{5, 5, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}, fSample{3, 3}, fSample{5, 5}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{2, 2}, fSample{3, 3}, fSample{5, 5}}), }, - expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{5, 5, nil, nil}}), + expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}, fSample{3, 3}, fSample{5, 5}}), }, { name: "three overlapping", input: []ChunkSeries{ - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{5, 5, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{6, 6, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{4, 4, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}, fSample{3, 3}, fSample{5, 5}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{2, 2}, fSample{3, 3}, fSample{6, 6}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{0, 0}, fSample{4, 4}}), }, - expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{4, 4, nil, nil}, sample{5, 5, nil, nil}, sample{6, 6, nil, nil}}), + expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}, fSample{2, 2}, fSample{3, 3}, fSample{4, 4}, fSample{5, 5}, fSample{6, 6}}), }, { name: "three in chained overlap", input: []ChunkSeries{ - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{5, 5, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{4, 4, nil, nil}, sample{6, 66, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{6, 6, nil, nil}, sample{10, 10, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}, fSample{3, 3}, fSample{5, 5}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{4, 4}, fSample{6, 66}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{6, 6}, fSample{10, 10}}), }, - expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{4, 4, nil, nil}, sample{5, 5, nil, nil}, sample{6, 66, nil, nil}, sample{10, 10, nil, nil}}), + expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}, fSample{3, 3}, fSample{4, 4}, fSample{5, 5}, fSample{6, 66}, fSample{10, 10}}), }, { name: "three in chained overlap complex", input: []ChunkSeries{ - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{5, 5, nil, nil}}, []tsdbutil.Sample{sample{10, 10, nil, nil}, sample{15, 15, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{2, 2, nil, nil}, sample{20, 20, nil, nil}}, []tsdbutil.Sample{sample{25, 25, nil, nil}, sample{30, 30, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{18, 18, nil, nil}, sample{26, 26, nil, nil}}, []tsdbutil.Sample{sample{31, 31, nil, nil}, sample{35, 35, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{0, 0}, fSample{5, 5}}, []tsdbutil.Sample{fSample{10, 10}, fSample{15, 15}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{2, 2}, fSample{20, 20}}, []tsdbutil.Sample{fSample{25, 25}, fSample{30, 30}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{18, 18}, fSample{26, 26}}, []tsdbutil.Sample{fSample{31, 31}, fSample{35, 35}}), }, expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), - []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{2, 2, nil, nil}, sample{5, 5, nil, nil}, sample{10, 10, nil, nil}, sample{15, 15, nil, nil}, sample{18, 18, nil, nil}, sample{20, 20, nil, nil}, sample{25, 25, nil, nil}, sample{26, 26, nil, nil}, sample{30, 30, nil, nil}}, - []tsdbutil.Sample{sample{31, 31, nil, nil}, sample{35, 35, nil, nil}}, + []tsdbutil.Sample{fSample{0, 0}, fSample{2, 2}, fSample{5, 5}, fSample{10, 10}, fSample{15, 15}, fSample{18, 18}, fSample{20, 20}, fSample{25, 25}, fSample{26, 26}, fSample{30, 30}}, + []tsdbutil.Sample{fSample{31, 31}, fSample{35, 35}}, ), }, { @@ -511,13 +511,13 @@ func TestCompactingChunkSeriesMerger(t *testing.T) { name: "histogram chunks overlapping with float chunks", input: []ChunkSeries{ NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{histogramSample(0), histogramSample(5)}, []tsdbutil.Sample{histogramSample(10), histogramSample(15)}), - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{12, 12, nil, nil}}, []tsdbutil.Sample{sample{14, 14, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{12, 12}}, []tsdbutil.Sample{fSample{14, 14}}), }, expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{histogramSample(0)}, - []tsdbutil.Sample{sample{1, 1, nil, nil}}, + []tsdbutil.Sample{fSample{1, 1}}, []tsdbutil.Sample{histogramSample(5), histogramSample(10)}, - []tsdbutil.Sample{sample{12, 12, nil, nil}, sample{14, 14, nil, nil}}, + []tsdbutil.Sample{fSample{12, 12}, fSample{14, 14}}, []tsdbutil.Sample{histogramSample(15)}, ), }, @@ -537,13 +537,13 @@ func TestCompactingChunkSeriesMerger(t *testing.T) { name: "float histogram chunks overlapping with float chunks", input: []ChunkSeries{ NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{floatHistogramSample(0), floatHistogramSample(5)}, []tsdbutil.Sample{floatHistogramSample(10), floatHistogramSample(15)}), - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{12, 12, nil, nil}}, []tsdbutil.Sample{sample{14, 14, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{12, 12}}, []tsdbutil.Sample{fSample{14, 14}}), }, expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{floatHistogramSample(0)}, - []tsdbutil.Sample{sample{1, 1, nil, nil}}, + []tsdbutil.Sample{fSample{1, 1}}, []tsdbutil.Sample{floatHistogramSample(5), floatHistogramSample(10)}, - []tsdbutil.Sample{sample{12, 12, nil, nil}, sample{14, 14, nil, nil}}, + []tsdbutil.Sample{fSample{12, 12}, fSample{14, 14}}, []tsdbutil.Sample{floatHistogramSample(15)}, ), }, @@ -592,9 +592,9 @@ func TestConcatenatingChunkSeriesMerger(t *testing.T) { { name: "single series", input: []ChunkSeries{ - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, []tsdbutil.Sample{sample{3, 3, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, []tsdbutil.Sample{fSample{3, 3}}), }, - expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, []tsdbutil.Sample{sample{3, 3, nil, nil}}), + expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, []tsdbutil.Sample{fSample{3, 3}}), }, { name: "two empty series", @@ -607,70 +607,70 @@ func TestConcatenatingChunkSeriesMerger(t *testing.T) { { name: "two non overlapping", input: []ChunkSeries{ - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, []tsdbutil.Sample{sample{3, 3, nil, nil}, sample{5, 5, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{7, 7, nil, nil}, sample{9, 9, nil, nil}}, []tsdbutil.Sample{sample{10, 10, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, []tsdbutil.Sample{fSample{3, 3}, fSample{5, 5}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{7, 7}, fSample{9, 9}}, []tsdbutil.Sample{fSample{10, 10}}), }, - expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, []tsdbutil.Sample{sample{3, 3, nil, nil}, sample{5, 5, nil, nil}}, []tsdbutil.Sample{sample{7, 7, nil, nil}, sample{9, 9, nil, nil}}, []tsdbutil.Sample{sample{10, 10, nil, nil}}), + expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, []tsdbutil.Sample{fSample{3, 3}, fSample{5, 5}}, []tsdbutil.Sample{fSample{7, 7}, fSample{9, 9}}, []tsdbutil.Sample{fSample{10, 10}}), }, { name: "two overlapping", input: []ChunkSeries{ - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, []tsdbutil.Sample{sample{3, 3, nil, nil}, sample{8, 8, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{7, 7, nil, nil}, sample{9, 9, nil, nil}}, []tsdbutil.Sample{sample{10, 10, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, []tsdbutil.Sample{fSample{3, 3}, fSample{8, 8}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{7, 7}, fSample{9, 9}}, []tsdbutil.Sample{fSample{10, 10}}), }, expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), - []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, []tsdbutil.Sample{sample{3, 3, nil, nil}, sample{8, 8, nil, nil}}, - []tsdbutil.Sample{sample{7, 7, nil, nil}, sample{9, 9, nil, nil}}, []tsdbutil.Sample{sample{10, 10, nil, nil}}, + []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, []tsdbutil.Sample{fSample{3, 3}, fSample{8, 8}}, + []tsdbutil.Sample{fSample{7, 7}, fSample{9, 9}}, []tsdbutil.Sample{fSample{10, 10}}, ), }, { name: "two duplicated", input: []ChunkSeries{ - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{5, 5, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{5, 5, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}, fSample{3, 3}, fSample{5, 5}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{2, 2}, fSample{3, 3}, fSample{5, 5}}), }, expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), - []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{5, 5, nil, nil}}, - []tsdbutil.Sample{sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{5, 5, nil, nil}}, + []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}, fSample{3, 3}, fSample{5, 5}}, + []tsdbutil.Sample{fSample{2, 2}, fSample{3, 3}, fSample{5, 5}}, ), }, { name: "three overlapping", input: []ChunkSeries{ - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{5, 5, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{6, 6, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{4, 4, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}, fSample{3, 3}, fSample{5, 5}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{2, 2}, fSample{3, 3}, fSample{6, 6}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{0, 0}, fSample{4, 4}}), }, expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), - []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{5, 5, nil, nil}}, - []tsdbutil.Sample{sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{6, 6, nil, nil}}, - []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{4, 4, nil, nil}}, + []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}, fSample{3, 3}, fSample{5, 5}}, + []tsdbutil.Sample{fSample{2, 2}, fSample{3, 3}, fSample{6, 6}}, + []tsdbutil.Sample{fSample{0, 0}, fSample{4, 4}}, ), }, { name: "three in chained overlap", input: []ChunkSeries{ - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{5, 5, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{4, 4, nil, nil}, sample{6, 66, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{6, 6, nil, nil}, sample{10, 10, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}, fSample{3, 3}, fSample{5, 5}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{4, 4}, fSample{6, 66}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{6, 6}, fSample{10, 10}}), }, expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), - []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{5, 5, nil, nil}}, - []tsdbutil.Sample{sample{4, 4, nil, nil}, sample{6, 66, nil, nil}}, - []tsdbutil.Sample{sample{6, 6, nil, nil}, sample{10, 10, nil, nil}}, + []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}, fSample{3, 3}, fSample{5, 5}}, + []tsdbutil.Sample{fSample{4, 4}, fSample{6, 66}}, + []tsdbutil.Sample{fSample{6, 6}, fSample{10, 10}}, ), }, { name: "three in chained overlap complex", input: []ChunkSeries{ - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{5, 5, nil, nil}}, []tsdbutil.Sample{sample{10, 10, nil, nil}, sample{15, 15, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{2, 2, nil, nil}, sample{20, 20, nil, nil}}, []tsdbutil.Sample{sample{25, 25, nil, nil}, sample{30, 30, nil, nil}}), - NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{sample{18, 18, nil, nil}, sample{26, 26, nil, nil}}, []tsdbutil.Sample{sample{31, 31, nil, nil}, sample{35, 35, nil, nil}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{0, 0}, fSample{5, 5}}, []tsdbutil.Sample{fSample{10, 10}, fSample{15, 15}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{2, 2}, fSample{20, 20}}, []tsdbutil.Sample{fSample{25, 25}, fSample{30, 30}}), + NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), []tsdbutil.Sample{fSample{18, 18}, fSample{26, 26}}, []tsdbutil.Sample{fSample{31, 31}, fSample{35, 35}}), }, expected: NewListChunkSeriesFromSamples(labels.FromStrings("bar", "baz"), - []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{5, 5, nil, nil}}, []tsdbutil.Sample{sample{10, 10, nil, nil}, sample{15, 15, nil, nil}}, - []tsdbutil.Sample{sample{2, 2, nil, nil}, sample{20, 20, nil, nil}}, []tsdbutil.Sample{sample{25, 25, nil, nil}, sample{30, 30, nil, nil}}, - []tsdbutil.Sample{sample{18, 18, nil, nil}, sample{26, 26, nil, nil}}, []tsdbutil.Sample{sample{31, 31, nil, nil}, sample{35, 35, nil, nil}}, + []tsdbutil.Sample{fSample{0, 0}, fSample{5, 5}}, []tsdbutil.Sample{fSample{10, 10}, fSample{15, 15}}, + []tsdbutil.Sample{fSample{2, 2}, fSample{20, 20}}, []tsdbutil.Sample{fSample{25, 25}, fSample{30, 30}}, + []tsdbutil.Sample{fSample{18, 18}, fSample{26, 26}}, []tsdbutil.Sample{fSample{31, 31}, fSample{35, 35}}, ), }, { @@ -807,38 +807,38 @@ func TestChainSampleIterator(t *testing.T) { }{ { input: []chunkenc.Iterator{ - NewListSeriesIterator(samples{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}}), + NewListSeriesIterator(samples{fSample{0, 0}, fSample{1, 1}}), }, - expected: []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}}, + expected: []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}}, }, { input: []chunkenc.Iterator{ - NewListSeriesIterator(samples{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}}), - NewListSeriesIterator(samples{sample{2, 2, nil, nil}, sample{3, 3, nil, nil}}), + NewListSeriesIterator(samples{fSample{0, 0}, fSample{1, 1}}), + NewListSeriesIterator(samples{fSample{2, 2}, fSample{3, 3}}), }, - expected: []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}}, + expected: []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}, fSample{2, 2}, fSample{3, 3}}, }, { input: []chunkenc.Iterator{ - NewListSeriesIterator(samples{sample{0, 0, nil, nil}, sample{3, 3, nil, nil}}), - NewListSeriesIterator(samples{sample{1, 1, nil, nil}, sample{4, 4, nil, nil}}), - NewListSeriesIterator(samples{sample{2, 2, nil, nil}, sample{5, 5, nil, nil}}), + NewListSeriesIterator(samples{fSample{0, 0}, fSample{3, 3}}), + NewListSeriesIterator(samples{fSample{1, 1}, fSample{4, 4}}), + NewListSeriesIterator(samples{fSample{2, 2}, fSample{5, 5}}), }, expected: []tsdbutil.Sample{ - sample{0, 0, nil, nil}, sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{4, 4, nil, nil}, sample{5, 5, nil, nil}, + fSample{0, 0}, fSample{1, 1}, fSample{2, 2}, fSample{3, 3}, fSample{4, 4}, fSample{5, 5}, }, }, // Overlap. { input: []chunkenc.Iterator{ - NewListSeriesIterator(samples{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}}), - NewListSeriesIterator(samples{sample{0, 0, nil, nil}, sample{2, 2, nil, nil}}), - NewListSeriesIterator(samples{sample{2, 2, nil, nil}, sample{3, 3, nil, nil}}), + NewListSeriesIterator(samples{fSample{0, 0}, fSample{1, 1}}), + NewListSeriesIterator(samples{fSample{0, 0}, fSample{2, 2}}), + NewListSeriesIterator(samples{fSample{2, 2}, fSample{3, 3}}), NewListSeriesIterator(samples{}), NewListSeriesIterator(samples{}), NewListSeriesIterator(samples{}), }, - expected: []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}}, + expected: []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}, fSample{2, 2}, fSample{3, 3}}, }, } { merged := ChainSampleIteratorFromIterators(nil, tc.input) @@ -856,42 +856,42 @@ func TestChainSampleIteratorSeek(t *testing.T) { }{ { input: []chunkenc.Iterator{ - NewListSeriesIterator(samples{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}), + NewListSeriesIterator(samples{fSample{0, 0}, fSample{1, 1}, fSample{2, 2}}), }, seek: 1, - expected: []tsdbutil.Sample{sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}, + expected: []tsdbutil.Sample{fSample{1, 1}, fSample{2, 2}}, }, { input: []chunkenc.Iterator{ - NewListSeriesIterator(samples{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}}), - NewListSeriesIterator(samples{sample{2, 2, nil, nil}, sample{3, 3, nil, nil}}), + NewListSeriesIterator(samples{fSample{0, 0}, fSample{1, 1}}), + NewListSeriesIterator(samples{fSample{2, 2}, fSample{3, 3}}), }, seek: 2, - expected: []tsdbutil.Sample{sample{2, 2, nil, nil}, sample{3, 3, nil, nil}}, + expected: []tsdbutil.Sample{fSample{2, 2}, fSample{3, 3}}, }, { input: []chunkenc.Iterator{ - NewListSeriesIterator(samples{sample{0, 0, nil, nil}, sample{3, 3, nil, nil}}), - NewListSeriesIterator(samples{sample{1, 1, nil, nil}, sample{4, 4, nil, nil}}), - NewListSeriesIterator(samples{sample{2, 2, nil, nil}, sample{5, 5, nil, nil}}), + NewListSeriesIterator(samples{fSample{0, 0}, fSample{3, 3}}), + NewListSeriesIterator(samples{fSample{1, 1}, fSample{4, 4}}), + NewListSeriesIterator(samples{fSample{2, 2}, fSample{5, 5}}), }, seek: 2, - expected: []tsdbutil.Sample{sample{2, 2, nil, nil}, sample{3, 3, nil, nil}, sample{4, 4, nil, nil}, sample{5, 5, nil, nil}}, + expected: []tsdbutil.Sample{fSample{2, 2}, fSample{3, 3}, fSample{4, 4}, fSample{5, 5}}, }, { input: []chunkenc.Iterator{ - NewListSeriesIterator(samples{sample{0, 0, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}}), - NewListSeriesIterator(samples{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}, sample{2, 2, nil, nil}}), + NewListSeriesIterator(samples{fSample{0, 0}, fSample{2, 2}, fSample{3, 3}}), + NewListSeriesIterator(samples{fSample{0, 0}, fSample{1, 1}, fSample{2, 2}}), }, seek: 0, - expected: []tsdbutil.Sample{sample{0, 0, nil, nil}, sample{1, 1, nil, nil}, sample{2, 2, nil, nil}, sample{3, 3, nil, nil}}, + expected: []tsdbutil.Sample{fSample{0, 0}, fSample{1, 1}, fSample{2, 2}, fSample{3, 3}}, }, } { merged := ChainSampleIteratorFromIterators(nil, tc.input) actual := []tsdbutil.Sample{} if merged.Seek(tc.seek) == chunkenc.ValFloat { - t, v := merged.At() - actual = append(actual, sample{t, v, nil, nil}) + t, f := merged.At() + actual = append(actual, fSample{t, f}) } s, err := ExpandSamples(merged, nil) require.NoError(t, err) @@ -906,7 +906,7 @@ func makeSeries(numSeries, numSamples int) []Series { labels := labels.FromStrings("foo", fmt.Sprintf("bar%d", j)) samples := []tsdbutil.Sample{} for k := 0; k < numSamples; k++ { - samples = append(samples, sample{t: int64(k), v: float64(k)}) + samples = append(samples, fSample{t: int64(k), f: float64(k)}) } series = append(series, NewListSeries(labels, samples)) } diff --git a/storage/series.go b/storage/series.go index dcb6dd82e6..3f90f77024 100644 --- a/storage/series.go +++ b/storage/series.go @@ -376,10 +376,17 @@ func (e errChunksIterator) Err() error { return e.err } // ExpandSamples iterates over all samples in the iterator, buffering all in slice. // Optionally it takes samples constructor, useful when you want to compare sample slices with different // sample implementations. if nil, sample type from this package will be used. -func ExpandSamples(iter chunkenc.Iterator, newSampleFn func(t int64, v float64, h *histogram.Histogram, fh *histogram.FloatHistogram) tsdbutil.Sample) ([]tsdbutil.Sample, error) { +func ExpandSamples(iter chunkenc.Iterator, newSampleFn func(t int64, f float64, h *histogram.Histogram, fh *histogram.FloatHistogram) tsdbutil.Sample) ([]tsdbutil.Sample, error) { if newSampleFn == nil { - newSampleFn = func(t int64, v float64, h *histogram.Histogram, fh *histogram.FloatHistogram) tsdbutil.Sample { - return sample{t, v, h, fh} + newSampleFn = func(t int64, f float64, h *histogram.Histogram, fh *histogram.FloatHistogram) tsdbutil.Sample { + switch { + case h != nil: + return hSample{t, h} + case fh != nil: + return fhSample{t, fh} + default: + return fSample{t, f} + } } } @@ -389,12 +396,12 @@ func ExpandSamples(iter chunkenc.Iterator, newSampleFn func(t int64, v float64, case chunkenc.ValNone: return result, iter.Err() case chunkenc.ValFloat: - t, v := iter.At() + t, f := iter.At() // NaNs can't be compared normally, so substitute for another value. - if math.IsNaN(v) { - v = -42 + if math.IsNaN(f) { + f = -42 } - result = append(result, newSampleFn(t, v, nil, nil)) + result = append(result, newSampleFn(t, f, nil, nil)) case chunkenc.ValHistogram: t, h := iter.AtHistogram() result = append(result, newSampleFn(t, 0, h, nil)) diff --git a/storage/series_test.go b/storage/series_test.go index 1f68d23b96..210a68e283 100644 --- a/storage/series_test.go +++ b/storage/series_test.go @@ -25,11 +25,11 @@ import ( func TestListSeriesIterator(t *testing.T) { it := NewListSeriesIterator(samples{ - sample{0, 0, nil, nil}, - sample{1, 1, nil, nil}, - sample{1, 1.5, nil, nil}, - sample{2, 2, nil, nil}, - sample{3, 3, nil, nil}, + fSample{0, 0}, + fSample{1, 1}, + fSample{1, 1.5}, + fSample{2, 2}, + fSample{3, 3}, }) // Seek to the first sample with ts=1. @@ -78,20 +78,20 @@ func TestChunkSeriesSetToSeriesSet(t *testing.T) { { lbs: labels.FromStrings("__name__", "up", "instance", "localhost:8080"), samples: []tsdbutil.Sample{ - sample{t: 1, v: 1}, - sample{t: 2, v: 2}, - sample{t: 3, v: 3}, - sample{t: 4, v: 4}, + fSample{t: 1, f: 1}, + fSample{t: 2, f: 2}, + fSample{t: 3, f: 3}, + fSample{t: 4, f: 4}, }, }, { lbs: labels.FromStrings("__name__", "up", "instance", "localhost:8081"), samples: []tsdbutil.Sample{ - sample{t: 1, v: 2}, - sample{t: 2, v: 3}, - sample{t: 3, v: 4}, - sample{t: 4, v: 5}, - sample{t: 5, v: 6}, - sample{t: 6, v: 7}, + fSample{t: 1, f: 2}, + fSample{t: 2, f: 3}, + fSample{t: 3, f: 4}, + fSample{t: 4, f: 5}, + fSample{t: 5, f: 6}, + fSample{t: 6, f: 7}, }, }, } @@ -114,7 +114,7 @@ func TestChunkSeriesSetToSeriesSet(t *testing.T) { j := 0 for iter.Next() == chunkenc.ValFloat { ts, v := iter.At() - require.EqualValues(t, series[i].samples[j], sample{t: ts, v: v}) + require.EqualValues(t, series[i].samples[j], fSample{t: ts, f: v}) j++ } } diff --git a/tsdb/tsdbutil/chunks.go b/tsdb/tsdbutil/chunks.go index f9981ffe16..4e025c5e6b 100644 --- a/tsdb/tsdbutil/chunks.go +++ b/tsdb/tsdbutil/chunks.go @@ -28,7 +28,7 @@ type Samples interface { type Sample interface { T() int64 - V() float64 + V() float64 // TODO(beorn7): Rename to F(). H() *histogram.Histogram FH() *histogram.FloatHistogram Type() chunkenc.ValueType diff --git a/web/federate.go b/web/federate.go index ad2574e80d..15e31b1a3e 100644 --- a/web/federate.go +++ b/web/federate.go @@ -115,38 +115,44 @@ Loop: var ( t int64 - v float64 - h *histogram.Histogram + f float64 fh *histogram.FloatHistogram - ok bool ) valueType := it.Seek(maxt) switch valueType { case chunkenc.ValFloat: - t, v = it.At() + t, f = it.At() case chunkenc.ValFloatHistogram, chunkenc.ValHistogram: t, fh = it.AtFloatHistogram() default: - t, v, h, fh, ok = it.PeekBack(1) + sample, ok := it.PeekBack(1) if !ok { continue Loop } - if h != nil { - fh = h.ToFloat() + t = sample.T() + switch sample.Type() { + case chunkenc.ValFloat: + f = sample.V() + case chunkenc.ValHistogram: + fh = sample.H().ToFloat() + case chunkenc.ValFloatHistogram: + fh = sample.FH() + default: + continue Loop } } // The exposition formats do not support stale markers, so drop them. This // is good enough for staleness handling of federated data, as the // interval-based limits on staleness will do the right thing for supported // use cases (which is to say federating aggregated time series). - if value.IsStaleNaN(v) { + if value.IsStaleNaN(f) || (fh != nil && value.IsStaleNaN(fh.Sum)) { continue } vec = append(vec, promql.Sample{ Metric: s.Labels(), T: t, - F: v, + F: f, H: fh, }) } From 462240bc787793dfe62c05c335f98b84401e4da0 Mon Sep 17 00:00:00 2001 From: beorn7 Date: Fri, 9 Dec 2022 00:44:48 +0100 Subject: [PATCH 087/231] storage: add specialized buffers to sampleRing This utilizes the fact that most sampleRings will only contain samples of one type. In this case, the generic interface is circumvented, and a bespoke buffer for the one actually occurring sample type is used. Should a sampleRing receive a sample of a different kind later, it will transparently switch to the generic behavior. Signed-off-by: beorn7 --- storage/buffer.go | 288 +++++++++++++++++++++++++++++++++++------ storage/buffer_test.go | 2 +- 2 files changed, 252 insertions(+), 38 deletions(-) diff --git a/storage/buffer.go b/storage/buffer.go index 68210facc1..9a9aff2af4 100644 --- a/storage/buffer.go +++ b/storage/buffer.go @@ -44,7 +44,7 @@ func NewBuffer(delta int64) *BufferedSeriesIterator { func NewBufferIterator(it chunkenc.Iterator, delta int64) *BufferedSeriesIterator { // TODO(codesome): based on encoding, allocate different buffer. bit := &BufferedSeriesIterator{ - buf: newSampleRing(delta, 16), + buf: newSampleRing(delta, 0, chunkenc.ValNone), delta: delta, } bit.Reset(it) @@ -121,13 +121,13 @@ func (b *BufferedSeriesIterator) Next() chunkenc.ValueType { return chunkenc.ValNone case chunkenc.ValFloat: t, f := b.it.At() - b.buf.add(fSample{t: t, f: f}) + b.buf.addF(fSample{t: t, f: f}) case chunkenc.ValHistogram: t, h := b.it.AtHistogram() - b.buf.add(hSample{t: t, h: h}) + b.buf.addH(hSample{t: t, h: h}) case chunkenc.ValFloatHistogram: t, fh := b.it.AtFloatHistogram() - b.buf.add(fhSample{t: t, fh: fh}) + b.buf.addFH(fhSample{t: t, fh: fh}) default: panic(fmt.Errorf("BufferedSeriesIterator: unknown value type %v", b.valueType)) } @@ -242,18 +242,44 @@ func (s fhSample) Type() chunkenc.ValueType { type sampleRing struct { delta int64 - buf []tsdbutil.Sample // lookback buffer - i int // position of most recent element in ring buffer - f int // position of first element in ring buffer - l int // number of elements in buffer + // Lookback buffers. We use buf for mixed samples, but one of the three + // concrete ones for homogenous samples. (Only one of the four bufs is + // allowed to be populated!) This avoids the overhead of the interface + // wrapper for the happy (and by far most common) case of homogenous + // samples. + buf []tsdbutil.Sample + fBuf []fSample + hBuf []hSample + fhBuf []fhSample + + i int // Position of most recent element in ring buffer. + f int // Position of first element in ring buffer. + l int // Number of elements in buffer. it sampleRingIterator } -func newSampleRing(delta int64, sz int) *sampleRing { - r := &sampleRing{delta: delta, buf: make([]tsdbutil.Sample, sz)} +// newSampleRing creates a new sampleRing. If you do not know the prefereed +// value type yet, use a size of 0 (in which case the provided typ doesn't +// matter). On the first add, a buffer of size 16 will be allocated with the +// preferred type being the type of the first added sample. +func newSampleRing(delta int64, size int, typ chunkenc.ValueType) *sampleRing { + r := &sampleRing{delta: delta} r.reset() - + if size <= 0 { + // Will initialize on first add. + return r + } + switch typ { + case chunkenc.ValFloat: + r.fBuf = make([]fSample, size) + case chunkenc.ValHistogram: + r.hBuf = make([]hSample, size) + case chunkenc.ValFloatHistogram: + r.fhBuf = make([]fhSample, size) + default: + r.buf = make([]tsdbutil.Sample, size) + } return r } @@ -274,7 +300,7 @@ type sampleRingIterator struct { r *sampleRing i int t int64 - v float64 + f float64 h *histogram.Histogram fh *histogram.FloatHistogram } @@ -284,6 +310,23 @@ func (it *sampleRingIterator) Next() chunkenc.ValueType { if it.i >= it.r.l { return chunkenc.ValNone } + switch { + case len(it.r.fBuf) > 0: + s := it.r.atF(it.i) + it.t = s.t + it.f = s.f + return chunkenc.ValFloat + case len(it.r.hBuf) > 0: + s := it.r.atH(it.i) + it.t = s.t + it.h = s.h + return chunkenc.ValHistogram + case len(it.r.fhBuf) > 0: + s := it.r.atFH(it.i) + it.t = s.t + it.fh = s.fh + return chunkenc.ValFloatHistogram + } s := it.r.at(it.i) it.t = s.T() switch s.Type() { @@ -294,7 +337,7 @@ func (it *sampleRingIterator) Next() chunkenc.ValueType { it.fh = s.FH() return chunkenc.ValFloatHistogram default: - it.v = s.V() + it.f = s.V() return chunkenc.ValFloat } } @@ -308,7 +351,7 @@ func (it *sampleRingIterator) Err() error { } func (it *sampleRingIterator) At() (int64, float64) { - return it.t, it.v + return it.t, it.f } func (it *sampleRingIterator) AtHistogram() (int64, *histogram.Histogram) { @@ -331,17 +374,128 @@ func (r *sampleRing) at(i int) tsdbutil.Sample { return r.buf[j] } -// add adds a sample to the ring buffer and frees all samples that fall -// out of the delta range. -func (r *sampleRing) add(s tsdbutil.Sample) { - l := len(r.buf) - // Grow the ring buffer if it fits no more elements. - if l == r.l { - buf := make([]tsdbutil.Sample, 2*l) - copy(buf[l+r.f:], r.buf[r.f:]) - copy(buf, r.buf[:r.f]) +func (r *sampleRing) atF(i int) fSample { + j := (r.f + i) % len(r.fBuf) + return r.fBuf[j] +} - r.buf = buf +func (r *sampleRing) atH(i int) hSample { + j := (r.f + i) % len(r.hBuf) + return r.hBuf[j] +} + +func (r *sampleRing) atFH(i int) fhSample { + j := (r.f + i) % len(r.fhBuf) + return r.fhBuf[j] +} + +// add adds a sample to the ring buffer and frees all samples that fall out of +// the delta range. Note that this method works for any sample +// implementation. If you know you are dealing with one of the implementations +// from this package (fSample, hSample, fhSample), call one of the specialized +// methods addF, addH, or addFH for better performance. +func (r *sampleRing) add(s tsdbutil.Sample) { + if len(r.buf) == 0 { + // Nothing added to the interface buf yet. Let's check if we can + // stay specialized. + switch s := s.(type) { + case fSample: + if len(r.hBuf)+len(r.fhBuf) == 0 { + r.fBuf = genericAdd(s, r.fBuf, r) + return + } + case hSample: + if len(r.fBuf)+len(r.fhBuf) == 0 { + r.hBuf = genericAdd(s, r.hBuf, r) + return + } + case fhSample: + if len(r.fBuf)+len(r.hBuf) == 0 { + r.fhBuf = genericAdd(s, r.fhBuf, r) + return + } + } + // The new sample isn't a fit for the already existing + // ones. Copy the latter into the interface buffer where needed. + switch { + case len(r.fBuf) > 0: + for _, s := range r.fBuf { + r.buf = append(r.buf, s) + } + r.fBuf = nil + case len(r.hBuf) > 0: + for _, s := range r.hBuf { + r.buf = append(r.buf, s) + } + r.hBuf = nil + case len(r.fhBuf) > 0: + for _, s := range r.fhBuf { + r.buf = append(r.buf, s) + } + r.fhBuf = nil + } + } + r.buf = genericAdd(s, r.buf, r) +} + +// addF is a version of the add method specialized for fSample. +func (r *sampleRing) addF(s fSample) { + switch { + case len(r.buf) > 0: + // Already have interface samples. Add to the interface buf. + r.buf = genericAdd[tsdbutil.Sample](s, r.buf, r) + case len(r.hBuf)+len(r.fhBuf) > 0: + // Already have specialized samples that are not fSamples. + // Need to call the checked add method for conversion. + r.add(s) + default: + r.fBuf = genericAdd(s, r.fBuf, r) + } +} + +// addH is a version of the add method specialized for hSample. +func (r *sampleRing) addH(s hSample) { + switch { + case len(r.buf) > 0: + // Already have interface samples. Add to the interface buf. + r.buf = genericAdd[tsdbutil.Sample](s, r.buf, r) + case len(r.fBuf)+len(r.fhBuf) > 0: + // Already have samples that are not hSamples. + // Need to call the checked add method for conversion. + r.add(s) + default: + r.hBuf = genericAdd(s, r.hBuf, r) + } +} + +// addFH is a version of the add method specialized for fhSample. +func (r *sampleRing) addFH(s fhSample) { + switch { + case len(r.buf) > 0: + // Already have interface samples. Add to the interface buf. + r.buf = genericAdd[tsdbutil.Sample](s, r.buf, r) + case len(r.fBuf)+len(r.hBuf) > 0: + // Already have samples that are not fhSamples. + // Need to call the checked add method for conversion. + r.add(s) + default: + r.fhBuf = genericAdd(s, r.fhBuf, r) + } +} + +func genericAdd[T tsdbutil.Sample](s T, buf []T, r *sampleRing) []T { + l := len(buf) + // Grow the ring buffer if it fits no more elements. + if l == 0 { + buf = make([]T, 16) + l = 16 + } + if l == r.l { + newBuf := make([]T, 2*l) + copy(newBuf[l+r.f:], buf[r.f:]) + copy(newBuf, buf[:r.f]) + + buf = newBuf r.i = r.f r.f += l l = 2 * l @@ -352,18 +506,19 @@ func (r *sampleRing) add(s tsdbutil.Sample) { } } - r.buf[r.i] = s + buf[r.i] = s r.l++ // Free head of the buffer of samples that just fell out of the range. tmin := s.T() - r.delta - for r.buf[r.f].T() < tmin { + for buf[r.f].T() < tmin { r.f++ if r.f >= l { r.f -= l } r.l-- } + return buf } // reduceDelta lowers the buffered time delta, dropping any samples that are @@ -378,17 +533,30 @@ func (r *sampleRing) reduceDelta(delta int64) bool { return true } + switch { + case len(r.fBuf) > 0: + genericReduceDelta(r.fBuf, r) + case len(r.hBuf) > 0: + genericReduceDelta(r.hBuf, r) + case len(r.fhBuf) > 0: + genericReduceDelta(r.fhBuf, r) + default: + genericReduceDelta(r.buf, r) + } + return true +} + +func genericReduceDelta[T tsdbutil.Sample](buf []T, r *sampleRing) { // Free head of the buffer of samples that just fell out of the range. - l := len(r.buf) - tmin := r.buf[r.i].T() - delta - for r.buf[r.f].T() < tmin { + l := len(buf) + tmin := buf[r.i].T() - r.delta + for buf[r.f].T() < tmin { r.f++ if r.f >= l { r.f -= l } r.l-- } - return true } // nthLast returns the nth most recent element added to the ring. @@ -396,7 +564,17 @@ func (r *sampleRing) nthLast(n int) (tsdbutil.Sample, bool) { if n > r.l { return fSample{}, false } - return r.at(r.l - n), true + i := r.l - n + switch { + case len(r.fBuf) > 0: + return r.atF(i), true + case len(r.hBuf) > 0: + return r.atH(i), true + case len(r.fhBuf) > 0: + return r.atFH(i), true + default: + return r.at(i), true + } } func (r *sampleRing) samples() []tsdbutil.Sample { @@ -404,13 +582,49 @@ func (r *sampleRing) samples() []tsdbutil.Sample { k := r.f + r.l var j int - if k > len(r.buf) { - k = len(r.buf) - j = r.l - k + r.f - } - n := copy(res, r.buf[r.f:k]) - copy(res[n:], r.buf[:j]) + switch { + case len(r.buf) > 0: + if k > len(r.buf) { + k = len(r.buf) + j = r.l - k + r.f + } + n := copy(res, r.buf[r.f:k]) + copy(res[n:], r.buf[:j]) + case len(r.fBuf) > 0: + if k > len(r.fBuf) { + k = len(r.fBuf) + j = r.l - k + r.f + } + resF := make([]fSample, r.l) + n := copy(resF, r.fBuf[r.f:k]) + copy(resF[n:], r.fBuf[:j]) + for i, s := range resF { + res[i] = s + } + case len(r.hBuf) > 0: + if k > len(r.hBuf) { + k = len(r.hBuf) + j = r.l - k + r.f + } + resH := make([]hSample, r.l) + n := copy(resH, r.hBuf[r.f:k]) + copy(resH[n:], r.hBuf[:j]) + for i, s := range resH { + res[i] = s + } + case len(r.fhBuf) > 0: + if k > len(r.fhBuf) { + k = len(r.fhBuf) + j = r.l - k + r.f + } + resFH := make([]fhSample, r.l) + n := copy(resFH, r.fhBuf[r.f:k]) + copy(resFH[n:], r.fhBuf[:j]) + for i, s := range resFH { + res[i] = s + } + } return res } diff --git a/storage/buffer_test.go b/storage/buffer_test.go index 7cfda4d4cf..2fd9e81d03 100644 --- a/storage/buffer_test.go +++ b/storage/buffer_test.go @@ -56,7 +56,7 @@ func TestSampleRing(t *testing.T) { }, } for _, c := range cases { - r := newSampleRing(c.delta, c.size) + r := newSampleRing(c.delta, c.size, chunkenc.ValFloat) input := []fSample{} for _, t := range c.input { From 817a2396cb2b9d15c6e10d88fa3b6fffdc914d3e Mon Sep 17 00:00:00 2001 From: beorn7 Date: Thu, 30 Mar 2023 19:50:13 +0200 Subject: [PATCH 088/231] Name float values as "floats", not as "values" In the past, every sample value was a float, so it was fine to call a variable holding such a float "value" or "sample". With native histograms, a sample might have a histogram value. And a histogram value is still a value. Calling a float value just "value" or "sample" or "V" is therefore misleading. Over the last few commits, I already renamed many variables, but this cleans up a few more places where the changes are more invasive. Note that we do not to attempt naming in the JSON APIs or in the protobufs. That would be quite a disruption. However, internally, we can call variables as we want, and we should go with the option of avoiding misunderstandings. Signed-off-by: beorn7 --- promql/functions.go | 20 +- storage/buffer.go | 8 +- storage/buffer_test.go | 4 +- storage/series.go | 2 +- tsdb/agent/db_test.go | 6 +- tsdb/block_test.go | 16 +- tsdb/blockwriter_test.go | 4 +- tsdb/compact_test.go | 4 +- tsdb/db_test.go | 66 +++--- tsdb/head.go | 4 +- tsdb/head_test.go | 42 ++-- tsdb/ooo_head.go | 4 +- tsdb/ooo_head_read_test.go | 402 ++++++++++++++++++------------------- tsdb/ooo_head_test.go | 6 +- tsdb/querier_test.go | 16 +- tsdb/tsdbutil/chunks.go | 14 +- util/jsonutil/marshal.go | 6 +- web/federate.go | 2 +- 18 files changed, 313 insertions(+), 313 deletions(-) diff --git a/promql/functions.go b/promql/functions.go index 8fc5e64a58..fd99703df2 100644 --- a/promql/functions.go +++ b/promql/functions.go @@ -70,7 +70,7 @@ func extrapolatedRate(vals []parser.Value, args parser.Expressions, enh *EvalNod samples = vals[0].(Matrix)[0] rangeStart = enh.Ts - durationMilliseconds(ms.Range+vs.Offset) rangeEnd = enh.Ts - durationMilliseconds(vs.Offset) - resultValue float64 + resultFloat float64 resultHistogram *histogram.FloatHistogram firstT, lastT int64 numSamplesMinusOne int @@ -99,7 +99,7 @@ func extrapolatedRate(vals []parser.Value, args parser.Expressions, enh *EvalNod numSamplesMinusOne = len(samples.Floats) - 1 firstT = samples.Floats[0].T lastT = samples.Floats[numSamplesMinusOne].T - resultValue = samples.Floats[numSamplesMinusOne].F - samples.Floats[0].F + resultFloat = samples.Floats[numSamplesMinusOne].F - samples.Floats[0].F if !isCounter { break } @@ -107,7 +107,7 @@ func extrapolatedRate(vals []parser.Value, args parser.Expressions, enh *EvalNod prevValue := samples.Floats[0].F for _, currPoint := range samples.Floats[1:] { if currPoint.F < prevValue { - resultValue += prevValue + resultFloat += prevValue } prevValue = currPoint.F } @@ -124,14 +124,14 @@ func extrapolatedRate(vals []parser.Value, args parser.Expressions, enh *EvalNod averageDurationBetweenSamples := sampledInterval / float64(numSamplesMinusOne) // TODO(beorn7): Do this for histograms, too. - if isCounter && resultValue > 0 && len(samples.Floats) > 0 && samples.Floats[0].F >= 0 { + if isCounter && resultFloat > 0 && len(samples.Floats) > 0 && samples.Floats[0].F >= 0 { // Counters cannot be negative. If we have any slope at all - // (i.e. resultValue went up), we can extrapolate the zero point + // (i.e. resultFloat went up), we can extrapolate the zero point // of the counter. If the duration to the zero point is shorter // than the durationToStart, we take the zero point as the start // of the series, thereby avoiding extrapolation to negative // counter values. - durationToZero := sampledInterval * (samples.Floats[0].F / resultValue) + durationToZero := sampledInterval * (samples.Floats[0].F / resultFloat) if durationToZero < durationToStart { durationToStart = durationToZero } @@ -159,12 +159,12 @@ func extrapolatedRate(vals []parser.Value, args parser.Expressions, enh *EvalNod factor /= ms.Range.Seconds() } if resultHistogram == nil { - resultValue *= factor + resultFloat *= factor } else { resultHistogram.Scale(factor) } - return append(enh.Out, Sample{F: resultValue, H: resultHistogram}) + return append(enh.Out, Sample{F: resultFloat, H: resultHistogram}) } // histogramRate is a helper function for extrapolatedRate. It requires @@ -418,10 +418,10 @@ func funcRound(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper toNearestInverse := 1.0 / toNearest for _, el := range vec { - v := math.Floor(el.F*toNearestInverse+0.5) / toNearestInverse + f := math.Floor(el.F*toNearestInverse+0.5) / toNearestInverse enh.Out = append(enh.Out, Sample{ Metric: enh.DropMetricName(el.Metric), - F: v, + F: f, }) } return enh.Out diff --git a/storage/buffer.go b/storage/buffer.go index 9a9aff2af4..0daadc0444 100644 --- a/storage/buffer.go +++ b/storage/buffer.go @@ -173,7 +173,7 @@ func (s fSample) T() int64 { return s.t } -func (s fSample) V() float64 { +func (s fSample) F() float64 { return s.f } @@ -198,7 +198,7 @@ func (s hSample) T() int64 { return s.t } -func (s hSample) V() float64 { +func (s hSample) F() float64 { panic("F() called for hSample") } @@ -223,7 +223,7 @@ func (s fhSample) T() int64 { return s.t } -func (s fhSample) V() float64 { +func (s fhSample) F() float64 { panic("F() called for fhSample") } @@ -337,7 +337,7 @@ func (it *sampleRingIterator) Next() chunkenc.ValueType { it.fh = s.FH() return chunkenc.ValFloatHistogram default: - it.f = s.V() + it.f = s.F() return chunkenc.ValFloat } } diff --git a/storage/buffer_test.go b/storage/buffer_test.go index 2fd9e81d03..ebe24d8df3 100644 --- a/storage/buffer_test.go +++ b/storage/buffer_test.go @@ -73,7 +73,7 @@ func TestSampleRing(t *testing.T) { for _, sold := range input[:i] { found := false for _, bs := range buffered { - if bs.T() == sold.t && bs.V() == sold.f { + if bs.T() == sold.t && bs.F() == sold.f { found = true break } @@ -110,7 +110,7 @@ func TestBufferedSeriesIterator(t *testing.T) { s, ok := it.PeekBack(1) require.Equal(t, eok, ok, "exist mismatch") require.Equal(t, ets, s.T(), "timestamp mismatch") - require.Equal(t, ev, s.V(), "value mismatch") + require.Equal(t, ev, s.F(), "value mismatch") } it = NewBufferIterator(NewListSeriesIterator(samples{ diff --git a/storage/series.go b/storage/series.go index 3f90f77024..f609df3f02 100644 --- a/storage/series.go +++ b/storage/series.go @@ -109,7 +109,7 @@ func (it *listSeriesIterator) Reset(samples Samples) { func (it *listSeriesIterator) At() (int64, float64) { s := it.samples.Get(it.idx) - return s.T(), s.V() + return s.T(), s.F() } func (it *listSeriesIterator) AtHistogram() (int64, *histogram.Histogram) { diff --git a/tsdb/agent/db_test.go b/tsdb/agent/db_test.go index 9b27aaa0bf..f654fdb90e 100644 --- a/tsdb/agent/db_test.go +++ b/tsdb/agent/db_test.go @@ -133,13 +133,13 @@ func TestCommit(t *testing.T) { for i := 0; i < numDatapoints; i++ { sample := tsdbutil.GenerateSamples(0, 1) - ref, err := app.Append(0, lset, sample[0].T(), sample[0].V()) + ref, err := app.Append(0, lset, sample[0].T(), sample[0].F()) require.NoError(t, err) e := exemplar.Exemplar{ Labels: lset, Ts: sample[0].T() + int64(i), - Value: sample[0].V(), + Value: sample[0].F(), HasTs: true, } _, err = app.AppendExemplar(ref, lset, e) @@ -248,7 +248,7 @@ func TestRollback(t *testing.T) { for i := 0; i < numDatapoints; i++ { sample := tsdbutil.GenerateSamples(0, 1) - _, err := app.Append(0, lset, sample[0].T(), sample[0].V()) + _, err := app.Append(0, lset, sample[0].T(), sample[0].F()) require.NoError(t, err) } } diff --git a/tsdb/block_test.go b/tsdb/block_test.go index ab33999623..49a997fc56 100644 --- a/tsdb/block_test.go +++ b/tsdb/block_test.go @@ -353,14 +353,14 @@ func TestReadIndexFormatV1(t *testing.T) { q, err := NewBlockQuerier(block, 0, 1000) require.NoError(t, err) require.Equal(t, query(t, q, labels.MustNewMatcher(labels.MatchEqual, "foo", "bar")), - map[string][]tsdbutil.Sample{`{foo="bar"}`: {sample{t: 1, v: 2}}}) + map[string][]tsdbutil.Sample{`{foo="bar"}`: {sample{t: 1, f: 2}}}) q, err = NewBlockQuerier(block, 0, 1000) require.NoError(t, err) require.Equal(t, query(t, q, labels.MustNewMatcher(labels.MatchNotRegexp, "foo", "^.?$")), map[string][]tsdbutil.Sample{ - `{foo="bar"}`: {sample{t: 1, v: 2}}, - `{foo="baz"}`: {sample{t: 3, v: 4}}, + `{foo="bar"}`: {sample{t: 1, f: 2}}, + `{foo="baz"}`: {sample{t: 3, f: 4}}, }) } @@ -568,7 +568,7 @@ func createHeadWithOOOSamples(tb testing.TB, w *wlog.WL, series []storage.Series count++ t, v := it.At() if count%oooSampleFrequency == 0 { - os = append(os, sample{t: t, v: v}) + os = append(os, sample{t: t, f: v}) continue } ref, err = app.Append(ref, lset, t, v) @@ -589,7 +589,7 @@ func createHeadWithOOOSamples(tb testing.TB, w *wlog.WL, series []storage.Series for i, lset := range oooSampleLabels { ref := storage.SeriesRef(0) for _, sample := range oooSamples[i] { - ref, err = app.Append(ref, lset, sample.T(), sample.V()) + ref, err = app.Append(ref, lset, sample.T(), sample.F()) require.NoError(tb, err) oooSamplesAppended++ } @@ -613,7 +613,7 @@ const ( // genSeries generates series of float64 samples with a given number of labels and values. func genSeries(totalSeries, labelCount int, mint, maxt int64) []storage.Series { return genSeriesFromSampleGenerator(totalSeries, labelCount, mint, maxt, 1, func(ts int64) tsdbutil.Sample { - return sample{t: ts, v: rand.Float64()} + return sample{t: ts, f: rand.Float64()} }) } @@ -657,7 +657,7 @@ func genHistogramAndFloatSeries(totalSeries, labelCount int, mint, maxt, step in count++ var s sample if floatSample { - s = sample{t: ts, v: rand.Float64()} + s = sample{t: ts, f: rand.Float64()} } else { h := &histogram.Histogram{ Count: 5 + uint64(ts*4), @@ -729,7 +729,7 @@ func populateSeries(lbls []map[string]string, mint, maxt int64) []storage.Series } samples := make([]tsdbutil.Sample, 0, maxt-mint+1) for t := mint; t <= maxt; t++ { - samples = append(samples, sample{t: t, v: rand.Float64()}) + samples = append(samples, sample{t: t, f: rand.Float64()}) } series = append(series, storage.NewListSeries(labels.FromMap(lbl), samples)) } diff --git a/tsdb/blockwriter_test.go b/tsdb/blockwriter_test.go index e6703b7989..84ea8d51b2 100644 --- a/tsdb/blockwriter_test.go +++ b/tsdb/blockwriter_test.go @@ -52,8 +52,8 @@ func TestBlockWriter(t *testing.T) { q, err := NewBlockQuerier(b, math.MinInt64, math.MaxInt64) require.NoError(t, err) series := query(t, q, labels.MustNewMatcher(labels.MatchRegexp, "", ".*")) - sample1 := []tsdbutil.Sample{sample{t: ts1, v: v1}} - sample2 := []tsdbutil.Sample{sample{t: ts2, v: v2}} + sample1 := []tsdbutil.Sample{sample{t: ts1, f: v1}} + sample2 := []tsdbutil.Sample{sample{t: ts2, f: v2}} expectedSeries := map[string][]tsdbutil.Sample{"{a=\"b\"}": sample1, "{c=\"d\"}": sample2} require.Equal(t, expectedSeries, series) diff --git a/tsdb/compact_test.go b/tsdb/compact_test.go index 1b06845215..6a7e6ea68b 100644 --- a/tsdb/compact_test.go +++ b/tsdb/compact_test.go @@ -975,7 +975,7 @@ func TestCompaction_populateBlock(t *testing.T) { s sample ) for iter.Next() == chunkenc.ValFloat { - s.t, s.v = iter.At() + s.t, s.f = iter.At() if firstTs == math.MaxInt64 { firstTs = s.t } @@ -1350,7 +1350,7 @@ func TestHeadCompactionWithHistograms(t *testing.T) { for tsMinute := from; tsMinute <= to; tsMinute++ { _, err := app.Append(0, lbls, minute(tsMinute), float64(tsMinute)) require.NoError(t, err) - *exp = append(*exp, sample{t: minute(tsMinute), v: float64(tsMinute)}) + *exp = append(*exp, sample{t: minute(tsMinute), f: float64(tsMinute)}) } require.NoError(t, app.Commit()) } diff --git a/tsdb/db_test.go b/tsdb/db_test.go index 70639085e4..c54fccf6f1 100644 --- a/tsdb/db_test.go +++ b/tsdb/db_test.go @@ -104,7 +104,7 @@ func query(t testing.TB, q storage.Querier, matchers ...*labels.Matcher) map[str switch typ { case chunkenc.ValFloat: ts, v := it.At() - samples = append(samples, sample{t: ts, v: v}) + samples = append(samples, sample{t: ts, f: v}) case chunkenc.ValHistogram: ts, h := it.AtHistogram() samples = append(samples, sample{t: ts, h: h}) @@ -233,7 +233,7 @@ func TestDataAvailableOnlyAfterCommit(t *testing.T) { seriesSet = query(t, querier, labels.MustNewMatcher(labels.MatchEqual, "foo", "bar")) - require.Equal(t, map[string][]tsdbutil.Sample{`{foo="bar"}`: {sample{t: 0, v: 0}}}, seriesSet) + require.Equal(t, map[string][]tsdbutil.Sample{`{foo="bar"}`: {sample{t: 0, f: 0}}}, seriesSet) } // TestNoPanicAfterWALCorruption ensures that querying the db after a WAL corruption doesn't cause a panic. @@ -251,7 +251,7 @@ func TestNoPanicAfterWALCorruption(t *testing.T) { for i := 0; i < 121; i++ { app := db.Appender(ctx) _, err := app.Append(0, labels.FromStrings("foo", "bar"), maxt, 0) - expSamples = append(expSamples, sample{t: maxt, v: 0}) + expSamples = append(expSamples, sample{t: maxt, f: 0}) require.NoError(t, err) require.NoError(t, app.Commit()) maxt++ @@ -364,11 +364,11 @@ func TestDBAppenderAddRef(t *testing.T) { require.Equal(t, map[string][]tsdbutil.Sample{ labels.FromStrings("a", "b").String(): { - sample{t: 123, v: 0}, - sample{t: 124, v: 1}, - sample{t: 125, v: 0}, - sample{t: 133, v: 1}, - sample{t: 143, v: 2}, + sample{t: 123, f: 0}, + sample{t: 124, f: 1}, + sample{t: 125, f: 0}, + sample{t: 133, f: 1}, + sample{t: 143, f: 2}, }, }, res) } @@ -1740,7 +1740,7 @@ func expandSeriesSet(ss storage.SeriesSet) ([]labels.Labels, map[string][]sample it = series.Iterator(it) for it.Next() == chunkenc.ValFloat { t, v := it.At() - samples = append(samples, sample{t: t, v: v}) + samples = append(samples, sample{t: t, f: v}) } resultLabels = append(resultLabels, series.Labels()) resultSamples[series.Labels().String()] = samples @@ -2617,7 +2617,7 @@ func TestDBCannotSeePartialCommits(t *testing.T) { values := map[float64]struct{}{} for _, series := range seriesSet { - values[series[len(series)-1].v] = struct{}{} + values[series[len(series)-1].f] = struct{}{} } if len(values) != 1 { inconsistencies++ @@ -2693,7 +2693,7 @@ func TestDBQueryDoesntSeeAppendsAfterCreation(t *testing.T) { _, seriesSet, ws, err = expandSeriesSet(ss) require.NoError(t, err) require.Equal(t, 0, len(ws)) - require.Equal(t, map[string][]sample{`{foo="bar"}`: {{t: 0, v: 0}}}, seriesSet) + require.Equal(t, map[string][]sample{`{foo="bar"}`: {{t: 0, f: 0}}}, seriesSet) } // TestChunkWriter_ReadAfterWrite ensures that chunk segment are cut at the set segment size and @@ -4575,7 +4575,7 @@ func Test_Querier_OOOQuery(t *testing.T) { for min := fromMins; min <= toMins; min += time.Minute.Milliseconds() { _, err := app.Append(0, series1, min, float64(min)) if min >= queryMinT && min <= queryMaxT { - expSamples = append(expSamples, sample{t: min, v: float64(min)}) + expSamples = append(expSamples, sample{t: min, f: float64(min)}) } require.NoError(t, err) totalAppended++ @@ -4660,7 +4660,7 @@ func Test_ChunkQuerier_OOOQuery(t *testing.T) { for min := fromMins; min <= toMins; min += time.Minute.Milliseconds() { _, err := app.Append(0, series1, min, float64(min)) if min >= queryMinT && min <= queryMaxT { - expSamples = append(expSamples, sample{t: min, v: float64(min)}) + expSamples = append(expSamples, sample{t: min, f: float64(min)}) } require.NoError(t, err) totalAppended++ @@ -4730,7 +4730,7 @@ func Test_ChunkQuerier_OOOQuery(t *testing.T) { it := chunk.Chunk.Iterator(nil) for it.Next() == chunkenc.ValFloat { ts, v := it.At() - gotSamples = append(gotSamples, sample{t: ts, v: v}) + gotSamples = append(gotSamples, sample{t: ts, f: v}) } } require.Equal(t, expSamples, gotSamples) @@ -4766,7 +4766,7 @@ func TestOOOAppendAndQuery(t *testing.T) { require.Error(t, err) } else { require.NoError(t, err) - appendedSamples[key] = append(appendedSamples[key], sample{t: min, v: val}) + appendedSamples[key] = append(appendedSamples[key], sample{t: min, f: val}) totalSamples++ } } @@ -4889,7 +4889,7 @@ func TestOOODisabled(t *testing.T) { failedSamples++ } else { require.NoError(t, err) - expSamples[key] = append(expSamples[key], sample{t: min, v: val}) + expSamples[key] = append(expSamples[key], sample{t: min, f: val}) totalSamples++ } } @@ -4952,7 +4952,7 @@ func TestWBLAndMmapReplay(t *testing.T) { val := rand.Float64() _, err := app.Append(0, lbls, min, val) require.NoError(t, err) - expSamples[key] = append(expSamples[key], sample{t: min, v: val}) + expSamples[key] = append(expSamples[key], sample{t: min, f: val}) totalSamples++ } require.NoError(t, app.Commit()) @@ -4995,7 +4995,7 @@ func TestWBLAndMmapReplay(t *testing.T) { it := chk.Iterator(nil) for it.Next() == chunkenc.ValFloat { ts, val := it.At() - s1MmapSamples = append(s1MmapSamples, sample{t: ts, v: val}) + s1MmapSamples = append(s1MmapSamples, sample{t: ts, f: val}) } } require.Greater(t, len(s1MmapSamples), 0) @@ -5273,9 +5273,9 @@ func TestWBLCorruption(t *testing.T) { ts := min * time.Minute.Milliseconds() _, err := app.Append(0, series1, ts, float64(ts)) require.NoError(t, err) - allSamples = append(allSamples, sample{t: ts, v: float64(ts)}) + allSamples = append(allSamples, sample{t: ts, f: float64(ts)}) if afterRestart { - expAfterRestart = append(expAfterRestart, sample{t: ts, v: float64(ts)}) + expAfterRestart = append(expAfterRestart, sample{t: ts, f: float64(ts)}) } } require.NoError(t, app.Commit()) @@ -5419,9 +5419,9 @@ func TestOOOMmapCorruption(t *testing.T) { ts := min * time.Minute.Milliseconds() _, err := app.Append(0, series1, ts, float64(ts)) require.NoError(t, err) - allSamples = append(allSamples, sample{t: ts, v: float64(ts)}) + allSamples = append(allSamples, sample{t: ts, f: float64(ts)}) if inMmapAfterCorruption { - expInMmapChunks = append(expInMmapChunks, sample{t: ts, v: float64(ts)}) + expInMmapChunks = append(expInMmapChunks, sample{t: ts, f: float64(ts)}) } } require.NoError(t, app.Commit()) @@ -5555,7 +5555,7 @@ func TestOutOfOrderRuntimeConfig(t *testing.T) { _, err := app.Append(0, series1, ts, float64(ts)) if success { require.NoError(t, err) - allSamples = append(allSamples, sample{t: ts, v: float64(ts)}) + allSamples = append(allSamples, sample{t: ts, f: float64(ts)}) } else { require.Error(t, err) } @@ -5769,7 +5769,7 @@ func TestNoGapAfterRestartWithOOO(t *testing.T) { var expSamples []tsdbutil.Sample for min := fromMins; min <= toMins; min++ { ts := min * time.Minute.Milliseconds() - expSamples = append(expSamples, sample{t: ts, v: float64(ts)}) + expSamples = append(expSamples, sample{t: ts, f: float64(ts)}) } expRes := map[string][]tsdbutil.Sample{ @@ -5876,7 +5876,7 @@ func TestWblReplayAfterOOODisableAndRestart(t *testing.T) { ts := min * time.Minute.Milliseconds() _, err := app.Append(0, series1, ts, float64(ts)) require.NoError(t, err) - allSamples = append(allSamples, sample{t: ts, v: float64(ts)}) + allSamples = append(allSamples, sample{t: ts, f: float64(ts)}) } require.NoError(t, app.Commit()) } @@ -5935,7 +5935,7 @@ func TestPanicOnApplyConfig(t *testing.T) { ts := min * time.Minute.Milliseconds() _, err := app.Append(0, series1, ts, float64(ts)) require.NoError(t, err) - allSamples = append(allSamples, sample{t: ts, v: float64(ts)}) + allSamples = append(allSamples, sample{t: ts, f: float64(ts)}) } require.NoError(t, app.Commit()) } @@ -5983,7 +5983,7 @@ func TestDiskFillingUpAfterDisablingOOO(t *testing.T) { ts := min * time.Minute.Milliseconds() _, err := app.Append(0, series1, ts, float64(ts)) require.NoError(t, err) - allSamples = append(allSamples, sample{t: ts, v: float64(ts)}) + allSamples = append(allSamples, sample{t: ts, f: float64(ts)}) } require.NoError(t, app.Commit()) } @@ -6090,7 +6090,7 @@ func testHistogramAppendAndQueryHelper(t *testing.T, floatHistogram bool) { _, err := app.Append(0, lbls, minute(tsMinute), val) require.NoError(t, err) require.NoError(t, app.Commit()) - *exp = append(*exp, sample{t: minute(tsMinute), v: val}) + *exp = append(*exp, sample{t: minute(tsMinute), f: val}) } testQuery := func(name, value string, exp map[string][]tsdbutil.Sample) { @@ -6346,7 +6346,7 @@ func TestQueryHistogramFromBlocksWithCompaction(t *testing.T) { switch typ { case chunkenc.ValFloat: ts, v := it.At() - slice = append(slice, sample{t: ts, v: v}) + slice = append(slice, sample{t: ts, f: v}) case chunkenc.ValHistogram: ts, h := it.AtHistogram() slice = append(slice, sample{t: ts, h: h}) @@ -6418,7 +6418,7 @@ func TestQueryHistogramFromBlocksWithCompaction(t *testing.T) { testBlockQuerying(t, genHistogramSeries(10, 5, minute(0), minute(119), minute(1), floatHistogram), genSeriesFromSampleGenerator(10, 5, minute(120), minute(239), minute(1), func(ts int64) tsdbutil.Sample { - return sample{t: ts, v: rand.Float64()} + return sample{t: ts, f: rand.Float64()} }), genHistogramSeries(10, 5, minute(240), minute(359), minute(1), floatHistogram), ) @@ -6430,7 +6430,7 @@ func TestQueryHistogramFromBlocksWithCompaction(t *testing.T) { genHistogramSeries(10, 5, minute(61), minute(120), minute(1), floatHistogram), genHistogramAndFloatSeries(10, 5, minute(121), minute(180), minute(1), floatHistogram), genSeriesFromSampleGenerator(10, 5, minute(181), minute(240), minute(1), func(ts int64) tsdbutil.Sample { - return sample{t: ts, v: rand.Float64()} + return sample{t: ts, f: rand.Float64()} }), ) }) @@ -6447,7 +6447,7 @@ func TestQueryHistogramFromBlocksWithCompaction(t *testing.T) { testBlockQuerying(t, genHistogramSeries(10, 5, minute(0), minute(120), minute(3), floatHistogram), genSeriesFromSampleGenerator(10, 5, minute(1), minute(120), minute(3), func(ts int64) tsdbutil.Sample { - return sample{t: ts, v: rand.Float64()} + return sample{t: ts, f: rand.Float64()} }), genHistogramSeries(10, 5, minute(2), minute(120), minute(3), floatHistogram), ) @@ -6459,7 +6459,7 @@ func TestQueryHistogramFromBlocksWithCompaction(t *testing.T) { genHistogramSeries(10, 5, minute(46), minute(100), minute(3), floatHistogram), genHistogramAndFloatSeries(10, 5, minute(89), minute(140), minute(3), floatHistogram), genSeriesFromSampleGenerator(10, 5, minute(126), minute(200), minute(3), func(ts int64) tsdbutil.Sample { - return sample{t: ts, v: rand.Float64()} + return sample{t: ts, f: rand.Float64()} }), ) }) diff --git a/tsdb/head.go b/tsdb/head.go index af8175cd03..4696884f21 100644 --- a/tsdb/head.go +++ b/tsdb/head.go @@ -1864,7 +1864,7 @@ func (s *stripeSeries) getOrSet(hash uint64, lset labels.Labels, createSeries fu type sample struct { t int64 - v float64 + f float64 h *histogram.Histogram fh *histogram.FloatHistogram } @@ -1874,7 +1874,7 @@ func newSample(t int64, v float64, h *histogram.Histogram, fh *histogram.FloatHi } func (s sample) T() int64 { return s.t } -func (s sample) V() float64 { return s.v } +func (s sample) F() float64 { return s.f } func (s sample) H() *histogram.Histogram { return s.h } func (s sample) FH() *histogram.FloatHistogram { return s.fh } diff --git a/tsdb/head_test.go b/tsdb/head_test.go index 0a2a2ee6f4..e80c197b23 100644 --- a/tsdb/head_test.go +++ b/tsdb/head_test.go @@ -465,8 +465,8 @@ func TestHead_HighConcurrencyReadAndWrite(t *testing.T) { if sample.T() != int64(expectedValue) { return false, fmt.Errorf("expected sample %d to have ts %d, got %d", sampleIdx, expectedValue, sample.T()) } - if sample.V() != float64(expectedValue) { - return false, fmt.Errorf("expected sample %d to have value %d, got %f", sampleIdx, expectedValue, sample.V()) + if sample.F() != float64(expectedValue) { + return false, fmt.Errorf("expected sample %d to have value %d, got %f", sampleIdx, expectedValue, sample.F()) } } @@ -575,7 +575,7 @@ func TestHead_ReadWAL(t *testing.T) { expandChunk := func(c chunkenc.Iterator) (x []sample) { for c.Next() == chunkenc.ValFloat { t, v := c.At() - x = append(x, sample{t: t, v: v}) + x = append(x, sample{t: t, f: v}) } require.NoError(t, c.Err()) return x @@ -870,7 +870,7 @@ func TestHeadDeleteSimple(t *testing.T) { buildSmpls := func(s []int64) []sample { ss := make([]sample, 0, len(s)) for _, t := range s { - ss = append(ss, sample{t: t, v: float64(t)}) + ss = append(ss, sample{t: t, f: float64(t)}) } return ss } @@ -925,7 +925,7 @@ func TestHeadDeleteSimple(t *testing.T) { app := head.Appender(context.Background()) for _, smpl := range smplsAll { - _, err := app.Append(0, lblsDefault, smpl.t, smpl.v) + _, err := app.Append(0, lblsDefault, smpl.t, smpl.f) require.NoError(t, err) } @@ -939,7 +939,7 @@ func TestHeadDeleteSimple(t *testing.T) { // Add more samples. app = head.Appender(context.Background()) for _, smpl := range c.addSamples { - _, err := app.Append(0, lblsDefault, smpl.t, smpl.v) + _, err := app.Append(0, lblsDefault, smpl.t, smpl.f) require.NoError(t, err) } @@ -1924,7 +1924,7 @@ func TestMemSeriesIsolation(t *testing.T) { require.Equal(t, 0, len(ws)) for _, series := range seriesSet { - return int(series[len(series)-1].v) + return int(series[len(series)-1].f) } return -1 } @@ -3088,7 +3088,7 @@ func TestHistogramInWALAndMmapChunk(t *testing.T) { ts++ _, err := app.Append(0, s2, int64(ts), float64(ts)) require.NoError(t, err) - exp[k2] = append(exp[k2], sample{t: int64(ts), v: float64(ts)}) + exp[k2] = append(exp[k2], sample{t: int64(ts), f: float64(ts)}) } require.NoError(t, app.Commit()) app = head.Appender(context.Background()) @@ -3125,7 +3125,7 @@ func TestHistogramInWALAndMmapChunk(t *testing.T) { ts++ _, err := app.Append(0, s2, int64(ts), float64(ts)) require.NoError(t, err) - exp[k2] = append(exp[k2], sample{t: int64(ts), v: float64(ts)}) + exp[k2] = append(exp[k2], sample{t: int64(ts), f: float64(ts)}) } require.NoError(t, app.Commit()) app = head.Appender(context.Background()) @@ -3812,7 +3812,7 @@ func TestAppendingDifferentEncodingToSameSeries(t *testing.T) { expChunks: 1, }, { - samples: []tsdbutil.Sample{sample{t: 200, v: 2}}, + samples: []tsdbutil.Sample{sample{t: 200, f: 2}}, expChunks: 2, }, { @@ -3836,7 +3836,7 @@ func TestAppendingDifferentEncodingToSameSeries(t *testing.T) { expChunks: 6, }, { - samples: []tsdbutil.Sample{sample{t: 100, v: 2}}, + samples: []tsdbutil.Sample{sample{t: 100, f: 2}}, err: storage.ErrOutOfOrderSample, }, { @@ -3847,13 +3847,13 @@ func TestAppendingDifferentEncodingToSameSeries(t *testing.T) { // Combination of histograms and float64 in the same commit. The behaviour is undefined, but we want to also // verify how TSDB would behave. Here the histogram is appended at the end, hence will be considered as out of order. samples: []tsdbutil.Sample{ - sample{t: 400, v: 4}, + sample{t: 400, f: 4}, sample{t: 500, h: hists[5]}, // This won't be committed. - sample{t: 600, v: 6}, + sample{t: 600, f: 6}, }, addToExp: []tsdbutil.Sample{ - sample{t: 400, v: 4}, - sample{t: 600, v: 6}, + sample{t: 400, f: 4}, + sample{t: 600, f: 6}, }, expChunks: 7, // Only 1 new chunk for float64. }, @@ -3861,11 +3861,11 @@ func TestAppendingDifferentEncodingToSameSeries(t *testing.T) { // Here the histogram is appended at the end, hence the first histogram is out of order. samples: []tsdbutil.Sample{ sample{t: 700, h: hists[7]}, // Out of order w.r.t. the next float64 sample that is appended first. - sample{t: 800, v: 8}, + sample{t: 800, f: 8}, sample{t: 900, h: hists[9]}, }, addToExp: []tsdbutil.Sample{ - sample{t: 800, v: 8}, + sample{t: 800, f: 8}, sample{t: 900, h: hists[9].Copy()}, }, expChunks: 8, // float64 added to old chunk, only 1 new for histograms. @@ -3890,7 +3890,7 @@ func TestAppendingDifferentEncodingToSameSeries(t *testing.T) { if s.H() != nil || s.FH() != nil { _, err = app.AppendHistogram(0, lbls, s.T(), s.H(), s.FH()) } else { - _, err = app.Append(0, lbls, s.T(), s.V()) + _, err = app.Append(0, lbls, s.T(), s.F()) } require.Equal(t, a.err, err) } @@ -4056,7 +4056,7 @@ func TestOOOWalReplay(t *testing.T) { require.NoError(t, app.Commit()) if isOOO { - expOOOSamples = append(expOOOSamples, sample{t: ts, v: v}) + expOOOSamples = append(expOOOSamples, sample{t: ts, f: v}) } } @@ -4100,7 +4100,7 @@ func TestOOOWalReplay(t *testing.T) { actOOOSamples := make([]sample, 0, len(expOOOSamples)) for it.Next() == chunkenc.ValFloat { ts, v := it.At() - actOOOSamples = append(actOOOSamples, sample{t: ts, v: v}) + actOOOSamples = append(actOOOSamples, sample{t: ts, f: v}) } // OOO chunk will be sorted. Hence sort the expected samples. @@ -4360,7 +4360,7 @@ func TestReplayAfterMmapReplayError(t *testing.T) { var ref storage.SeriesRef for i := 0; i < numSamples; i++ { ref, err = app.Append(ref, lbls, lastTs, float64(lastTs)) - expSamples = append(expSamples, sample{t: lastTs, v: float64(lastTs)}) + expSamples = append(expSamples, sample{t: lastTs, f: float64(lastTs)}) require.NoError(t, err) lastTs += itvl if i%10 == 0 { diff --git a/tsdb/ooo_head.go b/tsdb/ooo_head.go index 63d0b37127..45827889e6 100644 --- a/tsdb/ooo_head.go +++ b/tsdb/ooo_head.go @@ -78,7 +78,7 @@ func (o *OOOChunk) ToXOR() (*chunkenc.XORChunk, error) { return nil, err } for _, s := range o.samples { - app.Append(s.t, s.v) + app.Append(s.t, s.f) } return x, nil } @@ -96,7 +96,7 @@ func (o *OOOChunk) ToXORBetweenTimestamps(mint, maxt int64) (*chunkenc.XORChunk, if s.t > maxt { break } - app.Append(s.t, s.v) + app.Append(s.t, s.f) } return x, nil } diff --git a/tsdb/ooo_head_read_test.go b/tsdb/ooo_head_read_test.go index 177bd2326e..f3ec862f5b 100644 --- a/tsdb/ooo_head_read_test.go +++ b/tsdb/ooo_head_read_test.go @@ -504,8 +504,8 @@ func TestOOOHeadChunkReader_Chunk(t *testing.T) { queryMaxT: minutes(100), firstInOrderSampleAt: minutes(120), inputSamples: tsdbutil.SampleSlice{ - sample{t: minutes(30), v: float64(0)}, - sample{t: minutes(40), v: float64(0)}, + sample{t: minutes(30), f: float64(0)}, + sample{t: minutes(40), f: float64(0)}, }, expChunkError: false, // ts (in minutes) 0 10 20 30 40 50 60 70 80 90 100 @@ -514,8 +514,8 @@ func TestOOOHeadChunkReader_Chunk(t *testing.T) { // Output Graphically [--------] (With 2 samples) expChunksSamples: []tsdbutil.SampleSlice{ { - sample{t: minutes(30), v: float64(0)}, - sample{t: minutes(40), v: float64(0)}, + sample{t: minutes(30), f: float64(0)}, + sample{t: minutes(40), f: float64(0)}, }, }, }, @@ -526,15 +526,15 @@ func TestOOOHeadChunkReader_Chunk(t *testing.T) { firstInOrderSampleAt: minutes(120), inputSamples: tsdbutil.SampleSlice{ // opts.OOOCapMax is 5 so these will be mmapped to the first mmapped chunk - sample{t: minutes(41), v: float64(0)}, - sample{t: minutes(42), v: float64(0)}, - sample{t: minutes(43), v: float64(0)}, - sample{t: minutes(44), v: float64(0)}, - sample{t: minutes(45), v: float64(0)}, + sample{t: minutes(41), f: float64(0)}, + sample{t: minutes(42), f: float64(0)}, + sample{t: minutes(43), f: float64(0)}, + sample{t: minutes(44), f: float64(0)}, + sample{t: minutes(45), f: float64(0)}, // The following samples will go to the head chunk, and we want it // to overlap with the previous chunk - sample{t: minutes(30), v: float64(1)}, - sample{t: minutes(50), v: float64(1)}, + sample{t: minutes(30), f: float64(1)}, + sample{t: minutes(50), f: float64(1)}, }, expChunkError: false, // ts (in minutes) 0 10 20 30 40 50 60 70 80 90 100 @@ -544,13 +544,13 @@ func TestOOOHeadChunkReader_Chunk(t *testing.T) { // Output Graphically [-----------------] (With 7 samples) expChunksSamples: []tsdbutil.SampleSlice{ { - sample{t: minutes(30), v: float64(1)}, - sample{t: minutes(41), v: float64(0)}, - sample{t: minutes(42), v: float64(0)}, - sample{t: minutes(43), v: float64(0)}, - sample{t: minutes(44), v: float64(0)}, - sample{t: minutes(45), v: float64(0)}, - sample{t: minutes(50), v: float64(1)}, + sample{t: minutes(30), f: float64(1)}, + sample{t: minutes(41), f: float64(0)}, + sample{t: minutes(42), f: float64(0)}, + sample{t: minutes(43), f: float64(0)}, + sample{t: minutes(44), f: float64(0)}, + sample{t: minutes(45), f: float64(0)}, + sample{t: minutes(50), f: float64(1)}, }, }, }, @@ -561,26 +561,26 @@ func TestOOOHeadChunkReader_Chunk(t *testing.T) { firstInOrderSampleAt: minutes(120), inputSamples: tsdbutil.SampleSlice{ // Chunk 0 - sample{t: minutes(10), v: float64(0)}, - sample{t: minutes(12), v: float64(0)}, - sample{t: minutes(14), v: float64(0)}, - sample{t: minutes(16), v: float64(0)}, - sample{t: minutes(20), v: float64(0)}, + sample{t: minutes(10), f: float64(0)}, + sample{t: minutes(12), f: float64(0)}, + sample{t: minutes(14), f: float64(0)}, + sample{t: minutes(16), f: float64(0)}, + sample{t: minutes(20), f: float64(0)}, // Chunk 1 - sample{t: minutes(20), v: float64(1)}, - sample{t: minutes(22), v: float64(1)}, - sample{t: minutes(24), v: float64(1)}, - sample{t: minutes(26), v: float64(1)}, - sample{t: minutes(29), v: float64(1)}, + sample{t: minutes(20), f: float64(1)}, + sample{t: minutes(22), f: float64(1)}, + sample{t: minutes(24), f: float64(1)}, + sample{t: minutes(26), f: float64(1)}, + sample{t: minutes(29), f: float64(1)}, // Chunk 2 - sample{t: minutes(30), v: float64(2)}, - sample{t: minutes(32), v: float64(2)}, - sample{t: minutes(34), v: float64(2)}, - sample{t: minutes(36), v: float64(2)}, - sample{t: minutes(40), v: float64(2)}, + sample{t: minutes(30), f: float64(2)}, + sample{t: minutes(32), f: float64(2)}, + sample{t: minutes(34), f: float64(2)}, + sample{t: minutes(36), f: float64(2)}, + sample{t: minutes(40), f: float64(2)}, // Head - sample{t: minutes(40), v: float64(3)}, - sample{t: minutes(50), v: float64(3)}, + sample{t: minutes(40), f: float64(3)}, + sample{t: minutes(50), f: float64(3)}, }, expChunkError: false, // ts (in minutes) 0 10 20 30 40 50 60 70 80 90 100 @@ -592,23 +592,23 @@ func TestOOOHeadChunkReader_Chunk(t *testing.T) { // Output Graphically [----------------][-----------------] expChunksSamples: []tsdbutil.SampleSlice{ { - sample{t: minutes(10), v: float64(0)}, - sample{t: minutes(12), v: float64(0)}, - sample{t: minutes(14), v: float64(0)}, - sample{t: minutes(16), v: float64(0)}, - sample{t: minutes(20), v: float64(1)}, - sample{t: minutes(22), v: float64(1)}, - sample{t: minutes(24), v: float64(1)}, - sample{t: minutes(26), v: float64(1)}, - sample{t: minutes(29), v: float64(1)}, + sample{t: minutes(10), f: float64(0)}, + sample{t: minutes(12), f: float64(0)}, + sample{t: minutes(14), f: float64(0)}, + sample{t: minutes(16), f: float64(0)}, + sample{t: minutes(20), f: float64(1)}, + sample{t: minutes(22), f: float64(1)}, + sample{t: minutes(24), f: float64(1)}, + sample{t: minutes(26), f: float64(1)}, + sample{t: minutes(29), f: float64(1)}, }, { - sample{t: minutes(30), v: float64(2)}, - sample{t: minutes(32), v: float64(2)}, - sample{t: minutes(34), v: float64(2)}, - sample{t: minutes(36), v: float64(2)}, - sample{t: minutes(40), v: float64(3)}, - sample{t: minutes(50), v: float64(3)}, + sample{t: minutes(30), f: float64(2)}, + sample{t: minutes(32), f: float64(2)}, + sample{t: minutes(34), f: float64(2)}, + sample{t: minutes(36), f: float64(2)}, + sample{t: minutes(40), f: float64(3)}, + sample{t: minutes(50), f: float64(3)}, }, }, }, @@ -619,26 +619,26 @@ func TestOOOHeadChunkReader_Chunk(t *testing.T) { firstInOrderSampleAt: minutes(120), inputSamples: tsdbutil.SampleSlice{ // Chunk 0 - sample{t: minutes(40), v: float64(0)}, - sample{t: minutes(42), v: float64(0)}, - sample{t: minutes(44), v: float64(0)}, - sample{t: minutes(46), v: float64(0)}, - sample{t: minutes(50), v: float64(0)}, + sample{t: minutes(40), f: float64(0)}, + sample{t: minutes(42), f: float64(0)}, + sample{t: minutes(44), f: float64(0)}, + sample{t: minutes(46), f: float64(0)}, + sample{t: minutes(50), f: float64(0)}, // Chunk 1 - sample{t: minutes(30), v: float64(1)}, - sample{t: minutes(32), v: float64(1)}, - sample{t: minutes(34), v: float64(1)}, - sample{t: minutes(36), v: float64(1)}, - sample{t: minutes(40), v: float64(1)}, + sample{t: minutes(30), f: float64(1)}, + sample{t: minutes(32), f: float64(1)}, + sample{t: minutes(34), f: float64(1)}, + sample{t: minutes(36), f: float64(1)}, + sample{t: minutes(40), f: float64(1)}, // Chunk 2 - sample{t: minutes(20), v: float64(2)}, - sample{t: minutes(22), v: float64(2)}, - sample{t: minutes(24), v: float64(2)}, - sample{t: minutes(26), v: float64(2)}, - sample{t: minutes(29), v: float64(2)}, + sample{t: minutes(20), f: float64(2)}, + sample{t: minutes(22), f: float64(2)}, + sample{t: minutes(24), f: float64(2)}, + sample{t: minutes(26), f: float64(2)}, + sample{t: minutes(29), f: float64(2)}, // Head - sample{t: minutes(10), v: float64(3)}, - sample{t: minutes(20), v: float64(3)}, + sample{t: minutes(10), f: float64(3)}, + sample{t: minutes(20), f: float64(3)}, }, expChunkError: false, // ts (in minutes) 0 10 20 30 40 50 60 70 80 90 100 @@ -650,23 +650,23 @@ func TestOOOHeadChunkReader_Chunk(t *testing.T) { // Output Graphically [----------------][-----------------] expChunksSamples: []tsdbutil.SampleSlice{ { - sample{t: minutes(10), v: float64(3)}, - sample{t: minutes(20), v: float64(2)}, - sample{t: minutes(22), v: float64(2)}, - sample{t: minutes(24), v: float64(2)}, - sample{t: minutes(26), v: float64(2)}, - sample{t: minutes(29), v: float64(2)}, + sample{t: minutes(10), f: float64(3)}, + sample{t: minutes(20), f: float64(2)}, + sample{t: minutes(22), f: float64(2)}, + sample{t: minutes(24), f: float64(2)}, + sample{t: minutes(26), f: float64(2)}, + sample{t: minutes(29), f: float64(2)}, }, { - sample{t: minutes(30), v: float64(1)}, - sample{t: minutes(32), v: float64(1)}, - sample{t: minutes(34), v: float64(1)}, - sample{t: minutes(36), v: float64(1)}, - sample{t: minutes(40), v: float64(0)}, - sample{t: minutes(42), v: float64(0)}, - sample{t: minutes(44), v: float64(0)}, - sample{t: minutes(46), v: float64(0)}, - sample{t: minutes(50), v: float64(0)}, + sample{t: minutes(30), f: float64(1)}, + sample{t: minutes(32), f: float64(1)}, + sample{t: minutes(34), f: float64(1)}, + sample{t: minutes(36), f: float64(1)}, + sample{t: minutes(40), f: float64(0)}, + sample{t: minutes(42), f: float64(0)}, + sample{t: minutes(44), f: float64(0)}, + sample{t: minutes(46), f: float64(0)}, + sample{t: minutes(50), f: float64(0)}, }, }, }, @@ -677,26 +677,26 @@ func TestOOOHeadChunkReader_Chunk(t *testing.T) { firstInOrderSampleAt: minutes(120), inputSamples: tsdbutil.SampleSlice{ // Chunk 0 - sample{t: minutes(10), v: float64(0)}, - sample{t: minutes(12), v: float64(0)}, - sample{t: minutes(14), v: float64(0)}, - sample{t: minutes(16), v: float64(0)}, - sample{t: minutes(18), v: float64(0)}, + sample{t: minutes(10), f: float64(0)}, + sample{t: minutes(12), f: float64(0)}, + sample{t: minutes(14), f: float64(0)}, + sample{t: minutes(16), f: float64(0)}, + sample{t: minutes(18), f: float64(0)}, // Chunk 1 - sample{t: minutes(20), v: float64(1)}, - sample{t: minutes(22), v: float64(1)}, - sample{t: minutes(24), v: float64(1)}, - sample{t: minutes(26), v: float64(1)}, - sample{t: minutes(28), v: float64(1)}, + sample{t: minutes(20), f: float64(1)}, + sample{t: minutes(22), f: float64(1)}, + sample{t: minutes(24), f: float64(1)}, + sample{t: minutes(26), f: float64(1)}, + sample{t: minutes(28), f: float64(1)}, // Chunk 2 - sample{t: minutes(30), v: float64(2)}, - sample{t: minutes(32), v: float64(2)}, - sample{t: minutes(34), v: float64(2)}, - sample{t: minutes(36), v: float64(2)}, - sample{t: minutes(38), v: float64(2)}, + sample{t: minutes(30), f: float64(2)}, + sample{t: minutes(32), f: float64(2)}, + sample{t: minutes(34), f: float64(2)}, + sample{t: minutes(36), f: float64(2)}, + sample{t: minutes(38), f: float64(2)}, // Head - sample{t: minutes(40), v: float64(3)}, - sample{t: minutes(42), v: float64(3)}, + sample{t: minutes(40), f: float64(3)}, + sample{t: minutes(42), f: float64(3)}, }, expChunkError: false, // ts (in minutes) 0 10 20 30 40 50 60 70 80 90 100 @@ -708,29 +708,29 @@ func TestOOOHeadChunkReader_Chunk(t *testing.T) { // Output Graphically [-------][-------][-------][--------] expChunksSamples: []tsdbutil.SampleSlice{ { - sample{t: minutes(10), v: float64(0)}, - sample{t: minutes(12), v: float64(0)}, - sample{t: minutes(14), v: float64(0)}, - sample{t: minutes(16), v: float64(0)}, - sample{t: minutes(18), v: float64(0)}, + sample{t: minutes(10), f: float64(0)}, + sample{t: minutes(12), f: float64(0)}, + sample{t: minutes(14), f: float64(0)}, + sample{t: minutes(16), f: float64(0)}, + sample{t: minutes(18), f: float64(0)}, }, { - sample{t: minutes(20), v: float64(1)}, - sample{t: minutes(22), v: float64(1)}, - sample{t: minutes(24), v: float64(1)}, - sample{t: minutes(26), v: float64(1)}, - sample{t: minutes(28), v: float64(1)}, + sample{t: minutes(20), f: float64(1)}, + sample{t: minutes(22), f: float64(1)}, + sample{t: minutes(24), f: float64(1)}, + sample{t: minutes(26), f: float64(1)}, + sample{t: minutes(28), f: float64(1)}, }, { - sample{t: minutes(30), v: float64(2)}, - sample{t: minutes(32), v: float64(2)}, - sample{t: minutes(34), v: float64(2)}, - sample{t: minutes(36), v: float64(2)}, - sample{t: minutes(38), v: float64(2)}, + sample{t: minutes(30), f: float64(2)}, + sample{t: minutes(32), f: float64(2)}, + sample{t: minutes(34), f: float64(2)}, + sample{t: minutes(36), f: float64(2)}, + sample{t: minutes(38), f: float64(2)}, }, { - sample{t: minutes(40), v: float64(3)}, - sample{t: minutes(42), v: float64(3)}, + sample{t: minutes(40), f: float64(3)}, + sample{t: minutes(42), f: float64(3)}, }, }, }, @@ -741,20 +741,20 @@ func TestOOOHeadChunkReader_Chunk(t *testing.T) { firstInOrderSampleAt: minutes(120), inputSamples: tsdbutil.SampleSlice{ // Chunk 0 - sample{t: minutes(10), v: float64(0)}, - sample{t: minutes(15), v: float64(0)}, - sample{t: minutes(20), v: float64(0)}, - sample{t: minutes(25), v: float64(0)}, - sample{t: minutes(30), v: float64(0)}, + sample{t: minutes(10), f: float64(0)}, + sample{t: minutes(15), f: float64(0)}, + sample{t: minutes(20), f: float64(0)}, + sample{t: minutes(25), f: float64(0)}, + sample{t: minutes(30), f: float64(0)}, // Chunk 1 - sample{t: minutes(20), v: float64(1)}, - sample{t: minutes(25), v: float64(1)}, - sample{t: minutes(30), v: float64(1)}, - sample{t: minutes(35), v: float64(1)}, - sample{t: minutes(42), v: float64(1)}, + sample{t: minutes(20), f: float64(1)}, + sample{t: minutes(25), f: float64(1)}, + sample{t: minutes(30), f: float64(1)}, + sample{t: minutes(35), f: float64(1)}, + sample{t: minutes(42), f: float64(1)}, // Chunk 2 Head - sample{t: minutes(32), v: float64(2)}, - sample{t: minutes(50), v: float64(2)}, + sample{t: minutes(32), f: float64(2)}, + sample{t: minutes(50), f: float64(2)}, }, expChunkError: false, // ts (in minutes) 0 10 20 30 40 50 60 70 80 90 100 @@ -765,15 +765,15 @@ func TestOOOHeadChunkReader_Chunk(t *testing.T) { // Output Graphically [-----------------------------------] expChunksSamples: []tsdbutil.SampleSlice{ { - sample{t: minutes(10), v: float64(0)}, - sample{t: minutes(15), v: float64(0)}, - sample{t: minutes(20), v: float64(1)}, - sample{t: minutes(25), v: float64(1)}, - sample{t: minutes(30), v: float64(1)}, - sample{t: minutes(32), v: float64(2)}, - sample{t: minutes(35), v: float64(1)}, - sample{t: minutes(42), v: float64(1)}, - sample{t: minutes(50), v: float64(2)}, + sample{t: minutes(10), f: float64(0)}, + sample{t: minutes(15), f: float64(0)}, + sample{t: minutes(20), f: float64(1)}, + sample{t: minutes(25), f: float64(1)}, + sample{t: minutes(30), f: float64(1)}, + sample{t: minutes(32), f: float64(2)}, + sample{t: minutes(35), f: float64(1)}, + sample{t: minutes(42), f: float64(1)}, + sample{t: minutes(50), f: float64(2)}, }, }, }, @@ -784,20 +784,20 @@ func TestOOOHeadChunkReader_Chunk(t *testing.T) { firstInOrderSampleAt: minutes(120), inputSamples: tsdbutil.SampleSlice{ // Chunk 0 - sample{t: minutes(10), v: float64(0)}, - sample{t: minutes(15), v: float64(0)}, - sample{t: minutes(20), v: float64(0)}, - sample{t: minutes(25), v: float64(0)}, - sample{t: minutes(30), v: float64(0)}, + sample{t: minutes(10), f: float64(0)}, + sample{t: minutes(15), f: float64(0)}, + sample{t: minutes(20), f: float64(0)}, + sample{t: minutes(25), f: float64(0)}, + sample{t: minutes(30), f: float64(0)}, // Chunk 1 - sample{t: minutes(20), v: float64(1)}, - sample{t: minutes(25), v: float64(1)}, - sample{t: minutes(30), v: float64(1)}, - sample{t: minutes(35), v: float64(1)}, - sample{t: minutes(42), v: float64(1)}, + sample{t: minutes(20), f: float64(1)}, + sample{t: minutes(25), f: float64(1)}, + sample{t: minutes(30), f: float64(1)}, + sample{t: minutes(35), f: float64(1)}, + sample{t: minutes(42), f: float64(1)}, // Chunk 2 Head - sample{t: minutes(32), v: float64(2)}, - sample{t: minutes(50), v: float64(2)}, + sample{t: minutes(32), f: float64(2)}, + sample{t: minutes(50), f: float64(2)}, }, expChunkError: false, // ts (in minutes) 0 10 20 30 40 50 60 70 80 90 100 @@ -808,15 +808,15 @@ func TestOOOHeadChunkReader_Chunk(t *testing.T) { // Output Graphically [-----------------------------------] expChunksSamples: []tsdbutil.SampleSlice{ { - sample{t: minutes(10), v: float64(0)}, - sample{t: minutes(15), v: float64(0)}, - sample{t: minutes(20), v: float64(1)}, - sample{t: minutes(25), v: float64(1)}, - sample{t: minutes(30), v: float64(1)}, - sample{t: minutes(32), v: float64(2)}, - sample{t: minutes(35), v: float64(1)}, - sample{t: minutes(42), v: float64(1)}, - sample{t: minutes(50), v: float64(2)}, + sample{t: minutes(10), f: float64(0)}, + sample{t: minutes(15), f: float64(0)}, + sample{t: minutes(20), f: float64(1)}, + sample{t: minutes(25), f: float64(1)}, + sample{t: minutes(30), f: float64(1)}, + sample{t: minutes(32), f: float64(2)}, + sample{t: minutes(35), f: float64(1)}, + sample{t: minutes(42), f: float64(1)}, + sample{t: minutes(50), f: float64(2)}, }, }, }, @@ -833,7 +833,7 @@ func TestOOOHeadChunkReader_Chunk(t *testing.T) { // OOO few samples for s1. app = db.Appender(context.Background()) for _, s := range tc.inputSamples { - appendSample(app, s1, s.T(), s.V()) + appendSample(app, s1, s.T(), s.F()) } require.NoError(t, app.Commit()) @@ -855,7 +855,7 @@ func TestOOOHeadChunkReader_Chunk(t *testing.T) { it := c.Iterator(nil) for it.Next() == chunkenc.ValFloat { t, v := it.At() - resultSamples = append(resultSamples, sample{t: t, v: v}) + resultSamples = append(resultSamples, sample{t: t, f: v}) } require.Equal(t, tc.expChunksSamples[i], resultSamples) } @@ -902,19 +902,19 @@ func TestOOOHeadChunkReader_Chunk_ConsistentQueryResponseDespiteOfHeadExpanding( firstInOrderSampleAt: minutes(120), initialSamples: tsdbutil.SampleSlice{ // Chunk 0 - sample{t: minutes(20), v: float64(0)}, - sample{t: minutes(22), v: float64(0)}, - sample{t: minutes(24), v: float64(0)}, - sample{t: minutes(26), v: float64(0)}, - sample{t: minutes(30), v: float64(0)}, + sample{t: minutes(20), f: float64(0)}, + sample{t: minutes(22), f: float64(0)}, + sample{t: minutes(24), f: float64(0)}, + sample{t: minutes(26), f: float64(0)}, + sample{t: minutes(30), f: float64(0)}, // Chunk 1 Head - sample{t: minutes(25), v: float64(1)}, - sample{t: minutes(35), v: float64(1)}, + sample{t: minutes(25), f: float64(1)}, + sample{t: minutes(35), f: float64(1)}, }, samplesAfterSeriesCall: tsdbutil.SampleSlice{ - sample{t: minutes(10), v: float64(1)}, - sample{t: minutes(32), v: float64(1)}, - sample{t: minutes(50), v: float64(1)}, + sample{t: minutes(10), f: float64(1)}, + sample{t: minutes(32), f: float64(1)}, + sample{t: minutes(50), f: float64(1)}, }, expChunkError: false, // ts (in minutes) 0 10 20 30 40 50 60 70 80 90 100 @@ -926,14 +926,14 @@ func TestOOOHeadChunkReader_Chunk_ConsistentQueryResponseDespiteOfHeadExpanding( // Output Graphically [------------] (With 8 samples, samples newer than lastmint or older than lastmaxt are omitted but the ones in between are kept) expChunksSamples: []tsdbutil.SampleSlice{ { - sample{t: minutes(20), v: float64(0)}, - sample{t: minutes(22), v: float64(0)}, - sample{t: minutes(24), v: float64(0)}, - sample{t: minutes(25), v: float64(1)}, - sample{t: minutes(26), v: float64(0)}, - sample{t: minutes(30), v: float64(0)}, - sample{t: minutes(32), v: float64(1)}, // This sample was added after Series() but before Chunk() and its in between the lastmint and maxt so it should be kept - sample{t: minutes(35), v: float64(1)}, + sample{t: minutes(20), f: float64(0)}, + sample{t: minutes(22), f: float64(0)}, + sample{t: minutes(24), f: float64(0)}, + sample{t: minutes(25), f: float64(1)}, + sample{t: minutes(26), f: float64(0)}, + sample{t: minutes(30), f: float64(0)}, + sample{t: minutes(32), f: float64(1)}, // This sample was added after Series() but before Chunk() and its in between the lastmint and maxt so it should be kept + sample{t: minutes(35), f: float64(1)}, }, }, }, @@ -944,22 +944,22 @@ func TestOOOHeadChunkReader_Chunk_ConsistentQueryResponseDespiteOfHeadExpanding( firstInOrderSampleAt: minutes(120), initialSamples: tsdbutil.SampleSlice{ // Chunk 0 - sample{t: minutes(20), v: float64(0)}, - sample{t: minutes(22), v: float64(0)}, - sample{t: minutes(24), v: float64(0)}, - sample{t: minutes(26), v: float64(0)}, - sample{t: minutes(30), v: float64(0)}, + sample{t: minutes(20), f: float64(0)}, + sample{t: minutes(22), f: float64(0)}, + sample{t: minutes(24), f: float64(0)}, + sample{t: minutes(26), f: float64(0)}, + sample{t: minutes(30), f: float64(0)}, // Chunk 1 Head - sample{t: minutes(25), v: float64(1)}, - sample{t: minutes(35), v: float64(1)}, + sample{t: minutes(25), f: float64(1)}, + sample{t: minutes(35), f: float64(1)}, }, samplesAfterSeriesCall: tsdbutil.SampleSlice{ - sample{t: minutes(10), v: float64(1)}, - sample{t: minutes(32), v: float64(1)}, - sample{t: minutes(50), v: float64(1)}, + sample{t: minutes(10), f: float64(1)}, + sample{t: minutes(32), f: float64(1)}, + sample{t: minutes(50), f: float64(1)}, // Chunk 1 gets mmapped and Chunk 2, the new head is born - sample{t: minutes(25), v: float64(2)}, - sample{t: minutes(31), v: float64(2)}, + sample{t: minutes(25), f: float64(2)}, + sample{t: minutes(31), f: float64(2)}, }, expChunkError: false, // ts (in minutes) 0 10 20 30 40 50 60 70 80 90 100 @@ -972,14 +972,14 @@ func TestOOOHeadChunkReader_Chunk_ConsistentQueryResponseDespiteOfHeadExpanding( // Output Graphically [------------] (8 samples) It has 5 from Chunk 0 and 3 from Chunk 1 expChunksSamples: []tsdbutil.SampleSlice{ { - sample{t: minutes(20), v: float64(0)}, - sample{t: minutes(22), v: float64(0)}, - sample{t: minutes(24), v: float64(0)}, - sample{t: minutes(25), v: float64(1)}, - sample{t: minutes(26), v: float64(0)}, - sample{t: minutes(30), v: float64(0)}, - sample{t: minutes(32), v: float64(1)}, // This sample was added after Series() but before Chunk() and its in between the lastmint and maxt so it should be kept - sample{t: minutes(35), v: float64(1)}, + sample{t: minutes(20), f: float64(0)}, + sample{t: minutes(22), f: float64(0)}, + sample{t: minutes(24), f: float64(0)}, + sample{t: minutes(25), f: float64(1)}, + sample{t: minutes(26), f: float64(0)}, + sample{t: minutes(30), f: float64(0)}, + sample{t: minutes(32), f: float64(1)}, // This sample was added after Series() but before Chunk() and its in between the lastmint and maxt so it should be kept + sample{t: minutes(35), f: float64(1)}, }, }, }, @@ -996,7 +996,7 @@ func TestOOOHeadChunkReader_Chunk_ConsistentQueryResponseDespiteOfHeadExpanding( // OOO few samples for s1. app = db.Appender(context.Background()) for _, s := range tc.initialSamples { - appendSample(app, s1, s.T(), s.V()) + appendSample(app, s1, s.T(), s.F()) } require.NoError(t, app.Commit()) @@ -1013,7 +1013,7 @@ func TestOOOHeadChunkReader_Chunk_ConsistentQueryResponseDespiteOfHeadExpanding( // OOO few samples for s1. app = db.Appender(context.Background()) for _, s := range tc.samplesAfterSeriesCall { - appendSample(app, s1, s.T(), s.V()) + appendSample(app, s1, s.T(), s.F()) } require.NoError(t, app.Commit()) @@ -1026,7 +1026,7 @@ func TestOOOHeadChunkReader_Chunk_ConsistentQueryResponseDespiteOfHeadExpanding( it := c.Iterator(nil) for it.Next() == chunkenc.ValFloat { ts, v := it.At() - resultSamples = append(resultSamples, sample{t: ts, v: v}) + resultSamples = append(resultSamples, sample{t: ts, f: v}) } require.Equal(t, tc.expChunksSamples[i], resultSamples) } diff --git a/tsdb/ooo_head_test.go b/tsdb/ooo_head_test.go index dcab28b61f..6914087449 100644 --- a/tsdb/ooo_head_test.go +++ b/tsdb/ooo_head_test.go @@ -52,7 +52,7 @@ func TestOOOInsert(t *testing.T) { chunk := NewOOOChunk() chunk.samples = makeEvenSampleSlice(numPreExisting) newSample := samplify(valOdd(insertPos)) - chunk.Insert(newSample.t, newSample.v) + chunk.Insert(newSample.t, newSample.f) var expSamples []sample // Our expected new samples slice, will be first the original samples. @@ -81,9 +81,9 @@ func TestOOOInsertDuplicate(t *testing.T) { chunk.samples = makeEvenSampleSlice(num) dupSample := chunk.samples[dupPos] - dupSample.v = 0.123 + dupSample.f = 0.123 - ok := chunk.Insert(dupSample.t, dupSample.v) + ok := chunk.Insert(dupSample.t, dupSample.f) expSamples := makeEvenSampleSlice(num) // We expect no change. require.False(t, ok) diff --git a/tsdb/querier_test.go b/tsdb/querier_test.go index cf9867a4f3..fa3dd2418f 100644 --- a/tsdb/querier_test.go +++ b/tsdb/querier_test.go @@ -132,7 +132,7 @@ func createIdxChkReaders(t *testing.T, tc []seriesSamples) (IndexReader, ChunkRe chunk := chunkenc.NewXORChunk() app, _ := chunk.Appender() for _, smpl := range chk { - app.Append(smpl.t, smpl.v) + app.Append(smpl.t, smpl.f) } chkReader[chunkRef] = chunk chunkRef++ @@ -479,7 +479,7 @@ func TestBlockQuerier_AgainstHeadWithOpenChunks(t *testing.T) { for _, s := range testData { for _, chk := range s.chunks { for _, sample := range chk { - _, err = app.Append(0, labels.FromMap(s.lset), sample.t, sample.v) + _, err = app.Append(0, labels.FromMap(s.lset), sample.t, sample.f) require.NoError(t, err) } } @@ -882,7 +882,7 @@ func TestPopulateWithTombSeriesIterators(t *testing.T) { if tc.seekSuccess { // After successful seek iterator is ready. Grab the value. t, v := it.At() - r = append(r, sample{t: t, v: v}) + r = append(r, sample{t: t, f: v}) } } expandedResult, err := storage.ExpandSamples(it, newSample) @@ -1054,8 +1054,8 @@ func TestDeletedIterator(t *testing.T) { act := make([]sample, 1000) for i := 0; i < 1000; i++ { act[i].t = int64(i) - act[i].v = rand.Float64() - app.Append(act[i].t, act[i].v) + act[i].f = rand.Float64() + app.Append(act[i].t, act[i].f) } cases := []struct { @@ -1090,7 +1090,7 @@ func TestDeletedIterator(t *testing.T) { ts, v := it.At() require.Equal(t, act[i].t, ts) - require.Equal(t, act[i].v, v) + require.Equal(t, act[i].f, v) } // There has been an extra call to Next(). i++ @@ -1114,8 +1114,8 @@ func TestDeletedIterator_WithSeek(t *testing.T) { act := make([]sample, 1000) for i := 0; i < 1000; i++ { act[i].t = int64(i) - act[i].v = float64(i) - app.Append(act[i].t, act[i].v) + act[i].f = float64(i) + app.Append(act[i].t, act[i].f) } cases := []struct { diff --git a/tsdb/tsdbutil/chunks.go b/tsdb/tsdbutil/chunks.go index 4e025c5e6b..02a7dd6198 100644 --- a/tsdb/tsdbutil/chunks.go +++ b/tsdb/tsdbutil/chunks.go @@ -28,7 +28,7 @@ type Samples interface { type Sample interface { T() int64 - V() float64 // TODO(beorn7): Rename to F(). + F() float64 H() *histogram.Histogram FH() *histogram.FloatHistogram Type() chunkenc.ValueType @@ -69,7 +69,7 @@ func ChunkFromSamplesGeneric(s Samples) chunks.Meta { for i := 0; i < s.Len(); i++ { switch sampleType { case chunkenc.ValFloat: - ca.Append(s.Get(i).T(), s.Get(i).V()) + ca.Append(s.Get(i).T(), s.Get(i).F()) case chunkenc.ValHistogram: ca.AppendHistogram(s.Get(i).T(), s.Get(i).H()) case chunkenc.ValFloatHistogram: @@ -87,7 +87,7 @@ func ChunkFromSamplesGeneric(s Samples) chunks.Meta { type sample struct { t int64 - v float64 + f float64 h *histogram.Histogram fh *histogram.FloatHistogram } @@ -96,8 +96,8 @@ func (s sample) T() int64 { return s.t } -func (s sample) V() float64 { - return s.v +func (s sample) F() float64 { + return s.f } func (s sample) H() *histogram.Histogram { @@ -123,7 +123,7 @@ func (s sample) Type() chunkenc.ValueType { func PopulatedChunk(numSamples int, minTime int64) chunks.Meta { samples := make([]Sample, numSamples) for i := 0; i < numSamples; i++ { - samples[i] = sample{t: minTime + int64(i*1000), v: 1.0} + samples[i] = sample{t: minTime + int64(i*1000), f: 1.0} } return ChunkFromSamples(samples) } @@ -133,7 +133,7 @@ func GenerateSamples(start, numSamples int) []Sample { return generateSamples(start, numSamples, func(i int) Sample { return sample{ t: int64(i), - v: float64(i), + f: float64(i), } }) } diff --git a/util/jsonutil/marshal.go b/util/jsonutil/marshal.go index 4400385d0d..d715eabe68 100644 --- a/util/jsonutil/marshal.go +++ b/util/jsonutil/marshal.go @@ -45,12 +45,12 @@ func MarshalTimestamp(t int64, stream *jsoniter.Stream) { } // MarshalFloat marshals a float value using the passed jsoniter stream. -func MarshalFloat(v float64, stream *jsoniter.Stream) { +func MarshalFloat(f float64, stream *jsoniter.Stream) { stream.WriteRaw(`"`) // Taken from https://github.com/json-iterator/go/blob/master/stream_float.go#L71 as a workaround // to https://github.com/json-iterator/go/issues/365 (jsoniter, to follow json standard, doesn't allow inf/nan). buf := stream.Buffer() - abs := math.Abs(v) + abs := math.Abs(f) fmt := byte('f') // Note: Must use float32 comparisons for underlying float32 value to get precise cutoffs right. if abs != 0 { @@ -58,7 +58,7 @@ func MarshalFloat(v float64, stream *jsoniter.Stream) { fmt = 'e' } } - buf = strconv.AppendFloat(buf, v, fmt, -1, 64) + buf = strconv.AppendFloat(buf, f, fmt, -1, 64) stream.SetBuffer(buf) stream.WriteRaw(`"`) } diff --git a/web/federate.go b/web/federate.go index 15e31b1a3e..b0f4c06108 100644 --- a/web/federate.go +++ b/web/federate.go @@ -132,7 +132,7 @@ Loop: t = sample.T() switch sample.Type() { case chunkenc.ValFloat: - f = sample.V() + f = sample.F() case chunkenc.ValHistogram: fh = sample.H().ToFloat() case chunkenc.ValFloatHistogram: From 551de0346fe157f0910b99a2c99e8285d96d1b1c Mon Sep 17 00:00:00 2001 From: beorn7 Date: Wed, 5 Apr 2023 14:31:05 +0200 Subject: [PATCH 089/231] promql: Do not return nil slices to the pool Signed-off-by: beorn7 --- promql/engine.go | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/promql/engine.go b/promql/engine.go index 4c6751088f..b49be244f1 100644 --- a/promql/engine.go +++ b/promql/engine.go @@ -1874,7 +1874,9 @@ func getFPointSlice(sz int) []FPoint { } func putFPointSlice(p []FPoint) { - fPointPool.Put(p[:0]) + if p != nil { + fPointPool.Put(p[:0]) + } } func getHPointSlice(sz int) []HPoint { @@ -1885,7 +1887,9 @@ func getHPointSlice(sz int) []HPoint { } func putHPointSlice(p []HPoint) { - hPointPool.Put(p[:0]) + if p != nil { + hPointPool.Put(p[:0]) + } } // matrixSelector evaluates a *parser.MatrixSelector expression. From 717a3f8e2546863013e0b3284ae99fa4917115ef Mon Sep 17 00:00:00 2001 From: beorn7 Date: Thu, 13 Apr 2023 17:42:40 +0200 Subject: [PATCH 090/231] storage: Manually expand `genericAdd` for specific types This commit is doing what I would have expected that Go generics do for me. However, the PromQL benchmarks show a significant runtime and allocation increase with `genericAdd`, so this replaces it with hand-coded non-generic versions of it. Signed-off-by: beorn7 --- storage/buffer.go | 192 ++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 179 insertions(+), 13 deletions(-) diff --git a/storage/buffer.go b/storage/buffer.go index 0daadc0444..27ac21661b 100644 --- a/storage/buffer.go +++ b/storage/buffer.go @@ -401,17 +401,17 @@ func (r *sampleRing) add(s tsdbutil.Sample) { switch s := s.(type) { case fSample: if len(r.hBuf)+len(r.fhBuf) == 0 { - r.fBuf = genericAdd(s, r.fBuf, r) + r.fBuf = addF(s, r.fBuf, r) return } case hSample: if len(r.fBuf)+len(r.fhBuf) == 0 { - r.hBuf = genericAdd(s, r.hBuf, r) + r.hBuf = addH(s, r.hBuf, r) return } case fhSample: if len(r.fBuf)+len(r.hBuf) == 0 { - r.fhBuf = genericAdd(s, r.fhBuf, r) + r.fhBuf = addFH(s, r.fhBuf, r) return } } @@ -435,7 +435,7 @@ func (r *sampleRing) add(s tsdbutil.Sample) { r.fhBuf = nil } } - r.buf = genericAdd(s, r.buf, r) + r.buf = addSample(s, r.buf, r) } // addF is a version of the add method specialized for fSample. @@ -443,13 +443,13 @@ func (r *sampleRing) addF(s fSample) { switch { case len(r.buf) > 0: // Already have interface samples. Add to the interface buf. - r.buf = genericAdd[tsdbutil.Sample](s, r.buf, r) + r.buf = addSample(s, r.buf, r) case len(r.hBuf)+len(r.fhBuf) > 0: // Already have specialized samples that are not fSamples. // Need to call the checked add method for conversion. r.add(s) default: - r.fBuf = genericAdd(s, r.fBuf, r) + r.fBuf = addF(s, r.fBuf, r) } } @@ -458,13 +458,13 @@ func (r *sampleRing) addH(s hSample) { switch { case len(r.buf) > 0: // Already have interface samples. Add to the interface buf. - r.buf = genericAdd[tsdbutil.Sample](s, r.buf, r) + r.buf = addSample(s, r.buf, r) case len(r.fBuf)+len(r.fhBuf) > 0: // Already have samples that are not hSamples. // Need to call the checked add method for conversion. r.add(s) default: - r.hBuf = genericAdd(s, r.hBuf, r) + r.hBuf = addH(s, r.hBuf, r) } } @@ -473,25 +473,191 @@ func (r *sampleRing) addFH(s fhSample) { switch { case len(r.buf) > 0: // Already have interface samples. Add to the interface buf. - r.buf = genericAdd[tsdbutil.Sample](s, r.buf, r) + r.buf = addSample(s, r.buf, r) case len(r.fBuf)+len(r.hBuf) > 0: // Already have samples that are not fhSamples. // Need to call the checked add method for conversion. r.add(s) default: - r.fhBuf = genericAdd(s, r.fhBuf, r) + r.fhBuf = addFH(s, r.fhBuf, r) } } -func genericAdd[T tsdbutil.Sample](s T, buf []T, r *sampleRing) []T { +// genericAdd is a generic implementation of adding a tsdbutil.Sample +// implementation to a buffer of a sample ring. However, the Go compiler +// currently (go1.20) decides to not expand the code during compile time, but +// creates dynamic code to handle the different types. That has a significant +// overhead during runtime, noticeable in PromQL benchmarks. For example, the +// "RangeQuery/expr=rate(a_hundred[1d]),steps=.*" benchmarks show about 7% +// longer runtime, 9% higher allocation size, and 10% more allocations. +// Therefore, genericAdd has been manually implemented for all the types +// (addSample, addF, addH, addFH) below. +// +// func genericAdd[T tsdbutil.Sample](s T, buf []T, r *sampleRing) []T { +// l := len(buf) +// // Grow the ring buffer if it fits no more elements. +// if l == 0 { +// buf = make([]T, 16) +// l = 16 +// } +// if l == r.l { +// newBuf := make([]T, 2*l) +// copy(newBuf[l+r.f:], buf[r.f:]) +// copy(newBuf, buf[:r.f]) +// +// buf = newBuf +// r.i = r.f +// r.f += l +// l = 2 * l +// } else { +// r.i++ +// if r.i >= l { +// r.i -= l +// } +// } +// +// buf[r.i] = s +// r.l++ +// +// // Free head of the buffer of samples that just fell out of the range. +// tmin := s.T() - r.delta +// for buf[r.f].T() < tmin { +// r.f++ +// if r.f >= l { +// r.f -= l +// } +// r.l-- +// } +// return buf +// } + +// addSample is a handcoded specialization of genericAdd (see above). +func addSample(s tsdbutil.Sample, buf []tsdbutil.Sample, r *sampleRing) []tsdbutil.Sample { l := len(buf) // Grow the ring buffer if it fits no more elements. if l == 0 { - buf = make([]T, 16) + buf = make([]tsdbutil.Sample, 16) l = 16 } if l == r.l { - newBuf := make([]T, 2*l) + newBuf := make([]tsdbutil.Sample, 2*l) + copy(newBuf[l+r.f:], buf[r.f:]) + copy(newBuf, buf[:r.f]) + + buf = newBuf + r.i = r.f + r.f += l + l = 2 * l + } else { + r.i++ + if r.i >= l { + r.i -= l + } + } + + buf[r.i] = s + r.l++ + + // Free head of the buffer of samples that just fell out of the range. + tmin := s.T() - r.delta + for buf[r.f].T() < tmin { + r.f++ + if r.f >= l { + r.f -= l + } + r.l-- + } + return buf +} + +// addF is a handcoded specialization of genericAdd (see above). +func addF(s fSample, buf []fSample, r *sampleRing) []fSample { + l := len(buf) + // Grow the ring buffer if it fits no more elements. + if l == 0 { + buf = make([]fSample, 16) + l = 16 + } + if l == r.l { + newBuf := make([]fSample, 2*l) + copy(newBuf[l+r.f:], buf[r.f:]) + copy(newBuf, buf[:r.f]) + + buf = newBuf + r.i = r.f + r.f += l + l = 2 * l + } else { + r.i++ + if r.i >= l { + r.i -= l + } + } + + buf[r.i] = s + r.l++ + + // Free head of the buffer of samples that just fell out of the range. + tmin := s.T() - r.delta + for buf[r.f].T() < tmin { + r.f++ + if r.f >= l { + r.f -= l + } + r.l-- + } + return buf +} + +// addH is a handcoded specialization of genericAdd (see above). +func addH(s hSample, buf []hSample, r *sampleRing) []hSample { + l := len(buf) + // Grow the ring buffer if it fits no more elements. + if l == 0 { + buf = make([]hSample, 16) + l = 16 + } + if l == r.l { + newBuf := make([]hSample, 2*l) + copy(newBuf[l+r.f:], buf[r.f:]) + copy(newBuf, buf[:r.f]) + + buf = newBuf + r.i = r.f + r.f += l + l = 2 * l + } else { + r.i++ + if r.i >= l { + r.i -= l + } + } + + buf[r.i] = s + r.l++ + + // Free head of the buffer of samples that just fell out of the range. + tmin := s.T() - r.delta + for buf[r.f].T() < tmin { + r.f++ + if r.f >= l { + r.f -= l + } + r.l-- + } + return buf +} + +// addFH is a handcoded specialization of genericAdd (see above). +func addFH(s fhSample, buf []fhSample, r *sampleRing) []fhSample { + l := len(buf) + // Grow the ring buffer if it fits no more elements. + if l == 0 { + buf = make([]fhSample, 16) + l = 16 + } + if l == r.l { + newBuf := make([]fhSample, 2*l) copy(newBuf[l+r.f:], buf[r.f:]) copy(newBuf, buf[:r.f]) From fb3eb212306cad9c9552ccef60318d5c3c8fff19 Mon Sep 17 00:00:00 2001 From: Matthieu MOREL Date: Sun, 9 Apr 2023 09:08:40 +0200 Subject: [PATCH 091/231] enable gocritic, unconvert and unused linters Signed-off-by: Matthieu MOREL --- .github/workflows/ci.yml | 2 ++ .golangci.yml | 11 ++++++- cmd/prometheus/main.go | 2 +- cmd/prometheus/query_log_test.go | 2 +- cmd/promtool/backfill.go | 2 +- cmd/promtool/rules.go | 2 +- cmd/promtool/unittest.go | 2 +- discovery/kubernetes/endpointslice.go | 2 +- discovery/kubernetes/kubernetes.go | 7 +++-- discovery/legacymanager/registry.go | 2 +- discovery/linode/linode.go | 8 ++--- discovery/marathon/marathon.go | 7 +++-- discovery/ovhcloud/dedicated_server_test.go | 6 ++-- discovery/ovhcloud/vps_test.go | 6 ++-- discovery/registry.go | 2 +- discovery/vultr/vultr.go | 4 +-- .../remote_storage_adapter/influxdb/client.go | 11 ++++--- model/labels/labels.go | 14 +++++---- model/textparse/promparse_test.go | 2 +- promql/bench_test.go | 6 ++-- promql/engine.go | 24 +++++++------- promql/engine_test.go | 2 +- promql/functions.go | 8 ++--- promql/functions_test.go | 2 +- promql/parser/ast.go | 11 ++++--- promql/parser/parse.go | 14 ++++----- promql/parser/parse_test.go | 2 +- promql/parser/printer.go | 21 +++++++------ rules/manager.go | 7 +++-- rules/manager_test.go | 7 +++-- scrape/scrape.go | 6 ++-- scrape/scrape_test.go | 27 ++++++++-------- scrape/target.go | 4 +-- storage/merge.go | 7 +++-- storage/remote/codec.go | 27 ++++++++-------- storage/remote/queue_manager.go | 8 ++--- template/template.go | 2 +- tsdb/agent/db_test.go | 3 +- tsdb/block_test.go | 4 +-- tsdb/chunkenc/bstream.go | 6 ++-- tsdb/chunkenc/float_histogram.go | 2 +- tsdb/chunkenc/histogram.go | 6 ++-- tsdb/chunkenc/varbit.go | 2 +- tsdb/chunkenc/xor.go | 12 +++---- tsdb/chunks/head_chunks_test.go | 4 +-- tsdb/compact.go | 2 +- tsdb/compact_test.go | 6 ---- tsdb/db.go | 6 ++-- tsdb/db_test.go | 4 +-- tsdb/exemplar.go | 18 +++++------ tsdb/head.go | 2 +- tsdb/head_test.go | 31 ++++++++++--------- tsdb/head_wal.go | 4 +-- tsdb/index/index.go | 2 +- tsdb/index/postings.go | 13 ++++---- tsdb/index/postingsstats.go | 6 ++-- tsdb/isolation.go | 2 +- tsdb/querier.go | 26 +++++++++------- tsdb/querier_test.go | 4 +-- tsdb/wal.go | 4 +-- tsdb/wlog/wlog_test.go | 4 +-- util/runtime/limits_default.go | 2 +- util/runtime/statfs_default.go | 8 +++-- web/api/v1/api.go | 4 +-- web/api/v1/api_test.go | 4 +-- web/web.go | 4 +-- 66 files changed, 245 insertions(+), 229 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 17185c5bc0..07b1242c28 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -149,6 +149,8 @@ jobs: - name: Lint uses: golangci/golangci-lint-action@v3.4.0 with: + args: --verbose + skip-cache: true version: v1.51.2 fuzzing: uses: ./.github/workflows/fuzzing.yml diff --git a/.golangci.yml b/.golangci.yml index efa6b2044d..c0c20d425a 100644 --- a/.golangci.yml +++ b/.golangci.yml @@ -1,5 +1,5 @@ run: - deadline: 5m + timeout: 15m skip-files: # Skip autogenerated files. - ^.*\.(pb|y)\.go$ @@ -10,14 +10,23 @@ output: linters: enable: - depguard + - gocritic - gofumpt - goimports - revive - misspell + - unconvert + - unused issues: max-same-issues: 0 exclude-rules: + - linters: + - gocritic + text: "appendAssign" + - linters: + - gocritic + text: "singleCaseSwitch" - path: _test.go linters: - errcheck diff --git a/cmd/prometheus/main.go b/cmd/prometheus/main.go index f4f6af20df..cbe8f503d0 100644 --- a/cmd/prometheus/main.go +++ b/cmd/prometheus/main.go @@ -490,7 +490,7 @@ func main() { if cfgFile.StorageConfig.ExemplarsConfig == nil { cfgFile.StorageConfig.ExemplarsConfig = &config.DefaultExemplarsConfig } - cfg.tsdb.MaxExemplars = int64(cfgFile.StorageConfig.ExemplarsConfig.MaxExemplars) + cfg.tsdb.MaxExemplars = cfgFile.StorageConfig.ExemplarsConfig.MaxExemplars } if cfgFile.StorageConfig.TSDBConfig != nil { cfg.tsdb.OutOfOrderTimeWindow = cfgFile.StorageConfig.TSDBConfig.OutOfOrderTimeWindow diff --git a/cmd/prometheus/query_log_test.go b/cmd/prometheus/query_log_test.go index d5dfbea509..f20f2a22c0 100644 --- a/cmd/prometheus/query_log_test.go +++ b/cmd/prometheus/query_log_test.go @@ -193,7 +193,7 @@ func (p *queryLogTest) String() string { } name = name + ", " + p.host + ":" + strconv.Itoa(p.port) if p.enabledAtStart { - name = name + ", enabled at start" + name += ", enabled at start" } if p.prefix != "" { name = name + ", with prefix " + p.prefix diff --git a/cmd/promtool/backfill.go b/cmd/promtool/backfill.go index 3c23d2c037..39410881b2 100644 --- a/cmd/promtool/backfill.go +++ b/cmd/promtool/backfill.go @@ -101,7 +101,7 @@ func createBlocks(input []byte, mint, maxt, maxBlockDuration int64, maxSamplesIn nextSampleTs int64 = math.MaxInt64 ) - for t := mint; t <= maxt; t = t + blockDuration { + for t := mint; t <= maxt; t += blockDuration { tsUpper := t + blockDuration if nextSampleTs != math.MaxInt64 && nextSampleTs >= tsUpper { // The next sample is not in this timerange, we can avoid parsing diff --git a/cmd/promtool/rules.go b/cmd/promtool/rules.go index aedc7bcb9d..e430fe1898 100644 --- a/cmd/promtool/rules.go +++ b/cmd/promtool/rules.go @@ -100,7 +100,7 @@ func (importer *ruleImporter) importRule(ctx context.Context, ruleExpr, ruleName startInMs := start.Unix() * int64(time.Second/time.Millisecond) endInMs := end.Unix() * int64(time.Second/time.Millisecond) - for startOfBlock := blockDuration * (startInMs / blockDuration); startOfBlock <= endInMs; startOfBlock = startOfBlock + blockDuration { + for startOfBlock := blockDuration * (startInMs / blockDuration); startOfBlock <= endInMs; startOfBlock += blockDuration { endOfBlock := startOfBlock + blockDuration - 1 currStart := max(startOfBlock/int64(time.Second/time.Millisecond), start.Unix()) diff --git a/cmd/promtool/unittest.go b/cmd/promtool/unittest.go index b3e6f67f97..84dfd9ec79 100644 --- a/cmd/promtool/unittest.go +++ b/cmd/promtool/unittest.go @@ -130,7 +130,7 @@ func resolveAndGlobFilepaths(baseDir string, utf *unitTestFile) error { if err != nil { return err } - if len(m) <= 0 { + if len(m) == 0 { fmt.Fprintln(os.Stderr, " WARNING: no file match pattern", rf) } globbedFiles = append(globbedFiles, m...) diff --git a/discovery/kubernetes/endpointslice.go b/discovery/kubernetes/endpointslice.go index 135735154c..d5bff8a5f6 100644 --- a/discovery/kubernetes/endpointslice.go +++ b/discovery/kubernetes/endpointslice.go @@ -300,7 +300,7 @@ func (e *EndpointSlice) buildEndpointSlice(eps endpointSliceAdaptor) *targetgrou } if port.protocol() != nil { - target[endpointSlicePortProtocolLabel] = lv(string(*port.protocol())) + target[endpointSlicePortProtocolLabel] = lv(*port.protocol()) } if port.port() != nil { diff --git a/discovery/kubernetes/kubernetes.go b/discovery/kubernetes/kubernetes.go index 0f03e2cdb7..a44bd513ce 100644 --- a/discovery/kubernetes/kubernetes.go +++ b/discovery/kubernetes/kubernetes.go @@ -299,12 +299,13 @@ func New(l log.Logger, conf *SDConfig) (*Discovery, error) { err error ownNamespace string ) - if conf.KubeConfig != "" { + switch { + case conf.KubeConfig != "": kcfg, err = clientcmd.BuildConfigFromFlags("", conf.KubeConfig) if err != nil { return nil, err } - } else if conf.APIServer.URL == nil { + case conf.APIServer.URL == nil: // Use the Kubernetes provided pod service account // as described in https://kubernetes.io/docs/admin/service-accounts-admin/ kcfg, err = rest.InClusterConfig() @@ -324,7 +325,7 @@ func New(l log.Logger, conf *SDConfig) (*Discovery, error) { } level.Info(l).Log("msg", "Using pod service account via in-cluster config") - } else { + default: rt, err := config.NewRoundTripperFromConfig(conf.HTTPClientConfig, "kubernetes_sd") if err != nil { return nil, err diff --git a/discovery/legacymanager/registry.go b/discovery/legacymanager/registry.go index 687f093829..955705394d 100644 --- a/discovery/legacymanager/registry.go +++ b/discovery/legacymanager/registry.go @@ -254,7 +254,7 @@ func replaceYAMLTypeError(err error, oldTyp, newTyp reflect.Type) error { oldStr := oldTyp.String() newStr := newTyp.String() for i, s := range e.Errors { - e.Errors[i] = strings.Replace(s, oldStr, newStr, -1) + e.Errors[i] = strings.ReplaceAll(s, oldStr, newStr) } } return err diff --git a/discovery/linode/linode.go b/discovery/linode/linode.go index 0fd0a2c370..12b9575143 100644 --- a/discovery/linode/linode.go +++ b/discovery/linode/linode.go @@ -249,20 +249,20 @@ func (d *Discovery) refreshData(ctx context.Context) ([]*targetgroup.Group, erro if detailedIP.Address != ip.String() { continue } - - if detailedIP.Public && publicIPv4 == "" { + switch { + case detailedIP.Public && publicIPv4 == "": publicIPv4 = detailedIP.Address if detailedIP.RDNS != "" && detailedIP.RDNS != "null" { publicIPv4RDNS = detailedIP.RDNS } - } else if !detailedIP.Public && privateIPv4 == "" { + case !detailedIP.Public && privateIPv4 == "": privateIPv4 = detailedIP.Address if detailedIP.RDNS != "" && detailedIP.RDNS != "null" { privateIPv4RDNS = detailedIP.RDNS } - } else { + default: extraIPs = append(extraIPs, detailedIP.Address) } } diff --git a/discovery/marathon/marathon.go b/discovery/marathon/marathon.go index 079f93ad0b..c31daee1f4 100644 --- a/discovery/marathon/marathon.go +++ b/discovery/marathon/marathon.go @@ -400,19 +400,20 @@ func targetsForApp(app *app) []model.LabelSet { var labels []map[string]string var prefix string - if len(app.Container.PortMappings) != 0 { + switch { + case len(app.Container.PortMappings) != 0: // In Marathon 1.5.x the "container.docker.portMappings" object was moved // to "container.portMappings". ports, labels = extractPortMapping(app.Container.PortMappings, app.isContainerNet()) prefix = portMappingLabelPrefix - } else if len(app.Container.Docker.PortMappings) != 0 { + case len(app.Container.Docker.PortMappings) != 0: // Prior to Marathon 1.5 the port mappings could be found at the path // "container.docker.portMappings". ports, labels = extractPortMapping(app.Container.Docker.PortMappings, app.isContainerNet()) prefix = portMappingLabelPrefix - } else if len(app.PortDefinitions) != 0 { + case len(app.PortDefinitions) != 0: // PortDefinitions deprecates the "ports" array and can be used to specify // a list of ports with metadata in case a mapping is not required. ports = make([]uint32, len(app.PortDefinitions)) diff --git a/discovery/ovhcloud/dedicated_server_test.go b/discovery/ovhcloud/dedicated_server_test.go index 03a01005a9..e8ffa4a283 100644 --- a/discovery/ovhcloud/dedicated_server_test.go +++ b/discovery/ovhcloud/dedicated_server_test.go @@ -84,7 +84,7 @@ func MockDedicatedAPI(w http.ResponseWriter, r *http.Request) { return } w.Header().Set("Content-Type", "application/json") - if string(r.URL.Path) == "/dedicated/server" { + if r.URL.Path == "/dedicated/server" { dedicatedServersList, err := os.ReadFile("testdata/dedicated_server/dedicated_servers.json") if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) @@ -96,7 +96,7 @@ func MockDedicatedAPI(w http.ResponseWriter, r *http.Request) { return } } - if string(r.URL.Path) == "/dedicated/server/abcde" { + if r.URL.Path == "/dedicated/server/abcde" { dedicatedServer, err := os.ReadFile("testdata/dedicated_server/dedicated_servers_details.json") if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) @@ -108,7 +108,7 @@ func MockDedicatedAPI(w http.ResponseWriter, r *http.Request) { return } } - if string(r.URL.Path) == "/dedicated/server/abcde/ips" { + if r.URL.Path == "/dedicated/server/abcde/ips" { dedicatedServerIPs, err := os.ReadFile("testdata/dedicated_server/dedicated_servers_abcde_ips.json") if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) diff --git a/discovery/ovhcloud/vps_test.go b/discovery/ovhcloud/vps_test.go index 31b30fdfc6..b1177f215e 100644 --- a/discovery/ovhcloud/vps_test.go +++ b/discovery/ovhcloud/vps_test.go @@ -91,7 +91,7 @@ func MockVpsAPI(w http.ResponseWriter, r *http.Request) { return } w.Header().Set("Content-Type", "application/json") - if string(r.URL.Path) == "/vps" { + if r.URL.Path == "/vps" { dedicatedServersList, err := os.ReadFile("testdata/vps/vps.json") if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) @@ -103,7 +103,7 @@ func MockVpsAPI(w http.ResponseWriter, r *http.Request) { return } } - if string(r.URL.Path) == "/vps/abc" { + if r.URL.Path == "/vps/abc" { dedicatedServer, err := os.ReadFile("testdata/vps/vps_details.json") if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) @@ -115,7 +115,7 @@ func MockVpsAPI(w http.ResponseWriter, r *http.Request) { return } } - if string(r.URL.Path) == "/vps/abc/ips" { + if r.URL.Path == "/vps/abc/ips" { dedicatedServerIPs, err := os.ReadFile("testdata/vps/vps_abc_ips.json") if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) diff --git a/discovery/registry.go b/discovery/registry.go index 8274628c23..13168a07a7 100644 --- a/discovery/registry.go +++ b/discovery/registry.go @@ -253,7 +253,7 @@ func replaceYAMLTypeError(err error, oldTyp, newTyp reflect.Type) error { oldStr := oldTyp.String() newStr := newTyp.String() for i, s := range e.Errors { - e.Errors[i] = strings.Replace(s, oldStr, newStr, -1) + e.Errors[i] = strings.ReplaceAll(s, oldStr, newStr) } } return err diff --git a/discovery/vultr/vultr.go b/discovery/vultr/vultr.go index 2f489e7d45..42881d3c19 100644 --- a/discovery/vultr/vultr.go +++ b/discovery/vultr/vultr.go @@ -202,10 +202,8 @@ func (d *Discovery) listInstances(ctx context.Context) ([]govultr.Instance, erro if meta.Links.Next == "" { break - } else { - listOptions.Cursor = meta.Links.Next - continue } + listOptions.Cursor = meta.Links.Next } return instances, nil diff --git a/documentation/examples/remote_storage/remote_storage_adapter/influxdb/client.go b/documentation/examples/remote_storage/remote_storage_adapter/influxdb/client.go index fffbc9c2ae..e84ed9e129 100644 --- a/documentation/examples/remote_storage/remote_storage_adapter/influxdb/client.go +++ b/documentation/examples/remote_storage/remote_storage_adapter/influxdb/client.go @@ -184,11 +184,11 @@ func (c *Client) buildCommand(q *prompb.Query) (string, error) { } func escapeSingleQuotes(str string) string { - return strings.Replace(str, `'`, `\'`, -1) + return strings.ReplaceAll(str, `'`, `\'`) } func escapeSlashes(str string) string { - return strings.Replace(str, `/`, `\/`, -1) + return strings.ReplaceAll(str, `/`, `\/`) } func mergeResult(labelsToSeries map[string]*prompb.TimeSeries, results []influx.Result) error { @@ -290,13 +290,14 @@ func mergeSamples(a, b []prompb.Sample) []prompb.Sample { result := make([]prompb.Sample, 0, len(a)+len(b)) i, j := 0, 0 for i < len(a) && j < len(b) { - if a[i].Timestamp < b[j].Timestamp { + switch { + case a[i].Timestamp < b[j].Timestamp: result = append(result, a[i]) i++ - } else if a[i].Timestamp > b[j].Timestamp { + case a[i].Timestamp > b[j].Timestamp: result = append(result, b[j]) j++ - } else { + default: result = append(result, a[i]) i++ j++ diff --git a/model/labels/labels.go b/model/labels/labels.go index b7398d17f9..93524ddcfc 100644 --- a/model/labels/labels.go +++ b/model/labels/labels.go @@ -169,11 +169,12 @@ func (ls Labels) HashForLabels(b []byte, names ...string) (uint64, []byte) { b = b[:0] i, j := 0, 0 for i < len(ls) && j < len(names) { - if names[j] < ls[i].Name { + switch { + case names[j] < ls[i].Name: j++ - } else if ls[i].Name < names[j] { + case ls[i].Name < names[j]: i++ - } else { + default: b = append(b, ls[i].Name...) b = append(b, seps[0]) b = append(b, ls[i].Value...) @@ -213,11 +214,12 @@ func (ls Labels) BytesWithLabels(buf []byte, names ...string) []byte { b.WriteByte(labelSep) i, j := 0, 0 for i < len(ls) && j < len(names) { - if names[j] < ls[i].Name { + switch { + case names[j] < ls[i].Name: j++ - } else if ls[i].Name < names[j] { + case ls[i].Name < names[j]: i++ - } else { + default: if b.Len() > 1 { b.WriteByte(seps[0]) } diff --git a/model/textparse/promparse_test.go b/model/textparse/promparse_test.go index e0ecf62f5d..280f39b4f1 100644 --- a/model/textparse/promparse_test.go +++ b/model/textparse/promparse_test.go @@ -512,7 +512,7 @@ func BenchmarkGzip(b *testing.B) { k := b.N / promtestdataSampleCount b.ReportAllocs() - b.SetBytes(int64(n) / promtestdataSampleCount) + b.SetBytes(n / promtestdataSampleCount) b.ResetTimer() total := 0 diff --git a/promql/bench_test.go b/promql/bench_test.go index 88025d9325..d197da8881 100644 --- a/promql/bench_test.go +++ b/promql/bench_test.go @@ -194,9 +194,9 @@ func rangeQueryCases() []benchCase { if !strings.Contains(c.expr, "X") { tmp = append(tmp, c) } else { - tmp = append(tmp, benchCase{expr: strings.Replace(c.expr, "X", "one", -1), steps: c.steps}) - tmp = append(tmp, benchCase{expr: strings.Replace(c.expr, "X", "ten", -1), steps: c.steps}) - tmp = append(tmp, benchCase{expr: strings.Replace(c.expr, "X", "hundred", -1), steps: c.steps}) + tmp = append(tmp, benchCase{expr: strings.ReplaceAll(c.expr, "X", "one"), steps: c.steps}) + tmp = append(tmp, benchCase{expr: strings.ReplaceAll(c.expr, "X", "ten"), steps: c.steps}) + tmp = append(tmp, benchCase{expr: strings.ReplaceAll(c.expr, "X", "hundred"), steps: c.steps}) } } cases = tmp diff --git a/promql/engine.go b/promql/engine.go index b49be244f1..4dfa6b1192 100644 --- a/promql/engine.go +++ b/promql/engine.go @@ -783,7 +783,6 @@ func (ng *Engine) findMinMaxTime(s *parser.EvalStmt) (int64, int64) { maxTimestamp = end } evalRange = 0 - case *parser.MatrixSelector: evalRange = n.Range } @@ -816,20 +815,20 @@ func (ng *Engine) getTimeRangesForSelector(s *parser.EvalStmt, n *parser.VectorS } else { offsetMilliseconds := durationMilliseconds(subqOffset) start = start - offsetMilliseconds - durationMilliseconds(subqRange) - end = end - offsetMilliseconds + end -= offsetMilliseconds } if evalRange == 0 { - start = start - durationMilliseconds(s.LookbackDelta) + start -= durationMilliseconds(s.LookbackDelta) } else { // For all matrix queries we want to ensure that we have (end-start) + range selected // this way we have `range` data before the start time - start = start - durationMilliseconds(evalRange) + start -= durationMilliseconds(evalRange) } offsetMilliseconds := durationMilliseconds(n.OriginalOffset) - start = start - offsetMilliseconds - end = end - offsetMilliseconds + start -= offsetMilliseconds + end -= offsetMilliseconds return start, end } @@ -1745,7 +1744,7 @@ func (ev *evaluator) eval(expr parser.Expr) (parser.Value, storage.Warnings) { res, ws := newEv.eval(e.Expr) ev.currentSamples = newEv.currentSamples ev.samplesStats.UpdatePeakFromSubquery(newEv.samplesStats) - for ts, step := ev.startTimestamp, -1; ts <= ev.endTimestamp; ts = ts + ev.interval { + for ts, step := ev.startTimestamp, -1; ts <= ev.endTimestamp; ts += ev.interval { step++ ev.samplesStats.IncrementSamplesAtStep(step, newEv.samplesStats.TotalSamples) } @@ -1767,7 +1766,7 @@ func (ev *evaluator) eval(expr parser.Expr) (parser.Value, storage.Warnings) { if len(mat[i].Floats)+len(mat[i].Histograms) != 1 { panic(fmt.Errorf("unexpected number of samples")) } - for ts := ev.startTimestamp + ev.interval; ts <= ev.endTimestamp; ts = ts + ev.interval { + for ts := ev.startTimestamp + ev.interval; ts <= ev.endTimestamp; ts += ev.interval { if len(mat[i].Floats) > 0 { mat[i].Floats = append(mat[i].Floats, FPoint{ T: ts, @@ -2514,14 +2513,15 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without if !ok { var m labels.Labels enh.resetBuilder(metric) - if without { + switch { + case without: enh.lb.Del(grouping...) enh.lb.Del(labels.MetricName) m = enh.lb.Labels() - } else if len(grouping) > 0 { + case len(grouping) > 0: enh.lb.Keep(grouping...) m = enh.lb.Labels() - } else { + default: m = labels.EmptyLabels() } newAgg := &groupedAggregation{ @@ -2689,7 +2689,7 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without aggr.floatValue = float64(aggr.groupCount) case parser.STDVAR: - aggr.floatValue = aggr.floatValue / float64(aggr.groupCount) + aggr.floatValue /= float64(aggr.groupCount) case parser.STDDEV: aggr.floatValue = math.Sqrt(aggr.floatValue / float64(aggr.groupCount)) diff --git a/promql/engine_test.go b/promql/engine_test.go index b64e32ba46..056fd23555 100644 --- a/promql/engine_test.go +++ b/promql/engine_test.go @@ -3267,7 +3267,7 @@ func TestNativeHistogram_HistogramCountAndSum(t *testing.T) { require.Len(t, vector, 1) require.Nil(t, vector[0].H) if floatHisto { - require.Equal(t, float64(h.ToFloat().Count), vector[0].F) + require.Equal(t, h.ToFloat().Count, vector[0].F) } else { require.Equal(t, float64(h.Count), vector[0].F) } diff --git a/promql/functions.go b/promql/functions.go index fd99703df2..0e7a601e3d 100644 --- a/promql/functions.go +++ b/promql/functions.go @@ -877,10 +877,10 @@ func linearRegression(samples []FPoint, interceptTime int64) (slope, intercept f } return 0, initY } - sumX = sumX + cX - sumY = sumY + cY - sumXY = sumXY + cXY - sumX2 = sumX2 + cX2 + sumX += cX + sumY += cY + sumXY += cXY + sumX2 += cX2 covXY := sumXY - sumX*sumY/n varX := sumX2 - sumX*sumX/n diff --git a/promql/functions_test.go b/promql/functions_test.go index e552424b3d..8181481c0e 100644 --- a/promql/functions_test.go +++ b/promql/functions_test.go @@ -51,7 +51,7 @@ func TestDeriv(t *testing.T) { // https://github.com/prometheus/prometheus/issues/7180 for i = 0; i < 15; i++ { jitter := 12 * i % 2 - a.Append(0, metric, int64(start+interval*i+jitter), 1) + a.Append(0, metric, start+interval*i+jitter, 1) } require.NoError(t, a.Commit()) diff --git a/promql/parser/ast.go b/promql/parser/ast.go index 190af2d590..86f1394998 100644 --- a/promql/parser/ast.go +++ b/promql/parser/ast.go @@ -349,7 +349,7 @@ func (f inspector) Visit(node Node, path []Node) (Visitor, error) { // for all the non-nil children of node, recursively. func Inspect(node Node, f inspector) { //nolint: errcheck - Walk(inspector(f), node, nil) + Walk(f, node, nil) } // Children returns a list of all child nodes of a syntax tree node. @@ -368,13 +368,14 @@ func Children(node Node) []Node { case *AggregateExpr: // While this does not look nice, it should avoid unnecessary allocations // caused by slice resizing - if n.Expr == nil && n.Param == nil { + switch { + case n.Expr == nil && n.Param == nil: return nil - } else if n.Expr == nil { + case n.Expr == nil: return []Node{n.Param} - } else if n.Param == nil { + case n.Param == nil: return []Node{n.Expr} - } else { + default: return []Node{n.Expr, n.Param} } case *BinaryExpr: diff --git a/promql/parser/parse.go b/promql/parser/parse.go index 6c37ce6fc6..fa28097b28 100644 --- a/promql/parser/parse.go +++ b/promql/parser/parse.go @@ -523,15 +523,13 @@ func (p *parser) checkAST(node Node) (typ ValueType) { p.addParseErrf(n.PositionRange(), "vector matching only allowed between instant vectors") } n.VectorMatching = nil - } else { // Both operands are Vectors. - if n.Op.IsSetOperator() { - if n.VectorMatching.Card == CardOneToMany || n.VectorMatching.Card == CardManyToOne { - p.addParseErrf(n.PositionRange(), "no grouping allowed for %q operation", n.Op) - } - if n.VectorMatching.Card != CardManyToMany { - p.addParseErrf(n.PositionRange(), "set operations must always be many-to-many") - } + } else if n.Op.IsSetOperator() { + if n.VectorMatching.Card == CardOneToMany || n.VectorMatching.Card == CardManyToOne { + p.addParseErrf(n.PositionRange(), "no grouping allowed for %q operation", n.Op) + } + if n.VectorMatching.Card != CardManyToMany { + p.addParseErrf(n.PositionRange(), "set operations must always be many-to-many") } } diff --git a/promql/parser/parse_test.go b/promql/parser/parse_test.go index df66d9381a..7e6870ddbb 100644 --- a/promql/parser/parse_test.go +++ b/promql/parser/parse_test.go @@ -3592,7 +3592,7 @@ func TestNaNExpression(t *testing.T) { nl, ok := expr.(*NumberLiteral) require.True(t, ok, "expected number literal but got %T", expr) - require.True(t, math.IsNaN(float64(nl.Val)), "expected 'NaN' in number literal but got %v", nl.Val) + require.True(t, math.IsNaN(nl.Val), "expected 'NaN' in number literal but got %v", nl.Val) } var testSeries = []struct { diff --git a/promql/parser/printer.go b/promql/parser/printer.go index 1f15eeef33..4fff193e16 100644 --- a/promql/parser/printer.go +++ b/promql/parser/printer.go @@ -130,11 +130,12 @@ func (node *MatrixSelector) String() string { offset = fmt.Sprintf(" offset -%s", model.Duration(-vecSelector.OriginalOffset)) } at := "" - if vecSelector.Timestamp != nil { + switch { + case vecSelector.Timestamp != nil: at = fmt.Sprintf(" @ %.3f", float64(*vecSelector.Timestamp)/1000.0) - } else if vecSelector.StartOrEnd == START { + case vecSelector.StartOrEnd == START: at = " @ start()" - } else if vecSelector.StartOrEnd == END { + case vecSelector.StartOrEnd == END: at = " @ end()" } @@ -168,11 +169,12 @@ func (node *SubqueryExpr) getSubqueryTimeSuffix() string { offset = fmt.Sprintf(" offset -%s", model.Duration(-node.OriginalOffset)) } at := "" - if node.Timestamp != nil { + switch { + case node.Timestamp != nil: at = fmt.Sprintf(" @ %.3f", float64(*node.Timestamp)/1000.0) - } else if node.StartOrEnd == START { + case node.StartOrEnd == START: at = " @ start()" - } else if node.StartOrEnd == END { + case node.StartOrEnd == END: at = " @ end()" } return fmt.Sprintf("[%s:%s]%s%s", model.Duration(node.Range), step, at, offset) @@ -213,11 +215,12 @@ func (node *VectorSelector) String() string { offset = fmt.Sprintf(" offset -%s", model.Duration(-node.OriginalOffset)) } at := "" - if node.Timestamp != nil { + switch { + case node.Timestamp != nil: at = fmt.Sprintf(" @ %.3f", float64(*node.Timestamp)/1000.0) - } else if node.StartOrEnd == START { + case node.StartOrEnd == START: at = " @ start()" - } else if node.StartOrEnd == END { + case node.StartOrEnd == END: at = " @ end()" } diff --git a/rules/manager.go b/rules/manager.go index 07d50be1b9..82bbfd3947 100644 --- a/rules/manager.go +++ b/rules/manager.go @@ -866,12 +866,13 @@ func (g *Group) RestoreForState(ts time.Time) { timeSpentPending := downAt.Sub(restoredActiveAt) timeRemainingPending := alertHoldDuration - timeSpentPending - if timeRemainingPending <= 0 { + switch { + case timeRemainingPending <= 0: // It means that alert was firing when prometheus went down. // In the next Eval, the state of this alert will be set back to // firing again if it's still firing in that Eval. // Nothing to be done in this case. - } else if timeRemainingPending < g.opts.ForGracePeriod { + case timeRemainingPending < g.opts.ForGracePeriod: // (new) restoredActiveAt = (ts + m.opts.ForGracePeriod) - alertHoldDuration // /* new firing time */ /* moving back by hold duration */ // @@ -884,7 +885,7 @@ func (g *Group) RestoreForState(ts time.Time) { // = (ts + m.opts.ForGracePeriod) - ts // = m.opts.ForGracePeriod restoredActiveAt = ts.Add(g.opts.ForGracePeriod).Add(-alertHoldDuration) - } else { + default: // By shifting ActiveAt to the future (ActiveAt + some_duration), // the total pending time from the original ActiveAt // would be `alertHoldDuration + some_duration`. diff --git a/rules/manager_test.go b/rules/manager_test.go index 440e06c9af..85a74ac52c 100644 --- a/rules/manager_test.go +++ b/rules/manager_test.go @@ -481,17 +481,18 @@ func TestForStateRestore(t *testing.T) { }) // Checking if we have restored it correctly. - if tst.noRestore { + switch { + case tst.noRestore: require.Equal(t, tst.num, len(got)) for _, e := range got { require.Equal(t, e.ActiveAt, restoreTime) } - } else if tst.gracePeriod { + case tst.gracePeriod: require.Equal(t, tst.num, len(got)) for _, e := range got { require.Equal(t, opts.ForGracePeriod, e.ActiveAt.Add(alertForDuration).Sub(restoreTime)) } - } else { + default: exp := tst.alerts require.Equal(t, len(exp), len(got)) sortAlerts(exp) diff --git a/scrape/scrape.go b/scrape/scrape.go index f38527ff30..15c8867932 100644 --- a/scrape/scrape.go +++ b/scrape/scrape.go @@ -640,7 +640,7 @@ func verifyLabelLimits(lset labels.Labels, limits *labelLimits) error { met := lset.Get(labels.MetricName) if limits.labelLimit > 0 { nbLabels := lset.Len() - if nbLabels > int(limits.labelLimit) { + if nbLabels > limits.labelLimit { return fmt.Errorf("label_limit exceeded (metric: %.50s, number of labels: %d, limit: %d)", met, nbLabels, limits.labelLimit) } } @@ -652,14 +652,14 @@ func verifyLabelLimits(lset labels.Labels, limits *labelLimits) error { return lset.Validate(func(l labels.Label) error { if limits.labelNameLengthLimit > 0 { nameLength := len(l.Name) - if nameLength > int(limits.labelNameLengthLimit) { + if nameLength > limits.labelNameLengthLimit { return fmt.Errorf("label_name_length_limit exceeded (metric: %.50s, label name: %.50s, length: %d, limit: %d)", met, l.Name, nameLength, limits.labelNameLengthLimit) } } if limits.labelValueLengthLimit > 0 { valueLength := len(l.Value) - if valueLength > int(limits.labelValueLengthLimit) { + if valueLength > limits.labelValueLengthLimit { return fmt.Errorf("label_value_length_limit exceeded (metric: %.50s, label name: %.50s, value: %.50q, length: %d, limit: %d)", met, l.Name, l.Value, valueLength, limits.labelValueLengthLimit) } } diff --git a/scrape/scrape_test.go b/scrape/scrape_test.go index dcb3b48c10..07b9c3c875 100644 --- a/scrape/scrape_test.go +++ b/scrape/scrape_test.go @@ -322,7 +322,7 @@ func TestScrapePoolReloadPreserveRelabeledIntervalTimeout(t *testing.T) { ScrapeTimeout: model.Duration(2 * time.Second), } newLoop := func(opts scrapeLoopOptions) loop { - l := &testLoop{interval: time.Duration(opts.interval), timeout: time.Duration(opts.timeout)} + l := &testLoop{interval: opts.interval, timeout: opts.timeout} l.startFunc = func(interval, timeout time.Duration, errc chan<- error) { require.Equal(t, 5*time.Second, interval, "Unexpected scrape interval") require.Equal(t, 3*time.Second, timeout, "Unexpected scrape timeout") @@ -546,7 +546,7 @@ func TestScrapePoolRaces(t *testing.T) { require.Equal(t, expectedDropped, len(dropped), "Invalid number of dropped targets") for i := 0; i < 20; i++ { - time.Sleep(time.Duration(10 * time.Millisecond)) + time.Sleep(10 * time.Millisecond) sp.reload(newConfig()) } sp.stop() @@ -1199,14 +1199,14 @@ func TestScrapeLoopRunCreatesStaleMarkersOnParseFailure(t *testing.T) { // Succeed once, several failures, then stop. scraper.scrapeFunc = func(ctx context.Context, w io.Writer) error { numScrapes++ - - if numScrapes == 1 { + switch { + case numScrapes == 1: w.Write([]byte("metric_a 42\n")) return nil - } else if numScrapes == 2 { + case numScrapes == 2: w.Write([]byte("7&-\n")) return nil - } else if numScrapes == 3 { + case numScrapes == 3: cancel() } return errors.New("scrape failed") @@ -1282,14 +1282,14 @@ func TestScrapeLoopCache(t *testing.T) { } numScrapes++ - - if numScrapes == 1 { + switch { + case numScrapes == 1: w.Write([]byte("metric_a 42\nmetric_b 43\n")) return nil - } else if numScrapes == 3 { + case numScrapes == 3: w.Write([]byte("metric_a 44\n")) return nil - } else if numScrapes == 4 { + case numScrapes == 4: cancel() } return fmt.Errorf("scrape failed") @@ -2280,11 +2280,12 @@ func TestTargetScrapeScrapeCancel(t *testing.T) { go func() { _, err := ts.scrape(ctx, io.Discard) - if err == nil { + switch { + case err == nil: errc <- errors.New("Expected error but got nil") - } else if ctx.Err() != context.Canceled { + case ctx.Err() != context.Canceled: errc <- errors.Errorf("Expected context cancellation error but got: %s", ctx.Err()) - } else { + default: close(errc) } }() diff --git a/scrape/target.go b/scrape/target.go index f250910c10..6c47031186 100644 --- a/scrape/target.go +++ b/scrape/target.go @@ -413,9 +413,9 @@ func PopulateLabels(lb *labels.Builder, cfg *config.ScrapeConfig, noDefaultPort // Addresses reaching this point are already wrapped in [] if necessary. switch scheme { case "http", "": - addr = addr + ":80" + addr += ":80" case "https": - addr = addr + ":443" + addr += ":443" default: return labels.EmptyLabels(), labels.EmptyLabels(), errors.Errorf("invalid scheme: %q", cfg.Scheme) } diff --git a/storage/merge.go b/storage/merge.go index 8db1f7ae83..23a92df1e2 100644 --- a/storage/merge.go +++ b/storage/merge.go @@ -197,13 +197,14 @@ func mergeStrings(a, b []string) []string { res := make([]string, 0, maxl*10/9) for len(a) > 0 && len(b) > 0 { - if a[0] == b[0] { + switch { + case a[0] == b[0]: res = append(res, a[0]) a, b = a[1:], b[1:] - } else if a[0] < b[0] { + case a[0] < b[0]: res = append(res, a[0]) a = a[1:] - } else { + default: res = append(res, b[0]) b = b[1:] } diff --git a/storage/remote/codec.go b/storage/remote/codec.go index bfbd08d24b..02c84a3e6c 100644 --- a/storage/remote/codec.go +++ b/storage/remote/codec.go @@ -291,13 +291,14 @@ func MergeLabels(primary, secondary []prompb.Label) []prompb.Label { result := make([]prompb.Label, 0, len(primary)+len(secondary)) i, j := 0, 0 for i < len(primary) && j < len(secondary) { - if primary[i].Name < secondary[j].Name { + switch { + case primary[i].Name < secondary[j].Name: result = append(result, primary[i]) i++ - } else if primary[i].Name > secondary[j].Name { + case primary[i].Name > secondary[j].Name: result = append(result, secondary[j]) j++ - } else { + default: result = append(result, primary[i]) i++ j++ @@ -428,8 +429,8 @@ func (c *concreteSeriesIterator) Seek(t int64) chunkenc.ValueType { c.histogramsCur += sort.Search(len(c.series.histograms)-c.histogramsCur, func(n int) bool { return c.series.histograms[n+c.histogramsCur].Timestamp >= t }) - - if c.floatsCur < len(c.series.floats) && c.histogramsCur < len(c.series.histograms) { + switch { + case c.floatsCur < len(c.series.floats) && c.histogramsCur < len(c.series.histograms): // If float samples and histogram samples have overlapping timestamps prefer the float samples. if c.series.floats[c.floatsCur].Timestamp <= c.series.histograms[c.histogramsCur].Timestamp { c.curValType = chunkenc.ValFloat @@ -445,12 +446,11 @@ func (c *concreteSeriesIterator) Seek(t int64) chunkenc.ValueType { c.floatsCur-- } } - } else if c.floatsCur < len(c.series.floats) { + case c.floatsCur < len(c.series.floats): c.curValType = chunkenc.ValFloat - } else if c.histogramsCur < len(c.series.histograms) { + case c.histogramsCur < len(c.series.histograms): c.curValType = getHistogramValType(&c.series.histograms[c.histogramsCur]) } - return c.curValType } @@ -514,26 +514,25 @@ func (c *concreteSeriesIterator) Next() chunkenc.ValueType { peekHistTS = c.series.histograms[c.histogramsCur+1].Timestamp } c.curValType = chunkenc.ValNone - - if peekFloatTS < peekHistTS { + switch { + case peekFloatTS < peekHistTS: c.floatsCur++ c.curValType = chunkenc.ValFloat - } else if peekHistTS < peekFloatTS { + case peekHistTS < peekFloatTS: c.histogramsCur++ c.curValType = chunkenc.ValHistogram - } else if peekFloatTS == noTS && peekHistTS == noTS { + case peekFloatTS == noTS && peekHistTS == noTS: // This only happens when the iterator is exhausted; we set the cursors off the end to prevent // Seek() from returning anything afterwards. c.floatsCur = len(c.series.floats) c.histogramsCur = len(c.series.histograms) - } else { + default: // Prefer float samples to histogram samples if there's a conflict. We advance the cursor for histograms // anyway otherwise the histogram sample will get selected on the next call to Next(). c.floatsCur++ c.histogramsCur++ c.curValType = chunkenc.ValFloat } - return c.curValType } diff --git a/storage/remote/queue_manager.go b/storage/remote/queue_manager.go index 62bd17a66d..0fe6d0698b 100644 --- a/storage/remote/queue_manager.go +++ b/storage/remote/queue_manager.go @@ -609,7 +609,7 @@ outer: t.metrics.enqueueRetriesTotal.Inc() time.Sleep(time.Duration(backoff)) - backoff = backoff * 2 + backoff *= 2 // It is reasonable to use t.cfg.MaxBackoff here, as if we have hit // the full backoff we are likely waiting for external resources. if backoff > t.cfg.MaxBackoff { @@ -660,7 +660,7 @@ outer: t.metrics.enqueueRetriesTotal.Inc() time.Sleep(time.Duration(backoff)) - backoff = backoff * 2 + backoff *= 2 if backoff > t.cfg.MaxBackoff { backoff = t.cfg.MaxBackoff } @@ -707,7 +707,7 @@ outer: t.metrics.enqueueRetriesTotal.Inc() time.Sleep(time.Duration(backoff)) - backoff = backoff * 2 + backoff *= 2 if backoff > t.cfg.MaxBackoff { backoff = t.cfg.MaxBackoff } @@ -754,7 +754,7 @@ outer: t.metrics.enqueueRetriesTotal.Inc() time.Sleep(time.Duration(backoff)) - backoff = backoff * 2 + backoff *= 2 if backoff > t.cfg.MaxBackoff { backoff = t.cfg.MaxBackoff } diff --git a/template/template.go b/template/template.go index d61a880a22..01f6ec9a8a 100644 --- a/template/template.go +++ b/template/template.go @@ -421,7 +421,7 @@ func (te Expander) ExpandHTML(templateFiles []string) (result string, resultErr } } }() - + //nolint:unconvert // Before Go 1.19 conversion from text_template to html_template is mandatory tmpl := html_template.New(te.name).Funcs(html_template.FuncMap(te.funcMap)) tmpl.Option(te.options...) tmpl.Funcs(html_template.FuncMap{ diff --git a/tsdb/agent/db_test.go b/tsdb/agent/db_test.go index f654fdb90e..e284e1b77e 100644 --- a/tsdb/agent/db_test.go +++ b/tsdb/agent/db_test.go @@ -739,8 +739,7 @@ func TestStorage_DuplicateExemplarsIgnored(t *testing.T) { var dec record.Decoder for r.Next() { rec := r.Record() - switch dec.Type(rec) { - case record.Exemplars: + if dec.Type(rec) == record.Exemplars { var exemplars []record.RefExemplar exemplars, err = dec.Exemplars(rec, exemplars) require.NoError(t, err) diff --git a/tsdb/block_test.go b/tsdb/block_test.go index 49a997fc56..e9dc1a9d00 100644 --- a/tsdb/block_test.go +++ b/tsdb/block_test.go @@ -630,7 +630,7 @@ func genHistogramSeries(totalSeries, labelCount int, mint, maxt, step int64, flo {Offset: 0, Length: 2}, {Offset: 1, Length: 2}, }, - PositiveBuckets: []int64{int64(ts + 1), 1, -1, 0}, + PositiveBuckets: []int64{ts + 1, 1, -1, 0}, } if ts != mint { // By setting the counter reset hint to "no counter @@ -669,7 +669,7 @@ func genHistogramAndFloatSeries(totalSeries, labelCount int, mint, maxt, step in {Offset: 0, Length: 2}, {Offset: 1, Length: 2}, }, - PositiveBuckets: []int64{int64(ts + 1), 1, -1, 0}, + PositiveBuckets: []int64{ts + 1, 1, -1, 0}, } if count > 1 && count%5 != 1 { // Same rationale for this as above in diff --git a/tsdb/chunkenc/bstream.go b/tsdb/chunkenc/bstream.go index 60531023ba..7b17f4686b 100644 --- a/tsdb/chunkenc/bstream.go +++ b/tsdb/chunkenc/bstream.go @@ -182,7 +182,7 @@ func (b *bstreamReader) readBits(nbits uint8) (uint64, error) { } bitmask = (uint64(1) << nbits) - 1 - v = v | ((b.buffer >> (b.valid - nbits)) & bitmask) + v |= ((b.buffer >> (b.valid - nbits)) & bitmask) b.valid -= nbits return v, nil @@ -242,13 +242,13 @@ func (b *bstreamReader) loadNextBuffer(nbits uint8) bool { if b.streamOffset+nbytes == len(b.stream) { // There can be concurrent writes happening on the very last byte // of the stream, so use the copy we took at initialization time. - buffer = buffer | uint64(b.last) + buffer |= uint64(b.last) // Read up to the byte before skip = 1 } for i := 0; i < nbytes-skip; i++ { - buffer = buffer | (uint64(b.stream[b.streamOffset+i]) << uint(8*(nbytes-i-1))) + buffer |= (uint64(b.stream[b.streamOffset+i]) << uint(8*(nbytes-i-1))) } b.buffer = buffer diff --git a/tsdb/chunkenc/float_histogram.go b/tsdb/chunkenc/float_histogram.go index b462c6d9fd..6dd08a31c6 100644 --- a/tsdb/chunkenc/float_histogram.go +++ b/tsdb/chunkenc/float_histogram.go @@ -785,7 +785,7 @@ func (it *floatHistogramIterator) Next() ValueType { it.err = err return ValNone } - it.tDelta = it.tDelta + tDod + it.tDelta += tDod it.t += it.tDelta if ok := it.readXor(&it.cnt.value, &it.cnt.leading, &it.cnt.trailing); !ok { diff --git a/tsdb/chunkenc/histogram.go b/tsdb/chunkenc/histogram.go index 7b6a9cacb3..866fae36fd 100644 --- a/tsdb/chunkenc/histogram.go +++ b/tsdb/chunkenc/histogram.go @@ -875,7 +875,7 @@ func (it *histogramIterator) Next() ValueType { it.err = err return ValNone } - it.tDelta = it.tDelta + tDod + it.tDelta += tDod it.t += it.tDelta cntDod, err := readVarbitInt(&it.br) @@ -883,7 +883,7 @@ func (it *histogramIterator) Next() ValueType { it.err = err return ValNone } - it.cntDelta = it.cntDelta + cntDod + it.cntDelta += cntDod it.cnt = uint64(int64(it.cnt) + it.cntDelta) zcntDod, err := readVarbitInt(&it.br) @@ -891,7 +891,7 @@ func (it *histogramIterator) Next() ValueType { it.err = err return ValNone } - it.zCntDelta = it.zCntDelta + zcntDod + it.zCntDelta += zcntDod it.zCnt = uint64(int64(it.zCnt) + it.zCntDelta) ok := it.readSum() diff --git a/tsdb/chunkenc/varbit.go b/tsdb/chunkenc/varbit.go index b3b14cf417..449f9fbac2 100644 --- a/tsdb/chunkenc/varbit.go +++ b/tsdb/chunkenc/varbit.go @@ -122,7 +122,7 @@ func readVarbitInt(b *bstreamReader) (int64, error) { } if bits > (1 << (sz - 1)) { // Or something. - bits = bits - (1 << sz) + bits -= (1 << sz) } val = int64(bits) } diff --git a/tsdb/chunkenc/xor.go b/tsdb/chunkenc/xor.go index 2fa2f613cb..8ca04502a7 100644 --- a/tsdb/chunkenc/xor.go +++ b/tsdb/chunkenc/xor.go @@ -163,15 +163,15 @@ func (a *xorAppender) AppendFloatHistogram(t int64, h *histogram.FloatHistogram) func (a *xorAppender) Append(t int64, v float64) { var tDelta uint64 num := binary.BigEndian.Uint16(a.b.bytes()) - - if num == 0 { + switch { + case num == 0: buf := make([]byte, binary.MaxVarintLen64) for _, b := range buf[:binary.PutVarint(buf, t)] { a.b.writeByte(b) } a.b.writeBits(math.Float64bits(v), 64) - } else if num == 1 { + case num == 1: tDelta = uint64(t - a.t) buf := make([]byte, binary.MaxVarintLen64) @@ -181,7 +181,7 @@ func (a *xorAppender) Append(t int64, v float64) { a.writeVDelta(v) - } else { + default: tDelta = uint64(t - a.t) dod := int64(tDelta - a.tDelta) @@ -321,7 +321,7 @@ func (it *xorIterator) Next() ValueType { return ValNone } it.tDelta = tDelta - it.t = it.t + int64(it.tDelta) + it.t += int64(it.tDelta) return it.readValue() } @@ -384,7 +384,7 @@ func (it *xorIterator) Next() ValueType { } it.tDelta = uint64(int64(it.tDelta) + dod) - it.t = it.t + int64(it.tDelta) + it.t += int64(it.tDelta) return it.readValue() } diff --git a/tsdb/chunks/head_chunks_test.go b/tsdb/chunks/head_chunks_test.go index ac89ae3e59..20a4c2064b 100644 --- a/tsdb/chunks/head_chunks_test.go +++ b/tsdb/chunks/head_chunks_test.go @@ -503,10 +503,10 @@ func createChunkDiskMapper(t *testing.T, dir string) *ChunkDiskMapper { func randomChunk(t *testing.T) chunkenc.Chunk { chunk := chunkenc.NewXORChunk() - len := rand.Int() % 120 + length := rand.Int() % 120 app, err := chunk.Appender() require.NoError(t, err) - for i := 0; i < len; i++ { + for i := 0; i < length; i++ { app.Append(rand.Int63(), rand.Float64()) } return chunk diff --git a/tsdb/compact.go b/tsdb/compact.go index b2d4123754..7c061b0bbd 100644 --- a/tsdb/compact.go +++ b/tsdb/compact.go @@ -44,7 +44,7 @@ func ExponentialBlockRanges(minSize int64, steps, stepSize int) []int64 { curRange := minSize for i := 0; i < steps; i++ { ranges = append(ranges, curRange) - curRange = curRange * int64(stepSize) + curRange *= int64(stepSize) } return ranges diff --git a/tsdb/compact_test.go b/tsdb/compact_test.go index 6a7e6ea68b..5a9eadedad 100644 --- a/tsdb/compact_test.go +++ b/tsdb/compact_test.go @@ -1452,12 +1452,6 @@ func TestSparseHistogramSpaceSavings(t *testing.T) { {100, 15, 3, 5}, {100, 50, 3, 3}, {100, 100, 3, 2}, - //{1000, 15, 1, 0}, - //{1000, 50, 1, 0}, - //{1000, 100, 1, 0}, - //{1000, 15, 3, 5}, - //{1000, 50, 3, 3}, - //{1000, 100, 3, 2}, } type testSummary struct { diff --git a/tsdb/db.go b/tsdb/db.go index 659251c3ca..a10f07b1e1 100644 --- a/tsdb/db.go +++ b/tsdb/db.go @@ -260,7 +260,7 @@ func newDBMetrics(db *DB, r prometheus.Registerer) *dbMetrics { Help: "Size of symbol table in memory for loaded blocks", }, func() float64 { db.mtx.RLock() - blocks := db.blocks[:] + blocks := db.blocks db.mtx.RUnlock() symTblSize := uint64(0) for _, b := range blocks { @@ -1186,7 +1186,7 @@ func (db *DB) compactOOO(dest string, oooHead *OOOCompactionHead) (_ []ulid.ULID } }() - for t := blockSize * (oooHeadMint / blockSize); t <= oooHeadMaxt; t = t + blockSize { + for t := blockSize * (oooHeadMint / blockSize); t <= oooHeadMaxt; t += blockSize { mint, maxt := t, t+blockSize // Block intervals are half-open: [b.MinTime, b.MaxTime). Block intervals are always +1 than the total samples it includes. uid, err := db.compactor.Write(dest, oooHead.CloneForTimeRange(mint, maxt-1), mint, maxt, nil) @@ -1508,7 +1508,7 @@ func BeyondSizeRetention(db *DB, blocks []*Block) (deletable map[ulid.ULID]struc blocksSize := db.Head().Size() for i, block := range blocks { blocksSize += block.Size() - if blocksSize > int64(db.opts.MaxBytes) { + if blocksSize > db.opts.MaxBytes { // Add this and all following blocks for deletion. for _, b := range blocks[i:] { deletable[b.meta.ULID] = struct{}{} diff --git a/tsdb/db_test.go b/tsdb/db_test.go index c54fccf6f1..7e1f89a95a 100644 --- a/tsdb/db_test.go +++ b/tsdb/db_test.go @@ -1076,7 +1076,7 @@ func TestWALSegmentSizeOptions(t *testing.T) { dbDir := db.Dir() require.NoError(t, db.Close()) - testFunc(dbDir, int(opts.WALSegmentSize)) + testFunc(dbDir, opts.WALSegmentSize) }) } } @@ -2996,7 +2996,7 @@ func TestCompactHead(t *testing.T) { series = seriesSet.At().Iterator(series) for series.Next() == chunkenc.ValFloat { time, val := series.At() - actSamples = append(actSamples, sample{int64(time), val, nil, nil}) + actSamples = append(actSamples, sample{time, val, nil, nil}) } require.NoError(t, series.Err()) } diff --git a/tsdb/exemplar.go b/tsdb/exemplar.go index 5ba3567e41..ad3b2ef39b 100644 --- a/tsdb/exemplar.go +++ b/tsdb/exemplar.go @@ -115,17 +115,17 @@ func NewExemplarMetrics(reg prometheus.Registerer) *ExemplarMetrics { // 1GB of extra memory, accounting for the fact that this is heap allocated space. // If len <= 0, then the exemplar storage is essentially a noop storage but can later be // resized to store exemplars. -func NewCircularExemplarStorage(len int64, m *ExemplarMetrics) (ExemplarStorage, error) { - if len < 0 { - len = 0 +func NewCircularExemplarStorage(length int64, m *ExemplarMetrics) (ExemplarStorage, error) { + if length < 0 { + length = 0 } c := &CircularExemplarStorage{ - exemplars: make([]*circularBufferEntry, len), - index: make(map[string]*indexEntry, len/estimatedExemplarsPerSeries), + exemplars: make([]*circularBufferEntry, length), + index: make(map[string]*indexEntry, length/estimatedExemplarsPerSeries), metrics: m, } - c.metrics.maxExemplars.Set(float64(len)) + c.metrics.maxExemplars.Set(float64(length)) return c, nil } @@ -151,7 +151,7 @@ func (ce *CircularExemplarStorage) Querier(_ context.Context) (storage.ExemplarQ func (ce *CircularExemplarStorage) Select(start, end int64, matchers ...[]*labels.Matcher) ([]exemplar.QueryResult, error) { ret := make([]exemplar.QueryResult, 0) - if len(ce.exemplars) <= 0 { + if len(ce.exemplars) == 0 { return ret, nil } @@ -219,7 +219,7 @@ func (ce *CircularExemplarStorage) ValidateExemplar(l labels.Labels, e exemplar. // Not thread safe. The append parameters tells us whether this is an external validation, or internal // as a result of an AddExemplar call, in which case we should update any relevant metrics. func (ce *CircularExemplarStorage) validateExemplar(key []byte, e exemplar.Exemplar, append bool) error { - if len(ce.exemplars) <= 0 { + if len(ce.exemplars) == 0 { return storage.ErrExemplarsDisabled } @@ -334,7 +334,7 @@ func (ce *CircularExemplarStorage) migrate(entry *circularBufferEntry) { } func (ce *CircularExemplarStorage) AddExemplar(l labels.Labels, e exemplar.Exemplar) error { - if len(ce.exemplars) <= 0 { + if len(ce.exemplars) == 0 { return storage.ErrExemplarsDisabled } diff --git a/tsdb/head.go b/tsdb/head.go index 4696884f21..ca953b1755 100644 --- a/tsdb/head.go +++ b/tsdb/head.go @@ -1453,7 +1453,7 @@ func (h *Head) Delete(mint, maxt int64, ms ...*labels.Matcher) error { } } for _, s := range stones { - h.tombstones.AddInterval(storage.SeriesRef(s.Ref), s.Intervals[0]) + h.tombstones.AddInterval(s.Ref, s.Intervals[0]) } return nil diff --git a/tsdb/head_test.go b/tsdb/head_test.go index e80c197b23..9326ddbe1c 100644 --- a/tsdb/head_test.go +++ b/tsdb/head_test.go @@ -3005,7 +3005,7 @@ func TestHistogramInWALAndMmapChunk(t *testing.T) { hists = tsdbutil.GenerateTestHistograms(numHistograms) } for _, h := range hists { - h.Count = h.Count * 2 + h.Count *= 2 h.NegativeSpans = h.PositiveSpans h.NegativeBuckets = h.PositiveBuckets _, err := app.AppendHistogram(0, s1, ts, h, nil) @@ -3028,7 +3028,7 @@ func TestHistogramInWALAndMmapChunk(t *testing.T) { hists = tsdbutil.GenerateTestFloatHistograms(numHistograms) } for _, h := range hists { - h.Count = h.Count * 2 + h.Count *= 2 h.NegativeSpans = h.PositiveSpans h.NegativeBuckets = h.PositiveBuckets _, err := app.AppendHistogram(0, s1, ts, nil, h) @@ -3069,26 +3069,26 @@ func TestHistogramInWALAndMmapChunk(t *testing.T) { } for _, h := range hists { ts++ - h.Count = h.Count * 2 + h.Count *= 2 h.NegativeSpans = h.PositiveSpans h.NegativeBuckets = h.PositiveBuckets - _, err := app.AppendHistogram(0, s2, int64(ts), h, nil) + _, err := app.AppendHistogram(0, s2, ts, h, nil) require.NoError(t, err) eh := h.Copy() if !gauge && ts > 30 && (ts-10)%20 == 1 { // Need "unknown" hint after float sample. eh.CounterResetHint = histogram.UnknownCounterReset } - exp[k2] = append(exp[k2], sample{t: int64(ts), h: eh}) + exp[k2] = append(exp[k2], sample{t: ts, h: eh}) if ts%20 == 0 { require.NoError(t, app.Commit()) app = head.Appender(context.Background()) // Add some float. for i := 0; i < 10; i++ { ts++ - _, err := app.Append(0, s2, int64(ts), float64(ts)) + _, err := app.Append(0, s2, ts, float64(ts)) require.NoError(t, err) - exp[k2] = append(exp[k2], sample{t: int64(ts), f: float64(ts)}) + exp[k2] = append(exp[k2], sample{t: ts, f: float64(ts)}) } require.NoError(t, app.Commit()) app = head.Appender(context.Background()) @@ -3106,26 +3106,26 @@ func TestHistogramInWALAndMmapChunk(t *testing.T) { } for _, h := range hists { ts++ - h.Count = h.Count * 2 + h.Count *= 2 h.NegativeSpans = h.PositiveSpans h.NegativeBuckets = h.PositiveBuckets - _, err := app.AppendHistogram(0, s2, int64(ts), nil, h) + _, err := app.AppendHistogram(0, s2, ts, nil, h) require.NoError(t, err) eh := h.Copy() if !gauge && ts > 30 && (ts-10)%20 == 1 { // Need "unknown" hint after float sample. eh.CounterResetHint = histogram.UnknownCounterReset } - exp[k2] = append(exp[k2], sample{t: int64(ts), fh: eh}) + exp[k2] = append(exp[k2], sample{t: ts, fh: eh}) if ts%20 == 0 { require.NoError(t, app.Commit()) app = head.Appender(context.Background()) // Add some float. for i := 0; i < 10; i++ { ts++ - _, err := app.Append(0, s2, int64(ts), float64(ts)) + _, err := app.Append(0, s2, ts, float64(ts)) require.NoError(t, err) - exp[k2] = append(exp[k2], sample{t: int64(ts), f: float64(ts)}) + exp[k2] = append(exp[k2], sample{t: ts, f: float64(ts)}) } require.NoError(t, app.Commit()) app = head.Appender(context.Background()) @@ -4495,11 +4495,12 @@ func TestHistogramValidation(t *testing.T) { } err = ValidateFloatHistogram(tc.h.ToFloat()) - if tc.errMsgFloat != "" { + switch { + case tc.errMsgFloat != "": require.ErrorContains(t, err, tc.errMsgFloat) - } else if tc.errMsg != "" { + case tc.errMsg != "": require.ErrorContains(t, err, tc.errMsg) - } else { + default: require.NoError(t, err) } }) diff --git a/tsdb/head_wal.go b/tsdb/head_wal.go index 6a8a30d5ad..b3537d060a 100644 --- a/tsdb/head_wal.go +++ b/tsdb/head_wal.go @@ -299,7 +299,7 @@ Outer: unknownRefs.Inc() continue } - h.tombstones.AddInterval(storage.SeriesRef(s.Ref), itv) + h.tombstones.AddInterval(s.Ref, itv) } } tstonesPool.Put(v) @@ -382,7 +382,7 @@ Outer: floatHistogramsPool.Put(v) case []record.RefMetadata: for _, m := range v { - s := h.series.getByID(chunks.HeadSeriesRef(m.Ref)) + s := h.series.getByID(m.Ref) if s == nil { unknownMetadataRefs.Inc() continue diff --git a/tsdb/index/index.go b/tsdb/index/index.go index 9f584ee821..50a701d3a6 100644 --- a/tsdb/index/index.go +++ b/tsdb/index/index.go @@ -536,7 +536,7 @@ func (w *Writer) finishSymbols() error { // Write out the length and symbol count. w.buf1.Reset() w.buf1.PutBE32int(int(symbolTableSize)) - w.buf1.PutBE32int(int(w.numSymbols)) + w.buf1.PutBE32int(w.numSymbols) if err := w.writeAt(w.buf1.Get(), w.toc.Symbols); err != nil { return err } diff --git a/tsdb/index/postings.go b/tsdb/index/postings.go index b55d70df03..c57c085ec4 100644 --- a/tsdb/index/postings.go +++ b/tsdb/index/postings.go @@ -561,10 +561,8 @@ func newMergedPostings(p []Postings) (m *mergedPostings, nonEmpty bool) { // NOTE: mergedPostings struct requires the user to issue an initial Next. if it.Next() { ph = append(ph, it) - } else { - if it.Err() != nil { - return &mergedPostings{err: it.Err()}, true - } + } else if it.Err() != nil { + return &mergedPostings{err: it.Err()}, true } } @@ -699,15 +697,16 @@ func (rp *removedPostings) Next() bool { } fcur, rcur := rp.full.At(), rp.remove.At() - if fcur < rcur { + switch { + case fcur < rcur: rp.cur = fcur rp.fok = rp.full.Next() return true - } else if rcur < fcur { + case rcur < fcur: // Forward the remove postings to the right position. rp.rok = rp.remove.Seek(fcur) - } else { + default: // Skip the current posting. rp.fok = rp.full.Next() } diff --git a/tsdb/index/postingsstats.go b/tsdb/index/postingsstats.go index 5e5880720a..6b29bddabf 100644 --- a/tsdb/index/postingsstats.go +++ b/tsdb/index/postingsstats.go @@ -31,10 +31,10 @@ type maxHeap struct { Items []Stat } -func (m *maxHeap) init(len int) { - m.maxLength = len +func (m *maxHeap) init(length int) { + m.maxLength = length m.minValue = math.MaxUint64 - m.Items = make([]Stat, 0, len) + m.Items = make([]Stat, 0, length) } func (m *maxHeap) push(item Stat) { diff --git a/tsdb/isolation.go b/tsdb/isolation.go index 74d63c6af0..401e5885a0 100644 --- a/tsdb/isolation.go +++ b/tsdb/isolation.go @@ -254,7 +254,7 @@ func (txr *txRing) add(appendID uint64) { if txr.txIDCount == len(txr.txIDs) { // Ring buffer is full, expand by doubling. newRing := make([]uint64, txr.txIDCount*2) - idx := copy(newRing[:], txr.txIDs[txr.txIDFirst:]) + idx := copy(newRing, txr.txIDs[txr.txIDFirst:]) copy(newRing[idx:], txr.txIDs[:txr.txIDFirst]) txr.txIDs = newRing txr.txIDFirst = 0 diff --git a/tsdb/querier.go b/tsdb/querier.go index 4b3144c71d..8806c7e735 100644 --- a/tsdb/querier.go +++ b/tsdb/querier.go @@ -239,18 +239,20 @@ func PostingsForMatchers(ix IndexReader, ms ...*labels.Matcher) (index.Postings, } for _, m := range ms { - if m.Name == "" && m.Value == "" { // Special-case for AllPostings, used in tests at least. + switch { + case m.Name == "" && m.Value == "": // Special-case for AllPostings, used in tests at least. k, v := index.AllPostingsKey() allPostings, err := ix.Postings(k, v) if err != nil { return nil, err } its = append(its, allPostings) - } else if labelMustBeSet[m.Name] { + case labelMustBeSet[m.Name]: // If this matcher must be non-empty, we can be smarter. matchesEmpty := m.Matches("") isNot := m.Type == labels.MatchNotEqual || m.Type == labels.MatchNotRegexp - if isNot && matchesEmpty { // l!="foo" + switch { + case isNot && matchesEmpty: // l!="foo" // If the label can't be empty and is a Not and the inner matcher // doesn't match empty, then subtract it out at the end. inverse, err := m.Inverse() @@ -263,7 +265,7 @@ func PostingsForMatchers(ix IndexReader, ms ...*labels.Matcher) (index.Postings, return nil, err } notIts = append(notIts, it) - } else if isNot && !matchesEmpty { // l!="" + case isNot && !matchesEmpty: // l!="" // If the label can't be empty and is a Not, but the inner matcher can // be empty we need to use inversePostingsForMatcher. inverse, err := m.Inverse() @@ -279,7 +281,7 @@ func PostingsForMatchers(ix IndexReader, ms ...*labels.Matcher) (index.Postings, return index.EmptyPostings(), nil } its = append(its, it) - } else { // l="a" + default: // l="a" // Non-Not matcher, use normal postingsForMatcher. it, err := postingsForMatcher(ix, m) if err != nil { @@ -290,7 +292,7 @@ func PostingsForMatchers(ix IndexReader, ms ...*labels.Matcher) (index.Postings, } its = append(its, it) } - } else { // l="" + default: // l="" // If the matchers for a labelname selects an empty value, it selects all // the series which don't have the label name set too. See: // https://github.com/prometheus/prometheus/issues/3575 and @@ -965,24 +967,24 @@ func (m *mergedStringIter) Next() bool { if (!m.aok && !m.bok) || (m.Err() != nil) { return false } - - if !m.aok { + switch { + case !m.aok: m.cur = m.b.At() m.bok = m.b.Next() m.err = m.b.Err() - } else if !m.bok { + case !m.bok: m.cur = m.a.At() m.aok = m.a.Next() m.err = m.a.Err() - } else if m.b.At() > m.a.At() { + case m.b.At() > m.a.At(): m.cur = m.a.At() m.aok = m.a.Next() m.err = m.a.Err() - } else if m.a.At() > m.b.At() { + case m.a.At() > m.b.At(): m.cur = m.b.At() m.bok = m.b.Next() m.err = m.b.Err() - } else { // Equal. + default: // Equal. m.cur = m.b.At() m.aok = m.a.Next() m.err = m.a.Err() diff --git a/tsdb/querier_test.go b/tsdb/querier_test.go index fa3dd2418f..8f52fff28a 100644 --- a/tsdb/querier_test.go +++ b/tsdb/querier_test.go @@ -113,7 +113,7 @@ func createIdxChkReaders(t *testing.T, tc []seriesSamples) (IndexReader, ChunkRe var chunkRef chunks.ChunkRef for i, s := range tc { - i = i + 1 // 0 is not a valid posting. + i++ // 0 is not a valid posting. metas := make([]chunks.Meta, 0, len(s.chunks)) for _, chk := range s.chunks { if chk[0].t < blockMint { @@ -2012,7 +2012,7 @@ func BenchmarkQueries(b *testing.B) { for x := 0; x <= 10; x++ { block, err := OpenBlock(nil, createBlock(b, dir, series), nil) require.NoError(b, err) - q, err := NewBlockQuerier(block, 1, int64(nSamples)) + q, err := NewBlockQuerier(block, 1, nSamples) require.NoError(b, err) qs = append(qs, q) } diff --git a/tsdb/wal.go b/tsdb/wal.go index e0bc1ec699..a9af76d150 100644 --- a/tsdb/wal.go +++ b/tsdb/wal.go @@ -90,7 +90,7 @@ func newWalMetrics(r prometheus.Registerer) *walMetrics { // WAL is a write ahead log that can log new series labels and samples. // It must be completely read before new entries are logged. // -// DEPRECATED: use wlog pkg combined with the record codex instead. +// Deprecated: use wlog pkg combined with the record codex instead. type WAL interface { Reader() WALReader LogSeries([]record.RefSeries) error @@ -147,7 +147,7 @@ func newCRC32() hash.Hash32 { // SegmentWAL is a write ahead log for series data. // -// DEPRECATED: use wlog pkg combined with the record coders instead. +// Deprecated: use wlog pkg combined with the record coders instead. type SegmentWAL struct { mtx sync.Mutex metrics *walMetrics diff --git a/tsdb/wlog/wlog_test.go b/tsdb/wlog/wlog_test.go index ed8a9df2e3..7f9133a76d 100644 --- a/tsdb/wlog/wlog_test.go +++ b/tsdb/wlog/wlog_test.go @@ -428,10 +428,10 @@ func TestLogPartialWrite(t *testing.T) { faultyRecord: pageSize / (recordHeaderSize + len(record)), }, // TODO the current implementation suffers this: - //"partial write when logging a record overlapping two pages": { + // "partial write when logging a record overlapping two pages": { // numRecords: (pageSize / (recordHeaderSize + len(record))) + 10, // faultyRecord: pageSize/(recordHeaderSize+len(record)) + 1, - //}, + // }, } for testName, testData := range tests { diff --git a/util/runtime/limits_default.go b/util/runtime/limits_default.go index c3e0b4701a..1588a93c79 100644 --- a/util/runtime/limits_default.go +++ b/util/runtime/limits_default.go @@ -39,7 +39,7 @@ func getLimits(resource int, unit string) string { if err != nil { panic("syscall.Getrlimit failed: " + err.Error()) } - return fmt.Sprintf("(soft=%s, hard=%s)", limitToString(uint64(rlimit.Cur), unit), limitToString(uint64(rlimit.Max), unit)) + return fmt.Sprintf("(soft=%s, hard=%s)", limitToString(rlimit.Cur, unit), limitToString(rlimit.Max, unit)) } // FdLimits returns the soft and hard limits for file descriptors. diff --git a/util/runtime/statfs_default.go b/util/runtime/statfs_default.go index f850f2cd6d..2e31d93fca 100644 --- a/util/runtime/statfs_default.go +++ b/util/runtime/statfs_default.go @@ -72,11 +72,13 @@ func Statfs(path string) string { var fs syscall.Statfs_t err := syscall.Statfs(path, &fs) + //nolint:unconvert // This ensure Type format on all Platforms + localType := int64(fs.Type) if err != nil { - return strconv.FormatInt(int64(fs.Type), 16) + return strconv.FormatInt(localType, 16) } - if fsType, ok := fsTypes[int64(fs.Type)]; ok { + if fsType, ok := fsTypes[localType]; ok { return fsType } - return strconv.FormatInt(int64(fs.Type), 16) + return strconv.FormatInt(localType, 16) } diff --git a/web/api/v1/api.go b/web/api/v1/api.go index aeea87ca70..2e0016fd25 100644 --- a/web/api/v1/api.go +++ b/web/api/v1/api.go @@ -243,7 +243,7 @@ func NewAPI( remoteReadConcurrencyLimit int, remoteReadMaxBytesInFrame int, isAgent bool, - CORSOrigin *regexp.Regexp, + corsOrigin *regexp.Regexp, runtimeInfo func() (RuntimeInfo, error), buildInfo *PrometheusVersion, gatherer prometheus.Gatherer, @@ -269,7 +269,7 @@ func NewAPI( enableAdmin: enableAdmin, rulesRetriever: rr, logger: logger, - CORSOrigin: CORSOrigin, + CORSOrigin: corsOrigin, runtimeInfo: runtimeInfo, buildInfo: buildInfo, gatherer: gatherer, diff --git a/web/api/v1/api_test.go b/web/api/v1/api_test.go index efce042214..27cbab1b33 100644 --- a/web/api/v1/api_test.go +++ b/web/api/v1/api_test.go @@ -2791,7 +2791,7 @@ func TestRespondSuccess(t *testing.T) { } var res response - if err = json.Unmarshal([]byte(body), &res); err != nil { + if err = json.Unmarshal(body, &res); err != nil { t.Fatalf("Error unmarshaling JSON body: %s", err) } @@ -2827,7 +2827,7 @@ func TestRespondError(t *testing.T) { } var res response - if err = json.Unmarshal([]byte(body), &res); err != nil { + if err = json.Unmarshal(body, &res); err != nil { t.Fatalf("Error unmarshaling JSON body: %s", err) } diff --git a/web/web.go b/web/web.go index 9d63094f69..f4f64163df 100644 --- a/web/web.go +++ b/web/web.go @@ -719,9 +719,9 @@ func (h *Handler) runtimeInfo() (api_v1.RuntimeInfo, error) { } if h.options.TSDBMaxBytes != 0 { if status.StorageRetention != "" { - status.StorageRetention = status.StorageRetention + " or " + status.StorageRetention += " or " } - status.StorageRetention = status.StorageRetention + h.options.TSDBMaxBytes.String() + status.StorageRetention += h.options.TSDBMaxBytes.String() } metrics, err := h.gatherer.Gather() From 052993414a60254d71bd0e19cf3ee648ca697728 Mon Sep 17 00:00:00 2001 From: Justin Lei Date: Wed, 12 Apr 2023 09:48:35 -0700 Subject: [PATCH 092/231] Add storage.tsdb.samples-per-chunk flag Signed-off-by: Justin Lei --- cmd/prometheus/main.go | 5 +++++ tsdb/db.go | 5 +++++ tsdb/head.go | 19 +++++++++++++------ tsdb/head_append.go | 7 ++----- tsdb/head_test.go | 12 ++++++------ 5 files changed, 31 insertions(+), 17 deletions(-) diff --git a/cmd/prometheus/main.go b/cmd/prometheus/main.go index f4f6af20df..cafe2f819f 100644 --- a/cmd/prometheus/main.go +++ b/cmd/prometheus/main.go @@ -336,6 +336,9 @@ func main() { serverOnlyFlag(a, "storage.tsdb.head-chunks-write-queue-size", "Size of the queue through which head chunks are written to the disk to be m-mapped, 0 disables the queue completely. Experimental."). Default("0").IntVar(&cfg.tsdb.HeadChunksWriteQueueSize) + serverOnlyFlag(a, "storage.tsdb.samples-per-chunk", "Target number of samples per chunk."). + Default("120").Hidden().IntVar(&cfg.tsdb.SamplesPerChunk) + agentOnlyFlag(a, "storage.agent.path", "Base path for metrics storage."). Default("data-agent/").StringVar(&cfg.agentStoragePath) @@ -1542,6 +1545,7 @@ type tsdbOptions struct { NoLockfile bool WALCompression bool HeadChunksWriteQueueSize int + SamplesPerChunk int StripeSize int MinBlockDuration model.Duration MaxBlockDuration model.Duration @@ -1562,6 +1566,7 @@ func (opts tsdbOptions) ToTSDBOptions() tsdb.Options { AllowOverlappingCompaction: true, WALCompression: opts.WALCompression, HeadChunksWriteQueueSize: opts.HeadChunksWriteQueueSize, + SamplesPerChunk: opts.SamplesPerChunk, StripeSize: opts.StripeSize, MinBlockDuration: int64(time.Duration(opts.MinBlockDuration) / time.Millisecond), MaxBlockDuration: int64(time.Duration(opts.MaxBlockDuration) / time.Millisecond), diff --git a/tsdb/db.go b/tsdb/db.go index 659251c3ca..e0e9c69f05 100644 --- a/tsdb/db.go +++ b/tsdb/db.go @@ -78,6 +78,7 @@ func DefaultOptions() *Options { NoLockfile: false, AllowOverlappingCompaction: true, WALCompression: false, + SamplesPerChunk: DefaultSamplesPerChunk, StripeSize: DefaultStripeSize, HeadChunksWriteBufferSize: chunks.DefaultWriteBufferSize, IsolationDisabled: defaultIsolationDisabled, @@ -149,6 +150,9 @@ type Options struct { // HeadChunksWriteQueueSize configures the size of the chunk write queue used in the head chunks mapper. HeadChunksWriteQueueSize int + // SamplesPerChunk configures the target number of samples per chunk. + SamplesPerChunk int + // SeriesLifecycleCallback specifies a list of callbacks that will be called during a lifecycle of a series. // It is always a no-op in Prometheus and mainly meant for external users who import TSDB. SeriesLifecycleCallback SeriesLifecycleCallback @@ -778,6 +782,7 @@ func open(dir string, l log.Logger, r prometheus.Registerer, opts *Options, rngs headOpts.ChunkPool = db.chunkPool headOpts.ChunkWriteBufferSize = opts.HeadChunksWriteBufferSize headOpts.ChunkWriteQueueSize = opts.HeadChunksWriteQueueSize + headOpts.SamplesPerChunk = opts.SamplesPerChunk headOpts.StripeSize = opts.StripeSize headOpts.SeriesCallback = opts.SeriesLifecycleCallback headOpts.EnableExemplarStorage = opts.EnableExemplarStorage diff --git a/tsdb/head.go b/tsdb/head.go index af8175cd03..b4df1b2d05 100644 --- a/tsdb/head.go +++ b/tsdb/head.go @@ -150,6 +150,8 @@ type HeadOptions struct { ChunkWriteBufferSize int ChunkWriteQueueSize int + SamplesPerChunk int + // StripeSize sets the number of entries in the hash map, it must be a power of 2. // A larger StripeSize will allocate more memory up-front, but will increase performance when handling a large number of series. // A smaller StripeSize reduces the memory allocated, but can decrease performance with large number of series. @@ -169,6 +171,8 @@ type HeadOptions struct { const ( // DefaultOutOfOrderCapMax is the default maximum size of an in-memory out-of-order chunk. DefaultOutOfOrderCapMax int64 = 32 + // DefaultSamplesPerChunk provides a default target number of samples per chunk. + DefaultSamplesPerChunk = 120 ) func DefaultHeadOptions() *HeadOptions { @@ -178,6 +182,7 @@ func DefaultHeadOptions() *HeadOptions { ChunkPool: chunkenc.NewPool(), ChunkWriteBufferSize: chunks.DefaultWriteBufferSize, ChunkWriteQueueSize: chunks.DefaultWriteQueueSize, + SamplesPerChunk: DefaultSamplesPerChunk, StripeSize: DefaultStripeSize, SeriesCallback: &noopSeriesLifecycleCallback{}, IsolationDisabled: defaultIsolationDisabled, @@ -1607,7 +1612,7 @@ func (h *Head) getOrCreate(hash uint64, lset labels.Labels) (*memSeries, bool, e func (h *Head) getOrCreateWithID(id chunks.HeadSeriesRef, hash uint64, lset labels.Labels) (*memSeries, bool, error) { s, created, err := h.series.getOrSet(hash, lset, func() *memSeries { - return newMemSeries(lset, id, h.opts.IsolationDisabled) + return newMemSeries(lset, id, h.opts.IsolationDisabled, h.opts.SamplesPerChunk) }) if err != nil { return nil, false, err @@ -1915,7 +1920,8 @@ type memSeries struct { mmMaxTime int64 // Max time of any mmapped chunk, only used during WAL replay. - nextAt int64 // Timestamp at which to cut the next chunk. + samplesPerChunk int // Target number of samples per chunk. + nextAt int64 // Timestamp at which to cut the next chunk. // We keep the last value here (in addition to appending it to the chunk) so we can check for duplicates. lastValue float64 @@ -1943,11 +1949,12 @@ type memSeriesOOOFields struct { firstOOOChunkID chunks.HeadChunkID // HeadOOOChunkID for oooMmappedChunks[0]. } -func newMemSeries(lset labels.Labels, id chunks.HeadSeriesRef, isolationDisabled bool) *memSeries { +func newMemSeries(lset labels.Labels, id chunks.HeadSeriesRef, isolationDisabled bool, samplesPerChunk int) *memSeries { s := &memSeries{ - lset: lset, - ref: id, - nextAt: math.MinInt64, + lset: lset, + ref: id, + nextAt: math.MinInt64, + samplesPerChunk: samplesPerChunk, } if !isolationDisabled { s.txs = newTxRing(4) diff --git a/tsdb/head_append.go b/tsdb/head_append.go index 46180051ee..eb5b219ea8 100644 --- a/tsdb/head_append.go +++ b/tsdb/head_append.go @@ -1324,9 +1324,6 @@ func (s *memSeries) appendFloatHistogram(t int64, fh *histogram.FloatHistogram, func (s *memSeries) appendPreprocessor( t int64, e chunkenc.Encoding, chunkDiskMapper *chunks.ChunkDiskMapper, chunkRange int64, ) (c *memChunk, sampleInOrder, chunkCreated bool) { - // The basis for this number can be found here: https://github.com/prometheus/prometheus/pull/12055 - const samplesPerChunk = 220 - c = s.head() if c == nil { @@ -1363,7 +1360,7 @@ func (s *memSeries) appendPreprocessor( // for this chunk that will try to make samples equally distributed within // the remaining chunks in the current chunk range. // At latest it must happen at the timestamp set when the chunk was cut. - if numSamples == samplesPerChunk/4 { + if numSamples == s.samplesPerChunk/4 { s.nextAt = computeChunkEndTime(c.minTime, c.maxTime, s.nextAt) } // If numSamples > samplesPerChunk*2 then our previous prediction was invalid, @@ -1371,7 +1368,7 @@ func (s *memSeries) appendPreprocessor( // Since we assume that the rate is higher, we're being conservative and cutting at 2*samplesPerChunk // as we expect more chunks to come. // Note that next chunk will have its nextAt recalculated for the new rate. - if t >= s.nextAt || numSamples >= samplesPerChunk*2 { + if t >= s.nextAt || numSamples >= s.samplesPerChunk*2 { c = s.cutNewHeadChunk(t, e, chunkDiskMapper, chunkRange) chunkCreated = true } diff --git a/tsdb/head_test.go b/tsdb/head_test.go index 9a7220957a..df48e592d9 100644 --- a/tsdb/head_test.go +++ b/tsdb/head_test.go @@ -284,7 +284,7 @@ func BenchmarkLoadWAL(b *testing.B) { require.NoError(b, err) for k := 0; k < c.batches*c.seriesPerBatch; k++ { // Create one mmapped chunk per series, with one sample at the given time. - s := newMemSeries(labels.Labels{}, chunks.HeadSeriesRef(k)*101, defaultIsolationDisabled) + s := newMemSeries(labels.Labels{}, chunks.HeadSeriesRef(k)*101, defaultIsolationDisabled, DefaultSamplesPerChunk) s.append(c.mmappedChunkT, 42, 0, chunkDiskMapper, c.mmappedChunkT) s.mmapCurrentHeadChunk(chunkDiskMapper) } @@ -806,7 +806,7 @@ func TestMemSeries_truncateChunks(t *testing.T) { }, } - s := newMemSeries(labels.FromStrings("a", "b"), 1, defaultIsolationDisabled) + s := newMemSeries(labels.FromStrings("a", "b"), 1, defaultIsolationDisabled, DefaultSamplesPerChunk) for i := 0; i < 8000; i += 5 { ok, _ := s.append(int64(i), float64(i), 0, chunkDiskMapper, chunkRange) @@ -1337,7 +1337,7 @@ func TestMemSeries_append(t *testing.T) { }() const chunkRange = 500 - s := newMemSeries(labels.Labels{}, 1, defaultIsolationDisabled) + s := newMemSeries(labels.Labels{}, 1, defaultIsolationDisabled, DefaultSamplesPerChunk) // Add first two samples at the very end of a chunk range and the next two // on and after it. @@ -1391,7 +1391,7 @@ func TestMemSeries_appendHistogram(t *testing.T) { }() chunkRange := int64(1000) - s := newMemSeries(labels.Labels{}, 1, defaultIsolationDisabled) + s := newMemSeries(labels.Labels{}, 1, defaultIsolationDisabled, DefaultSamplesPerChunk) histograms := tsdbutil.GenerateTestHistograms(4) histogramWithOneMoreBucket := histograms[3].Copy() @@ -1447,7 +1447,7 @@ func TestMemSeries_append_atVariableRate(t *testing.T) { }) chunkRange := DefaultBlockDuration - s := newMemSeries(labels.Labels{}, 1, defaultIsolationDisabled) + s := newMemSeries(labels.Labels{}, 1, defaultIsolationDisabled, DefaultSamplesPerChunk) // At this slow rate, we will fill the chunk in two block durations. slowRate := (DefaultBlockDuration * 2) / samplesPerChunk @@ -2609,7 +2609,7 @@ func TestIteratorSeekIntoBuffer(t *testing.T) { }() const chunkRange = 500 - s := newMemSeries(labels.Labels{}, 1, defaultIsolationDisabled) + s := newMemSeries(labels.Labels{}, 1, defaultIsolationDisabled, DefaultSamplesPerChunk) for i := 0; i < 7; i++ { ok, _ := s.append(int64(i), float64(i), 0, chunkDiskMapper, chunkRange) From c3e6b8563109ab09ccea33283ae27bfe3121b77a Mon Sep 17 00:00:00 2001 From: Justin Lei Date: Thu, 13 Apr 2023 14:35:29 -0700 Subject: [PATCH 093/231] Reverse test changes Signed-off-by: Justin Lei --- storage/remote/read_handler_test.go | 135 ++++++++++++++-------------- tsdb/db_test.go | 12 +-- tsdb/head_test.go | 24 ++--- 3 files changed, 84 insertions(+), 87 deletions(-) diff --git a/storage/remote/read_handler_test.go b/storage/remote/read_handler_test.go index 0c186097af..261c28e215 100644 --- a/storage/remote/read_handler_test.go +++ b/storage/remote/read_handler_test.go @@ -202,18 +202,15 @@ func BenchmarkStreamReadEndpoint(b *testing.B) { } func TestStreamReadEndpoint(t *testing.T) { - // Note: samplesPerChunk is set to 220, but that isn't cleanly divisible by the chunkRange of 24 hrs and 1 min - // resolution used in this test so tsdb.computeChunkEndTime will put 240 samples in each chunk. - // - // First with 239 samples; we expect 1 frame with 1 full chunk. - // Second with 241 samples; we expect 1 frame with 2 chunks. - // Third with 481 samples; we expect 1 frame with 2 chunks, and 1 frame with 1 chunk for the same series due to bytes limit. + // First with 120 samples. We expect 1 frame with 1 chunk. + // Second with 121 samples, We expect 1 frame with 2 chunks. + // Third with 241 samples. We expect 1 frame with 2 chunks, and 1 frame with 1 chunk for the same series due to bytes limit. // Fourth with 120 histogram samples. We expect 1 frame with 1 chunk. suite, err := promql.NewTest(t, ` load 1m - test_metric1{foo="bar1",baz="qux"} 0+100x239 - test_metric1{foo="bar2",baz="qux"} 0+100x240 - test_metric1{foo="bar3",baz="qux"} 0+100x480 + test_metric1{foo="bar1",baz="qux"} 0+100x119 + test_metric1{foo="bar2",baz="qux"} 0+100x120 + test_metric1{foo="bar3",baz="qux"} 0+100x240 `) require.NoError(t, err) defer suite.Close() @@ -231,8 +228,8 @@ func TestStreamReadEndpoint(t *testing.T) { } }, 1e6, 1, - // Labelset has 57 bytes. Full chunk in test data has roughly 440 bytes. This allows us to have at max 2 chunks in this test. - 57+880, + // Labelset has 57 bytes. Full chunk in test data has roughly 240 bytes. This allows us to have at max 2 chunks in this test. + 57+480, ) // Encode the request. @@ -248,19 +245,19 @@ func TestStreamReadEndpoint(t *testing.T) { matcher4, err := labels.NewMatcher(labels.MatchEqual, "__name__", "test_histogram_metric1") require.NoError(t, err) - query1, err := ToQuery(0, 32460001, []*labels.Matcher{matcher1, matcher2}, &storage.SelectHints{ + query1, err := ToQuery(0, 14400001, []*labels.Matcher{matcher1, matcher2}, &storage.SelectHints{ Step: 1, Func: "avg", Start: 0, - End: 32460001, + End: 14400001, }) require.NoError(t, err) - query2, err := ToQuery(0, 32460001, []*labels.Matcher{matcher1, matcher3}, &storage.SelectHints{ + query2, err := ToQuery(0, 14400001, []*labels.Matcher{matcher1, matcher3}, &storage.SelectHints{ Step: 1, Func: "avg", Start: 0, - End: 32460001, + End: 14400001, }) require.NoError(t, err) @@ -319,8 +316,8 @@ func TestStreamReadEndpoint(t *testing.T) { Chunks: []prompb.Chunk{ { Type: prompb.Chunk_XOR, - MaxTimeMs: 14340000, - Data: []byte("\x00\xf0\x00\x00\x00\x00\x00\x00\x00\x00\x00\xe0\xd4\x03\xc2|\x05\x94\x00\xc1\xac}\xe9z2\xd0O\xed\xb4n[\aΔ\xa3md\xf9\xd0\xfd\x1aPm\nS\x9d\x0eQ\xad\x06P\xbd\xa8\xbfʁZ\x03(3\xa0R\x87\xda\x05(\x0f\xa0ҁ\xda=(\x13\xa02\x83Z\a(w\xa02\x81Z\x0f(\x13\xb5\x97\xf4P\x1b@\xa5\a\xf4\nP\x1bC\xa5\x02t\x1eP+@e\x1e\xf4\x0ePk@e\x02t:P;A\xa5\x01\xf4\nS\xfb@\xa5\x06t\x06P+C\xe5\x02\xf4\x06Pk@\xe5\x1et\nP\x1bA\xa5\x03\xf4:P\x1b@\xa5\x06t\x0eZJ\xff\\\x85ˈ\u05f8\x0f\xe5+F[\xc8\xe7E)\xed\x14\xa1\xf6\xe2}(v\x8d(N\x83)և(ރ(V\xdaW\xf2\x82t4\xa0m\x05(\xffAJ\x06\xd0҂t\xfc\xa0\xad\x03(oA\xca:\xd02\x82t4\xa0\xed\xb0\xbfҀ\xfa\x05(=\xa0R\x87:\x03(\x15\xa0\xf2\x81z\x03(\xf5\xa0r\x83:\x05(\r\xa1ҁ\xfa\r(\r\xa0R\x9f:\a(5\xa02\x81z\x1f(\x15\xa02\x83:\x05(\xfd\xa0R\x80\xfa\r(\x1d\xa1ҁ:\x03(5\xa0r\xd6g\xf4\xb9\x8b\x97!\xabq\x1f\xcaN\xe0GJ\x1bE)\xfd\x14\xa1\xb6\xf2}(N\x8f(V\x83)އ(փ(N\xdcW҃\xb44\xa0}\x05(\xfbAJ\x19\xd02\x82\xb4\xfc\xa0\xbd\x03(kA\xca9\xd0R\x81\xb44\xa0\xfd\xb4\xbfҀ\xda\x05(3\xa0r\x87Z\x03(\x17\xa0\xf2\x81Z\x03(\xf3\xa0R\x83\xda\x05(\x0f\xa1ҁ\xda\r(\x13\xa02\x9fZ\a(7\xa02\x81Z\x1f(\x13\xa0Ҁ\xda\x05(\xff\xa0R\x80\xda\r "), + MaxTimeMs: 7140000, + Data: []byte("\000x\000\000\000\000\000\000\000\000\000\340\324\003\302|\005\224\000\301\254}\351z2\320O\355\264n[\007\316\224\243md\371\320\375\032Pm\nS\235\016Q\255\006P\275\250\277\312\201Z\003(3\240R\207\332\005(\017\240\322\201\332=(\023\2402\203Z\007(w\2402\201Z\017(\023\265\227\364P\033@\245\007\364\nP\033C\245\002t\036P+@e\036\364\016Pk@e\002t:P;A\245\001\364\nS\373@\245\006t\006P+C\345\002\364\006Pk@\345\036t\nP\033A\245\003\364:P\033@\245\006t\016ZJ\377\\\205\313\210\327\270\017\345+F[\310\347E)\355\024\241\366\342}(v\215(N\203)\326\207(\336\203(V\332W\362\202t4\240m\005(\377AJ\006\320\322\202t\374\240\255\003(oA\312:\3202"), }, }, }, @@ -339,61 +336,61 @@ func TestStreamReadEndpoint(t *testing.T) { Chunks: []prompb.Chunk{ { Type: prompb.Chunk_XOR, - MaxTimeMs: 14340000, - Data: []byte("\x00\xf0\x00\x00\x00\x00\x00\x00\x00\x00\x00\xe0\xd4\x03\xc2|\x05\x94\x00\xc1\xac}\xe9z2\xd0O\xed\xb4n[\aΔ\xa3md\xf9\xd0\xfd\x1aPm\nS\x9d\x0eQ\xad\x06P\xbd\xa8\xbfʁZ\x03(3\xa0R\x87\xda\x05(\x0f\xa0ҁ\xda=(\x13\xa02\x83Z\a(w\xa02\x81Z\x0f(\x13\xb5\x97\xf4P\x1b@\xa5\a\xf4\nP\x1bC\xa5\x02t\x1eP+@e\x1e\xf4\x0ePk@e\x02t:P;A\xa5\x01\xf4\nS\xfb@\xa5\x06t\x06P+C\xe5\x02\xf4\x06Pk@\xe5\x1et\nP\x1bA\xa5\x03\xf4:P\x1b@\xa5\x06t\x0eZJ\xff\\\x85ˈ\u05f8\x0f\xe5+F[\xc8\xe7E)\xed\x14\xa1\xf6\xe2}(v\x8d(N\x83)և(ރ(V\xdaW\xf2\x82t4\xa0m\x05(\xffAJ\x06\xd0҂t\xfc\xa0\xad\x03(oA\xca:\xd02\x82t4\xa0\xed\xb0\xbfҀ\xfa\x05(=\xa0R\x87:\x03(\x15\xa0\xf2\x81z\x03(\xf5\xa0r\x83:\x05(\r\xa1ҁ\xfa\r(\r\xa0R\x9f:\a(5\xa02\x81z\x1f(\x15\xa02\x83:\x05(\xfd\xa0R\x80\xfa\r(\x1d\xa1ҁ:\x03(5\xa0r\xd6g\xf4\xb9\x8b\x97!\xabq\x1f\xcaN\xe0GJ\x1bE)\xfd\x14\xa1\xb6\xf2}(N\x8f(V\x83)އ(փ(N\xdcW҃\xb44\xa0}\x05(\xfbAJ\x19\xd02\x82\xb4\xfc\xa0\xbd\x03(kA\xca9\xd0R\x81\xb44\xa0\xfd\xb4\xbfҀ\xda\x05(3\xa0r\x87Z\x03(\x17\xa0\xf2\x81Z\x03(\xf3\xa0R\x83\xda\x05(\x0f\xa1ҁ\xda\r(\x13\xa02\x9fZ\a(7\xa02\x81Z\x1f(\x13\xa0Ҁ\xda\x05(\xff\xa0R\x80\xda\r "), + MaxTimeMs: 7140000, + Data: []byte("\000x\000\000\000\000\000\000\000\000\000\340\324\003\302|\005\224\000\301\254}\351z2\320O\355\264n[\007\316\224\243md\371\320\375\032Pm\nS\235\016Q\255\006P\275\250\277\312\201Z\003(3\240R\207\332\005(\017\240\322\201\332=(\023\2402\203Z\007(w\2402\201Z\017(\023\265\227\364P\033@\245\007\364\nP\033C\245\002t\036P+@e\036\364\016Pk@e\002t:P;A\245\001\364\nS\373@\245\006t\006P+C\345\002\364\006Pk@\345\036t\nP\033A\245\003\364:P\033@\245\006t\016ZJ\377\\\205\313\210\327\270\017\345+F[\310\347E)\355\024\241\366\342}(v\215(N\203)\326\207(\336\203(V\332W\362\202t4\240m\005(\377AJ\006\320\322\202t\374\240\255\003(oA\312:\3202"), }, + { + Type: prompb.Chunk_XOR, + MinTimeMs: 7200000, + MaxTimeMs: 7200000, + Data: []byte("\000\001\200\364\356\006@\307p\000\000\000\000\000\000"), + }, + }, + }, + }, + }, + { + ChunkedSeries: []*prompb.ChunkedSeries{ + { + Labels: []prompb.Label{ + {Name: "__name__", Value: "test_metric1"}, + {Name: "b", Value: "c"}, + {Name: "baz", Value: "qux"}, + {Name: "d", Value: "e"}, + {Name: "foo", Value: "bar3"}, + }, + Chunks: []prompb.Chunk{ + { + Type: prompb.Chunk_XOR, + MaxTimeMs: 7140000, + Data: []byte("\000x\000\000\000\000\000\000\000\000\000\340\324\003\302|\005\224\000\301\254}\351z2\320O\355\264n[\007\316\224\243md\371\320\375\032Pm\nS\235\016Q\255\006P\275\250\277\312\201Z\003(3\240R\207\332\005(\017\240\322\201\332=(\023\2402\203Z\007(w\2402\201Z\017(\023\265\227\364P\033@\245\007\364\nP\033C\245\002t\036P+@e\036\364\016Pk@e\002t:P;A\245\001\364\nS\373@\245\006t\006P+C\345\002\364\006Pk@\345\036t\nP\033A\245\003\364:P\033@\245\006t\016ZJ\377\\\205\313\210\327\270\017\345+F[\310\347E)\355\024\241\366\342}(v\215(N\203)\326\207(\336\203(V\332W\362\202t4\240m\005(\377AJ\006\320\322\202t\374\240\255\003(oA\312:\3202"), + }, + { + Type: prompb.Chunk_XOR, + MinTimeMs: 7200000, + MaxTimeMs: 14340000, + Data: []byte("\000x\200\364\356\006@\307p\000\000\000\000\000\340\324\003\340>\224\355\260\277\322\200\372\005(=\240R\207:\003(\025\240\362\201z\003(\365\240r\203:\005(\r\241\322\201\372\r(\r\240R\237:\007(5\2402\201z\037(\025\2402\203:\005(\375\240R\200\372\r(\035\241\322\201:\003(5\240r\326g\364\271\213\227!\253q\037\312N\340GJ\033E)\375\024\241\266\362}(N\217(V\203)\336\207(\326\203(N\334W\322\203\2644\240}\005(\373AJ\031\3202\202\264\374\240\275\003(kA\3129\320R\201\2644\240\375\264\277\322\200\332\005(3\240r\207Z\003(\027\240\362\201Z\003(\363\240R\203\332\005(\017\241\322\201\332\r(\023\2402\237Z\007(7\2402\201Z\037(\023\240\322\200\332\005(\377\240R\200\332\r "), + }, + }, + }, + }, + }, + { + ChunkedSeries: []*prompb.ChunkedSeries{ + { + Labels: []prompb.Label{ + {Name: "__name__", Value: "test_metric1"}, + {Name: "b", Value: "c"}, + {Name: "baz", Value: "qux"}, + {Name: "d", Value: "e"}, + {Name: "foo", Value: "bar3"}, + }, + Chunks: []prompb.Chunk{ { Type: prompb.Chunk_XOR, MinTimeMs: 14400000, MaxTimeMs: 14400000, - Data: []byte("\x00\x01\x80\xe8\xdd\r@\xd7p\x00\x00\x00\x00\x00\x00"), - }, - }, - }, - }, - }, - { - ChunkedSeries: []*prompb.ChunkedSeries{ - { - Labels: []prompb.Label{ - {Name: "__name__", Value: "test_metric1"}, - {Name: "b", Value: "c"}, - {Name: "baz", Value: "qux"}, - {Name: "d", Value: "e"}, - {Name: "foo", Value: "bar3"}, - }, - Chunks: []prompb.Chunk{ - { - Type: prompb.Chunk_XOR, - MaxTimeMs: 14340000, - Data: []byte("\x00\xf0\x00\x00\x00\x00\x00\x00\x00\x00\x00\xe0\xd4\x03\xc2|\x05\x94\x00\xc1\xac}\xe9z2\xd0O\xed\xb4n[\aΔ\xa3md\xf9\xd0\xfd\x1aPm\nS\x9d\x0eQ\xad\x06P\xbd\xa8\xbfʁZ\x03(3\xa0R\x87\xda\x05(\x0f\xa0ҁ\xda=(\x13\xa02\x83Z\a(w\xa02\x81Z\x0f(\x13\xb5\x97\xf4P\x1b@\xa5\a\xf4\nP\x1bC\xa5\x02t\x1eP+@e\x1e\xf4\x0ePk@e\x02t:P;A\xa5\x01\xf4\nS\xfb@\xa5\x06t\x06P+C\xe5\x02\xf4\x06Pk@\xe5\x1et\nP\x1bA\xa5\x03\xf4:P\x1b@\xa5\x06t\x0eZJ\xff\\\x85ˈ\u05f8\x0f\xe5+F[\xc8\xe7E)\xed\x14\xa1\xf6\xe2}(v\x8d(N\x83)և(ރ(V\xdaW\xf2\x82t4\xa0m\x05(\xffAJ\x06\xd0҂t\xfc\xa0\xad\x03(oA\xca:\xd02\x82t4\xa0\xed\xb0\xbfҀ\xfa\x05(=\xa0R\x87:\x03(\x15\xa0\xf2\x81z\x03(\xf5\xa0r\x83:\x05(\r\xa1ҁ\xfa\r(\r\xa0R\x9f:\a(5\xa02\x81z\x1f(\x15\xa02\x83:\x05(\xfd\xa0R\x80\xfa\r(\x1d\xa1ҁ:\x03(5\xa0r\xd6g\xf4\xb9\x8b\x97!\xabq\x1f\xcaN\xe0GJ\x1bE)\xfd\x14\xa1\xb6\xf2}(N\x8f(V\x83)އ(փ(N\xdcW҃\xb44\xa0}\x05(\xfbAJ\x19\xd02\x82\xb4\xfc\xa0\xbd\x03(kA\xca9\xd0R\x81\xb44\xa0\xfd\xb4\xbfҀ\xda\x05(3\xa0r\x87Z\x03(\x17\xa0\xf2\x81Z\x03(\xf3\xa0R\x83\xda\x05(\x0f\xa1ҁ\xda\r(\x13\xa02\x9fZ\a(7\xa02\x81Z\x1f(\x13\xa0Ҁ\xda\x05(\xff\xa0R\x80\xda\r "), - }, - { - Type: prompb.Chunk_XOR, - MinTimeMs: 14400000, - MaxTimeMs: 28740000, - Data: []byte("\x00\xf0\x80\xe8\xdd\r@\xd7p\x00\x00\x00\x00\x00\xe0\xd4\x03\xe0G\xca+C)\xbd\x1c\xb6\x19\xfdh\x06P\x13\xa0i@v\x83\xa5\x00\xfa\x02\x94\x0fh\nP\xf3\xa0\x19@V\x81\xe5\x01z\x01\x94\x1dh\x0eP3\xa0)@6\x8f\xa5\x01\xfa\x06\x94\x03h\nPs\xa09@րe\x01z\x1f\x94\x05h\x06P3\xa0)A\xf6\x80\xa5\x00\xfa\x06\x94\ai\xfaP\x13\xa0\x19@ր\xe5\az\x01\x94\x05h\x1eP\x13\xa1\xe9@6\x80\xa5\x03\xfa\x02\x94\x03h:P\x13\xa0y@V\x80e\x1fz\x03\x94\rh\x06P\x13\xa0\xe9@v\x81\xa5\x00\xfa\x02\x94?h\nP3\xa0\x19@V\x83\xe5\x01z\x01\x94\rh\x0eZ\x8e\xff\xad\xccjSnC\xe9O\xdcH\xe9Ch\xa53\xa3\x97\x02}h2\x85\xe8\xf2\x85h2\x9c\xe8R\x8fhR\x83\xed\xe5}(;CJ\t\xd02\x8e\xb4\x1c\xa1\xbd\x03(+O\xca\t\xd0ҁ\xb4\x14\xa3\xfd\x05(\x1bCJ\tۋ\xff(\x15\xa02\x83z\a(u\xa02\x81:\r(\x1d\xa3Ҁ\xfa\x05(=\xa0R\x87:\x03(\x15\xa0\xf2\x81z\x03)\xf5\xa0r\x83:\x05(\r\xa1ҁ\xfa\r(\r\xa0R\x8f:\a(5\xa02\x81z\x1f(\x15\xa02\x83:\x05-\xa6\x7f\xda\x02\x94\x03\xe8\x1aP\x1d\xa0\xe9@N\x80e\x03Z\x03\x94=\xe8\x06P\x15\xa0y@N\x83\xa5\x00\xda\x02\x94\x0f\xe8\nP\r\xa3\xe9@N\x81\xe5\x01Z\x01\x94\x1d\xe8\x0eP5\xa0\x19@N\x87\xa5\x01\xda\x06\x94\x03\xe8\nP}\xa0)@\u0380e\x01Z\x7f\x94\x05\xe8\x06P5\xa09A\u0380\xa5\x00\xda\x06\x94\a\xe8zP\r\xa0)@\u0380\xe5\aZ\x01\x94\x05\xe8\x1eP\x15\xa0\x19G\u0380\xa5\x03\xda\x02\x94\x03\xe8:P\x1d\xa0i"), - }, - }, - }, - }, - }, - { - ChunkedSeries: []*prompb.ChunkedSeries{ - { - Labels: []prompb.Label{ - {Name: "__name__", Value: "test_metric1"}, - {Name: "b", Value: "c"}, - {Name: "baz", Value: "qux"}, - {Name: "d", Value: "e"}, - {Name: "foo", Value: "bar3"}, - }, - Chunks: []prompb.Chunk{ - { - Type: prompb.Chunk_XOR, - MinTimeMs: 28800000, - MaxTimeMs: 28800000, - Data: []byte("\x00\x01\x80л\x1b@\xe7p\x00\x00\x00\x00\x00\x00"), + Data: []byte("\000\001\200\350\335\r@\327p\000\000\000\000\000\000"), }, }, }, @@ -412,8 +409,8 @@ func TestStreamReadEndpoint(t *testing.T) { Chunks: []prompb.Chunk{ { Type: prompb.Chunk_XOR, - MaxTimeMs: 14340000, - Data: []byte("\x00\xf0\x00\x00\x00\x00\x00\x00\x00\x00\x00\xe0\xd4\x03\xc2|\x05\x94\x00\xc1\xac}\xe9z2\xd0O\xed\xb4n[\aΔ\xa3md\xf9\xd0\xfd\x1aPm\nS\x9d\x0eQ\xad\x06P\xbd\xa8\xbfʁZ\x03(3\xa0R\x87\xda\x05(\x0f\xa0ҁ\xda=(\x13\xa02\x83Z\a(w\xa02\x81Z\x0f(\x13\xb5\x97\xf4P\x1b@\xa5\a\xf4\nP\x1bC\xa5\x02t\x1eP+@e\x1e\xf4\x0ePk@e\x02t:P;A\xa5\x01\xf4\nS\xfb@\xa5\x06t\x06P+C\xe5\x02\xf4\x06Pk@\xe5\x1et\nP\x1bA\xa5\x03\xf4:P\x1b@\xa5\x06t\x0eZJ\xff\\\x85ˈ\u05f8\x0f\xe5+F[\xc8\xe7E)\xed\x14\xa1\xf6\xe2}(v\x8d(N\x83)և(ރ(V\xdaW\xf2\x82t4\xa0m\x05(\xffAJ\x06\xd0҂t\xfc\xa0\xad\x03(oA\xca:\xd02\x82t4\xa0\xed\xb0\xbfҀ\xfa\x05(=\xa0R\x87:\x03(\x15\xa0\xf2\x81z\x03(\xf5\xa0r\x83:\x05(\r\xa1ҁ\xfa\r(\r\xa0R\x9f:\a(5\xa02\x81z\x1f(\x15\xa02\x83:\x05(\xfd\xa0R\x80\xfa\r(\x1d\xa1ҁ:\x03(5\xa0r\xd6g\xf4\xb9\x8b\x97!\xabq\x1f\xcaN\xe0GJ\x1bE)\xfd\x14\xa1\xb6\xf2}(N\x8f(V\x83)އ(փ(N\xdcW҃\xb44\xa0}\x05(\xfbAJ\x19\xd02\x82\xb4\xfc\xa0\xbd\x03(kA\xca9\xd0R\x81\xb44\xa0\xfd\xb4\xbfҀ\xda\x05(3\xa0r\x87Z\x03(\x17\xa0\xf2\x81Z\x03(\xf3\xa0R\x83\xda\x05(\x0f\xa1ҁ\xda\r(\x13\xa02\x9fZ\a(7\xa02\x81Z\x1f(\x13\xa0Ҁ\xda\x05(\xff\xa0R\x80\xda\r "), + MaxTimeMs: 7140000, + Data: []byte("\000x\000\000\000\000\000\000\000\000\000\340\324\003\302|\005\224\000\301\254}\351z2\320O\355\264n[\007\316\224\243md\371\320\375\032Pm\nS\235\016Q\255\006P\275\250\277\312\201Z\003(3\240R\207\332\005(\017\240\322\201\332=(\023\2402\203Z\007(w\2402\201Z\017(\023\265\227\364P\033@\245\007\364\nP\033C\245\002t\036P+@e\036\364\016Pk@e\002t:P;A\245\001\364\nS\373@\245\006t\006P+C\345\002\364\006Pk@\345\036t\nP\033A\245\003\364:P\033@\245\006t\016ZJ\377\\\205\313\210\327\270\017\345+F[\310\347E)\355\024\241\366\342}(v\215(N\203)\326\207(\336\203(V\332W\362\202t4\240m\005(\377AJ\006\320\322\202t\374\240\255\003(oA\312:\3202"), }, }, }, diff --git a/tsdb/db_test.go b/tsdb/db_test.go index 14297c3dc3..70639085e4 100644 --- a/tsdb/db_test.go +++ b/tsdb/db_test.go @@ -247,8 +247,8 @@ func TestNoPanicAfterWALCorruption(t *testing.T) { var maxt int64 ctx := context.Background() { - // Appending 221 samples because on the 221st a new chunk will be created. - for i := 0; i < 221; i++ { + // Appending 121 samples because on the 121st a new chunk will be created. + for i := 0; i < 121; i++ { app := db.Appender(ctx) _, err := app.Append(0, labels.FromStrings("foo", "bar"), maxt, 0) expSamples = append(expSamples, sample{t: maxt, v: 0}) @@ -1089,9 +1089,9 @@ func TestWALReplayRaceOnSamplesLoggedBeforeSeries(t *testing.T) { numSamplesBeforeSeriesCreation = 1000 ) - // We test both with few and many samples appended after series creation. If samples are < 220 then there's no + // We test both with few and many samples appended after series creation. If samples are < 120 then there's no // mmap-ed chunk, otherwise there's at least 1 mmap-ed chunk when replaying the WAL. - for _, numSamplesAfterSeriesCreation := range []int{1, 2000} { + for _, numSamplesAfterSeriesCreation := range []int{1, 1000} { for run := 1; run <= numRuns; run++ { t.Run(fmt.Sprintf("samples after series creation = %d, run = %d", numSamplesAfterSeriesCreation, run), func(t *testing.T) { testWALReplayRaceOnSamplesLoggedBeforeSeries(t, numSamplesBeforeSeriesCreation, numSamplesAfterSeriesCreation) @@ -1160,8 +1160,8 @@ func testWALReplayRaceOnSamplesLoggedBeforeSeries(t *testing.T, numSamplesBefore } require.NoError(t, chunksIt.Err()) - // We expect 1 chunk every 220 samples after series creation. - require.Equalf(t, (numSamplesAfterSeriesCreation/220)+1, actualChunks, "series: %s", set.At().Labels().String()) + // We expect 1 chunk every 120 samples after series creation. + require.Equalf(t, (numSamplesAfterSeriesCreation/120)+1, actualChunks, "series: %s", set.At().Labels().String()) } require.NoError(t, set.Err()) diff --git a/tsdb/head_test.go b/tsdb/head_test.go index df48e592d9..80b71e9271 100644 --- a/tsdb/head_test.go +++ b/tsdb/head_test.go @@ -808,7 +808,7 @@ func TestMemSeries_truncateChunks(t *testing.T) { s := newMemSeries(labels.FromStrings("a", "b"), 1, defaultIsolationDisabled, DefaultSamplesPerChunk) - for i := 0; i < 8000; i += 5 { + for i := 0; i < 4000; i += 5 { ok, _ := s.append(int64(i), float64(i), 0, chunkDiskMapper, chunkRange) require.True(t, ok, "sample append failed") } @@ -825,9 +825,9 @@ func TestMemSeries_truncateChunks(t *testing.T) { require.NotNil(t, chk) require.NoError(t, err) - s.truncateChunksBefore(4000, 0) + s.truncateChunksBefore(2000, 0) - require.Equal(t, int64(4000), s.mmappedChunks[0].minTime) + require.Equal(t, int64(2000), s.mmappedChunks[0].minTime) _, _, err = s.chunk(0, chunkDiskMapper, &memChunkPool) require.Equal(t, storage.ErrNotFound, err, "first chunks not gone") require.Equal(t, countBefore/2, len(s.mmappedChunks)+1) // +1 for the head chunk. @@ -1364,9 +1364,9 @@ func TestMemSeries_append(t *testing.T) { require.Equal(t, int64(1000), s.headChunk.minTime, "wrong chunk range") require.Equal(t, int64(1001), s.headChunk.maxTime, "wrong chunk range") - // Fill the range [1000,3000) with many samples. Intermediate chunks should be cut - // at approximately 220 samples per chunk. - for i := 1; i < 2000; i++ { + // Fill the range [1000,2000) with many samples. Intermediate chunks should be cut + // at approximately 120 samples per chunk. + for i := 1; i < 1000; i++ { ok, _ := s.append(1001+int64(i), float64(i), 0, chunkDiskMapper, chunkRange) require.True(t, ok, "append failed") } @@ -1437,7 +1437,7 @@ func TestMemSeries_appendHistogram(t *testing.T) { } func TestMemSeries_append_atVariableRate(t *testing.T) { - const samplesPerChunk = 220 + const samplesPerChunk = 120 dir := t.TempDir() // This is usually taken from the Head, but passing manually here. chunkDiskMapper, err := chunks.NewChunkDiskMapper(nil, dir, chunkenc.NewPool(), chunks.DefaultWriteBufferSize, chunks.DefaultWriteQueueSize) @@ -2983,7 +2983,7 @@ func TestAppendHistogram(t *testing.T) { } func TestHistogramInWALAndMmapChunk(t *testing.T) { - head, _ := newTestHead(t, 6000, false, false) + head, _ := newTestHead(t, 3000, false, false) t.Cleanup(func() { require.NoError(t, head.Close()) }) @@ -2992,7 +2992,7 @@ func TestHistogramInWALAndMmapChunk(t *testing.T) { // Series with only histograms. s1 := labels.FromStrings("a", "b1") k1 := s1.String() - numHistograms := 600 + numHistograms := 300 exp := map[string][]tsdbutil.Sample{} ts := int64(0) var app storage.Appender @@ -3728,7 +3728,7 @@ func TestHistogramCounterResetHeader(t *testing.T) { checkExpCounterResetHeader(chunkenc.CounterReset) // Add 2 non-counter reset histograms. - for i := 0; i < 500; i++ { + for i := 0; i < 250; i++ { appendHistogram(h) } checkExpCounterResetHeader(chunkenc.NotCounterReset, chunkenc.NotCounterReset) @@ -3756,7 +3756,7 @@ func TestHistogramCounterResetHeader(t *testing.T) { checkExpCounterResetHeader(chunkenc.CounterReset) // Add 2 non-counter reset histograms. Just to have some non-counter reset chunks in between. - for i := 0; i < 500; i++ { + for i := 0; i < 250; i++ { appendHistogram(h) } checkExpCounterResetHeader(chunkenc.NotCounterReset, chunkenc.NotCounterReset) @@ -4223,7 +4223,7 @@ func TestHeadInit_DiscardChunksWithUnsupportedEncoding(t *testing.T) { h.chunkDiskMapper.WriteChunk(chunks.HeadSeriesRef(seriesRef), 500, 600, uc, false, func(err error) { require.NoError(t, err) }) app = h.Appender(ctx) - for i := 700; i < 1700; i++ { + for i := 700; i < 1200; i++ { _, err := app.Append(0, seriesLabels, int64(i), float64(i)) require.NoError(t, err) } From 5d4ec08a1fd65ff84ea16bca22ad3b20b91604f6 Mon Sep 17 00:00:00 2001 From: Sebastian Rabenhorst Date: Fri, 14 Apr 2023 11:59:30 +0200 Subject: [PATCH 094/231] Fixed sampleRingIterator for mixed histograms Signed-off-by: Sebastian Rabenhorst Fixed sampleRingIterator for mixed histograms Signed-off-by: Sebastian Rabenhorst Fixed lint --- storage/buffer.go | 2 ++ storage/buffer_test.go | 23 +++++++++++++++++++++++ 2 files changed, 25 insertions(+) diff --git a/storage/buffer.go b/storage/buffer.go index 27ac21661b..2229e52591 100644 --- a/storage/buffer.go +++ b/storage/buffer.go @@ -332,9 +332,11 @@ func (it *sampleRingIterator) Next() chunkenc.ValueType { switch s.Type() { case chunkenc.ValHistogram: it.h = s.H() + it.fh = nil return chunkenc.ValHistogram case chunkenc.ValFloatHistogram: it.fh = s.FH() + it.h = nil return chunkenc.ValFloatHistogram default: it.f = s.F() diff --git a/storage/buffer_test.go b/storage/buffer_test.go index ebe24d8df3..70cd7f4ff5 100644 --- a/storage/buffer_test.go +++ b/storage/buffer_test.go @@ -21,6 +21,7 @@ import ( "github.com/prometheus/prometheus/model/histogram" "github.com/prometheus/prometheus/tsdb/chunkenc" + "github.com/prometheus/prometheus/tsdb/tsdbutil" ) func TestSampleRing(t *testing.T) { @@ -180,6 +181,28 @@ func TestBufferedSeriesIteratorNoBadAt(t *testing.T) { it.Next() } +func TestBufferedSeriesIteratorMixedHistograms(t *testing.T) { + histograms := tsdbutil.GenerateTestHistograms(2) + + it := NewBufferIterator(NewListSeriesIterator(samples{ + fhSample{t: 1, fh: histograms[0].ToFloat()}, + hSample{t: 2, h: histograms[1]}, + }), 2) + + require.Equal(t, chunkenc.ValNone, it.Seek(3)) + require.NoError(t, it.Err()) + + buf := it.Buffer() + + require.Equal(t, chunkenc.ValFloatHistogram, buf.Next()) + _, fh := buf.AtFloatHistogram() + require.Equal(t, histograms[0].ToFloat(), fh) + + require.Equal(t, chunkenc.ValHistogram, buf.Next()) + _, fh = buf.AtFloatHistogram() + require.Equal(t, histograms[1].ToFloat(), fh) +} + func BenchmarkBufferedSeriesIterator(b *testing.B) { // Simulate a 5 minute rate. it := NewBufferIterator(newFakeSeriesIterator(int64(b.N), 30), 5*60) From 1801cd41966be6c9566bb0104cfa7bafc2a6a7fa Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Fri, 14 Apr 2023 14:37:35 +0000 Subject: [PATCH 095/231] labels: small optimization to stringlabels Add a fast path for the common case that a string is less than 127 bytes long, to skip a shift and the loop. Signed-off-by: Bryan Boreham --- model/labels/labels_string.go | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/model/labels/labels_string.go b/model/labels/labels_string.go index f293c91675..ff46103ebc 100644 --- a/model/labels/labels_string.go +++ b/model/labels/labels_string.go @@ -56,8 +56,14 @@ func (ls labelSlice) Swap(i, j int) { ls[i], ls[j] = ls[j], ls[i] } func (ls labelSlice) Less(i, j int) bool { return ls[i].Name < ls[j].Name } func decodeSize(data string, index int) (int, int) { - var size int - for shift := uint(0); ; shift += 7 { + // Fast-path for common case of a single byte, value 0..127. + b := data[index] + index++ + if b < 0x80 { + return int(b), index + } + size := int(b & 0x7F) + for shift := uint(7); ; shift += 7 { // Just panic if we go of the end of data, since all Labels strings are constructed internally and // malformed data indicates a bug, or memory corruption. b := data[index] From fd3630b9a3c7accecbf7fd438706c43612cfc568 Mon Sep 17 00:00:00 2001 From: Ben Ye Date: Mon, 17 Apr 2023 21:32:38 -0700 Subject: [PATCH 096/231] add ctx to QueryEngine interface Signed-off-by: Ben Ye --- cmd/promtool/unittest.go | 2 +- promql/bench_test.go | 5 ++-- promql/engine.go | 4 ++-- promql/engine_test.go | 52 +++++++++++++++++++++------------------- promql/functions_test.go | 5 ++-- promql/promql_test.go | 5 ++-- promql/test.go | 4 ++-- rules/manager.go | 2 +- web/api/v1/api.go | 8 +++---- 9 files changed, 46 insertions(+), 41 deletions(-) diff --git a/cmd/promtool/unittest.go b/cmd/promtool/unittest.go index b3e6f67f97..b9d8e05275 100644 --- a/cmd/promtool/unittest.go +++ b/cmd/promtool/unittest.go @@ -434,7 +434,7 @@ func (tg *testGroup) maxEvalTime() time.Duration { } func query(ctx context.Context, qs string, t time.Time, engine *promql.Engine, qu storage.Queryable) (promql.Vector, error) { - q, err := engine.NewInstantQuery(qu, nil, qs, t) + q, err := engine.NewInstantQuery(ctx, qu, nil, qs, t) if err != nil { return nil, err } diff --git a/promql/bench_test.go b/promql/bench_test.go index 88025d9325..3d35884477 100644 --- a/promql/bench_test.go +++ b/promql/bench_test.go @@ -240,16 +240,17 @@ func BenchmarkRangeQuery(b *testing.B) { for _, c := range cases { name := fmt.Sprintf("expr=%s,steps=%d", c.expr, c.steps) b.Run(name, func(b *testing.B) { + ctx := context.Background() b.ReportAllocs() for i := 0; i < b.N; i++ { qry, err := engine.NewRangeQuery( - stor, nil, c.expr, + ctx, stor, nil, c.expr, time.Unix(int64((numIntervals-c.steps)*10), 0), time.Unix(int64(numIntervals*10), 0), time.Second*10) if err != nil { b.Fatal(err) } - res := qry.Exec(context.Background()) + res := qry.Exec(ctx) if res.Err != nil { b.Fatal(res.Err) } diff --git a/promql/engine.go b/promql/engine.go index b49be244f1..fae4f09320 100644 --- a/promql/engine.go +++ b/promql/engine.go @@ -400,7 +400,7 @@ func (ng *Engine) SetQueryLogger(l QueryLogger) { } // NewInstantQuery returns an evaluation query for the given expression at the given time. -func (ng *Engine) NewInstantQuery(q storage.Queryable, opts *QueryOpts, qs string, ts time.Time) (Query, error) { +func (ng *Engine) NewInstantQuery(ctx context.Context, q storage.Queryable, opts *QueryOpts, qs string, ts time.Time) (Query, error) { expr, err := parser.ParseExpr(qs) if err != nil { return nil, err @@ -416,7 +416,7 @@ func (ng *Engine) NewInstantQuery(q storage.Queryable, opts *QueryOpts, qs strin // NewRangeQuery returns an evaluation query for the given time range and with // the resolution set by the interval. -func (ng *Engine) NewRangeQuery(q storage.Queryable, opts *QueryOpts, qs string, start, end time.Time, interval time.Duration) (Query, error) { +func (ng *Engine) NewRangeQuery(ctx context.Context, q storage.Queryable, opts *QueryOpts, qs string, start, end time.Time, interval time.Duration) (Query, error) { expr, err := parser.ParseExpr(qs) if err != nil { return nil, err diff --git a/promql/engine_test.go b/promql/engine_test.go index b64e32ba46..2941a22408 100644 --- a/promql/engine_test.go +++ b/promql/engine_test.go @@ -231,14 +231,14 @@ func TestQueryError(t *testing.T) { ctx, cancelCtx := context.WithCancel(context.Background()) defer cancelCtx() - vectorQuery, err := engine.NewInstantQuery(queryable, nil, "foo", time.Unix(1, 0)) + vectorQuery, err := engine.NewInstantQuery(ctx, queryable, nil, "foo", time.Unix(1, 0)) require.NoError(t, err) res := vectorQuery.Exec(ctx) require.Error(t, res.Err, "expected error on failed select but got none") require.True(t, errors.Is(res.Err, errStorage), "expected error doesn't match") - matrixQuery, err := engine.NewInstantQuery(queryable, nil, "foo[1m]", time.Unix(1, 0)) + matrixQuery, err := engine.NewInstantQuery(ctx, queryable, nil, "foo[1m]", time.Unix(1, 0)) require.NoError(t, err) res = matrixQuery.Exec(ctx) @@ -564,14 +564,15 @@ func TestSelectHintsSetCorrectly(t *testing.T) { query Query err error ) + ctx := context.Background() if tc.end == 0 { - query, err = engine.NewInstantQuery(hintsRecorder, nil, tc.query, timestamp.Time(tc.start)) + query, err = engine.NewInstantQuery(ctx, hintsRecorder, nil, tc.query, timestamp.Time(tc.start)) } else { - query, err = engine.NewRangeQuery(hintsRecorder, nil, tc.query, timestamp.Time(tc.start), timestamp.Time(tc.end), time.Second) + query, err = engine.NewRangeQuery(ctx, hintsRecorder, nil, tc.query, timestamp.Time(tc.start), timestamp.Time(tc.end), time.Second) } require.NoError(t, err) - res := query.Exec(context.Background()) + res := query.Exec(ctx) require.NoError(t, res.Err) require.Equal(t, tc.expected, hintsRecorder.hints) @@ -727,9 +728,9 @@ load 10s var err error var qry Query if c.Interval == 0 { - qry, err = test.QueryEngine().NewInstantQuery(test.Queryable(), nil, c.Query, c.Start) + qry, err = test.QueryEngine().NewInstantQuery(test.context, test.Queryable(), nil, c.Query, c.Start) } else { - qry, err = test.QueryEngine().NewRangeQuery(test.Queryable(), nil, c.Query, c.Start, c.End, c.Interval) + qry, err = test.QueryEngine().NewRangeQuery(test.context, test.Queryable(), nil, c.Query, c.Start, c.End, c.Interval) } require.NoError(t, err) @@ -1204,9 +1205,9 @@ load 10s var err error var qry Query if c.Interval == 0 { - qry, err = engine.NewInstantQuery(test.Queryable(), opts, c.Query, c.Start) + qry, err = engine.NewInstantQuery(test.context, test.Queryable(), opts, c.Query, c.Start) } else { - qry, err = engine.NewRangeQuery(test.Queryable(), opts, c.Query, c.Start, c.End, c.Interval) + qry, err = engine.NewRangeQuery(test.context, test.Queryable(), opts, c.Query, c.Start, c.End, c.Interval) } require.NoError(t, err) @@ -1387,9 +1388,9 @@ load 10s var err error var qry Query if c.Interval == 0 { - qry, err = engine.NewInstantQuery(test.Queryable(), nil, c.Query, c.Start) + qry, err = engine.NewInstantQuery(test.context, test.Queryable(), nil, c.Query, c.Start) } else { - qry, err = engine.NewRangeQuery(test.Queryable(), nil, c.Query, c.Start, c.End, c.Interval) + qry, err = engine.NewRangeQuery(test.context, test.Queryable(), nil, c.Query, c.Start, c.End, c.Interval) } require.NoError(t, err) @@ -1628,9 +1629,9 @@ load 1ms var err error var qry Query if c.end == 0 { - qry, err = test.QueryEngine().NewInstantQuery(test.Queryable(), nil, c.query, start) + qry, err = test.QueryEngine().NewInstantQuery(test.context, test.Queryable(), nil, c.query, start) } else { - qry, err = test.QueryEngine().NewRangeQuery(test.Queryable(), nil, c.query, start, end, interval) + qry, err = test.QueryEngine().NewRangeQuery(test.context, test.Queryable(), nil, c.query, start, end, interval) } require.NoError(t, err) @@ -1961,7 +1962,7 @@ func TestSubquerySelector(t *testing.T) { engine := test.QueryEngine() for _, c := range tst.cases { t.Run(c.Query, func(t *testing.T) { - qry, err := engine.NewInstantQuery(test.Queryable(), nil, c.Query, c.Start) + qry, err := engine.NewInstantQuery(test.context, test.Queryable(), nil, c.Query, c.Start) require.NoError(t, err) res := qry.Exec(test.Context()) @@ -2909,6 +2910,7 @@ func TestPreprocessAndWrapWithStepInvariantExpr(t *testing.T) { } func TestEngineOptsValidation(t *testing.T) { + ctx := context.Background() cases := []struct { opts EngineOpts query string @@ -2968,8 +2970,8 @@ func TestEngineOptsValidation(t *testing.T) { for _, c := range cases { eng := NewEngine(c.opts) - _, err1 := eng.NewInstantQuery(nil, nil, c.query, time.Unix(10, 0)) - _, err2 := eng.NewRangeQuery(nil, nil, c.query, time.Unix(0, 0), time.Unix(10, 0), time.Second) + _, err1 := eng.NewInstantQuery(ctx, nil, nil, c.query, time.Unix(10, 0)) + _, err2 := eng.NewRangeQuery(ctx, nil, nil, c.query, time.Unix(0, 0), time.Unix(10, 0), time.Second) if c.fail { require.Equal(t, c.expError, err1) require.Equal(t, c.expError, err2) @@ -3116,7 +3118,7 @@ func TestRangeQuery(t *testing.T) { err = test.Run() require.NoError(t, err) - qry, err := test.QueryEngine().NewRangeQuery(test.Queryable(), nil, c.Query, c.Start, c.End, c.Interval) + qry, err := test.QueryEngine().NewRangeQuery(test.context, test.Queryable(), nil, c.Query, c.Start, c.End, c.Interval) require.NoError(t, err) res := qry.Exec(test.Context()) @@ -3147,7 +3149,7 @@ func TestNativeHistogramRate(t *testing.T) { engine := test.QueryEngine() queryString := fmt.Sprintf("rate(%s[1m])", seriesName) - qry, err := engine.NewInstantQuery(test.Queryable(), nil, queryString, timestamp.Time(int64(5*time.Minute/time.Millisecond))) + qry, err := engine.NewInstantQuery(test.context, test.Queryable(), nil, queryString, timestamp.Time(int64(5*time.Minute/time.Millisecond))) require.NoError(t, err) res := qry.Exec(test.Context()) require.NoError(t, res.Err) @@ -3191,7 +3193,7 @@ func TestNativeFloatHistogramRate(t *testing.T) { engine := test.QueryEngine() queryString := fmt.Sprintf("rate(%s[1m])", seriesName) - qry, err := engine.NewInstantQuery(test.Queryable(), nil, queryString, timestamp.Time(int64(5*time.Minute/time.Millisecond))) + qry, err := engine.NewInstantQuery(test.context, test.Queryable(), nil, queryString, timestamp.Time(int64(5*time.Minute/time.Millisecond))) require.NoError(t, err) res := qry.Exec(test.Context()) require.NoError(t, res.Err) @@ -3255,7 +3257,7 @@ func TestNativeHistogram_HistogramCountAndSum(t *testing.T) { require.NoError(t, app.Commit()) queryString := fmt.Sprintf("histogram_count(%s)", seriesName) - qry, err := engine.NewInstantQuery(test.Queryable(), nil, queryString, timestamp.Time(ts)) + qry, err := engine.NewInstantQuery(test.context, test.Queryable(), nil, queryString, timestamp.Time(ts)) require.NoError(t, err) res := qry.Exec(test.Context()) @@ -3273,7 +3275,7 @@ func TestNativeHistogram_HistogramCountAndSum(t *testing.T) { } queryString = fmt.Sprintf("histogram_sum(%s)", seriesName) - qry, err = engine.NewInstantQuery(test.Queryable(), nil, queryString, timestamp.Time(ts)) + qry, err = engine.NewInstantQuery(test.context, test.Queryable(), nil, queryString, timestamp.Time(ts)) require.NoError(t, err) res = qry.Exec(test.Context()) @@ -3509,7 +3511,7 @@ func TestNativeHistogram_HistogramQuantile(t *testing.T) { for j, sc := range c.subCases { t.Run(fmt.Sprintf("%d %s", j, sc.quantile), func(t *testing.T) { queryString := fmt.Sprintf("histogram_quantile(%s, %s)", sc.quantile, seriesName) - qry, err := engine.NewInstantQuery(test.Queryable(), nil, queryString, timestamp.Time(ts)) + qry, err := engine.NewInstantQuery(test.context, test.Queryable(), nil, queryString, timestamp.Time(ts)) require.NoError(t, err) res := qry.Exec(test.Context()) @@ -3940,7 +3942,7 @@ func TestNativeHistogram_HistogramFraction(t *testing.T) { for j, sc := range c.subCases { t.Run(fmt.Sprintf("%d %s %s", j, sc.lower, sc.upper), func(t *testing.T) { queryString := fmt.Sprintf("histogram_fraction(%s, %s, %s)", sc.lower, sc.upper, seriesName) - qry, err := engine.NewInstantQuery(test.Queryable(), nil, queryString, timestamp.Time(ts)) + qry, err := engine.NewInstantQuery(test.context, test.Queryable(), nil, queryString, timestamp.Time(ts)) require.NoError(t, err) res := qry.Exec(test.Context()) @@ -4077,7 +4079,7 @@ func TestNativeHistogram_Sum_Count_AddOperator(t *testing.T) { require.NoError(t, app.Commit()) queryAndCheck := func(queryString string, exp Vector) { - qry, err := engine.NewInstantQuery(test.Queryable(), nil, queryString, timestamp.Time(ts)) + qry, err := engine.NewInstantQuery(test.context, test.Queryable(), nil, queryString, timestamp.Time(ts)) require.NoError(t, err) res := qry.Exec(test.Context()) @@ -4186,7 +4188,7 @@ metric 0 1 2 opts := &QueryOpts{ LookbackDelta: c.queryLookback, } - qry, err := eng.NewInstantQuery(test.Queryable(), opts, query, c.ts) + qry, err := eng.NewInstantQuery(test.context, test.Queryable(), opts, query, c.ts) require.NoError(t, err) res := qry.Exec(test.Context()) diff --git a/promql/functions_test.go b/promql/functions_test.go index e552424b3d..8df5684b8c 100644 --- a/promql/functions_test.go +++ b/promql/functions_test.go @@ -56,10 +56,11 @@ func TestDeriv(t *testing.T) { require.NoError(t, a.Commit()) - query, err := engine.NewInstantQuery(storage, nil, "deriv(foo[30m])", timestamp.Time(1493712846939)) + ctx := context.Background() + query, err := engine.NewInstantQuery(ctx, storage, nil, "deriv(foo[30m])", timestamp.Time(1493712846939)) require.NoError(t, err) - result := query.Exec(context.Background()) + result := query.Exec(ctx) require.NoError(t, result.Err) vec, _ := result.Vector() diff --git a/promql/promql_test.go b/promql/promql_test.go index 68bfa1d95f..a07a0f5cb7 100644 --- a/promql/promql_test.go +++ b/promql/promql_test.go @@ -81,14 +81,15 @@ func TestConcurrentRangeQueries(t *testing.T) { defer func() { sem <- struct{}{} }() + ctx := context.Background() qry, err := engine.NewRangeQuery( - stor, nil, c.expr, + ctx, stor, nil, c.expr, time.Unix(int64((numIntervals-c.steps)*10), 0), time.Unix(int64(numIntervals*10), 0), time.Second*10) if err != nil { return err } - res := qry.Exec(context.Background()) + res := qry.Exec(ctx) if res.Err != nil { t.Logf("Query: %q, steps: %d, result: %s", c.expr, c.steps, res.Err) return res.Err diff --git a/promql/test.go b/promql/test.go index 0b3958393f..64fa66d5d3 100644 --- a/promql/test.go +++ b/promql/test.go @@ -538,7 +538,7 @@ func (t *Test) exec(tc testCommand) error { } queries = append([]atModifierTestCase{{expr: cmd.expr, evalTime: cmd.start}}, queries...) for _, iq := range queries { - q, err := t.QueryEngine().NewInstantQuery(t.storage, nil, iq.expr, iq.evalTime) + q, err := t.QueryEngine().NewInstantQuery(t.context, t.storage, nil, iq.expr, iq.evalTime) if err != nil { return err } @@ -560,7 +560,7 @@ func (t *Test) exec(tc testCommand) error { // Check query returns same result in range mode, // by checking against the middle step. - q, err = t.queryEngine.NewRangeQuery(t.storage, nil, iq.expr, iq.evalTime.Add(-time.Minute), iq.evalTime.Add(time.Minute), time.Minute) + q, err = t.queryEngine.NewRangeQuery(t.context, t.storage, nil, iq.expr, iq.evalTime.Add(-time.Minute), iq.evalTime.Add(time.Minute), time.Minute) if err != nil { return err } diff --git a/rules/manager.go b/rules/manager.go index 07d50be1b9..b6513e82d4 100644 --- a/rules/manager.go +++ b/rules/manager.go @@ -189,7 +189,7 @@ type QueryFunc func(ctx context.Context, q string, t time.Time) (promql.Vector, // It converts scalar into vector results. func EngineQueryFunc(engine *promql.Engine, q storage.Queryable) QueryFunc { return func(ctx context.Context, qs string, t time.Time) (promql.Vector, error) { - q, err := engine.NewInstantQuery(q, nil, qs, t) + q, err := engine.NewInstantQuery(ctx, q, nil, qs, t) if err != nil { return nil, err } diff --git a/web/api/v1/api.go b/web/api/v1/api.go index aeea87ca70..dde814eb0f 100644 --- a/web/api/v1/api.go +++ b/web/api/v1/api.go @@ -177,8 +177,8 @@ type TSDBAdminStats interface { // QueryEngine defines the interface for the *promql.Engine, so it can be replaced, wrapped or mocked. type QueryEngine interface { SetQueryLogger(l promql.QueryLogger) - NewInstantQuery(q storage.Queryable, opts *promql.QueryOpts, qs string, ts time.Time) (promql.Query, error) - NewRangeQuery(q storage.Queryable, opts *promql.QueryOpts, qs string, start, end time.Time, interval time.Duration) (promql.Query, error) + NewInstantQuery(ctx context.Context, q storage.Queryable, opts *promql.QueryOpts, qs string, ts time.Time) (promql.Query, error) + NewRangeQuery(ctx context.Context, q storage.Queryable, opts *promql.QueryOpts, qs string, start, end time.Time, interval time.Duration) (promql.Query, error) } // API can register a set of endpoints in a router and handle @@ -417,7 +417,7 @@ func (api *API) query(r *http.Request) (result apiFuncResult) { if err != nil { return apiFuncResult{nil, &apiError{errorBadData, err}, nil, nil} } - qry, err := api.QueryEngine.NewInstantQuery(api.Queryable, opts, r.FormValue("query"), ts) + qry, err := api.QueryEngine.NewInstantQuery(ctx, api.Queryable, opts, r.FormValue("query"), ts) if err != nil { return invalidParamError(err, "query") } @@ -520,7 +520,7 @@ func (api *API) queryRange(r *http.Request) (result apiFuncResult) { if err != nil { return apiFuncResult{nil, &apiError{errorBadData, err}, nil, nil} } - qry, err := api.QueryEngine.NewRangeQuery(api.Queryable, opts, r.FormValue("query"), start, end, step) + qry, err := api.QueryEngine.NewRangeQuery(ctx, api.Queryable, opts, r.FormValue("query"), start, end, step) if err != nil { return apiFuncResult{nil, &apiError{errorBadData, err}, nil, nil} } From f3394bf7a1f362b88d89cb0784be47b75d8fa152 Mon Sep 17 00:00:00 2001 From: gotjosh Date: Tue, 18 Apr 2023 10:07:32 +0100 Subject: [PATCH 097/231] Rules API: Allow filtering by rule name Introduces support for a new query parameter in the `/rules` API endpoint that allows filtering by rule names. If all the rules of a group are filtered, we skip the group entirely. Signed-off-by: gotjosh --- docs/querying/api.md | 2 ++ web/api/v1/api.go | 33 +++++++++++++++++++++++++++------ web/api/v1/api_test.go | 27 +++++++++++++++++++++++++++ 3 files changed, 56 insertions(+), 6 deletions(-) diff --git a/docs/querying/api.md b/docs/querying/api.md index f2182a2052..0cc549b65e 100644 --- a/docs/querying/api.md +++ b/docs/querying/api.md @@ -673,7 +673,9 @@ GET /api/v1/rules ``` URL query parameters: + - `type=alert|record`: return only the alerting rules (e.g. `type=alert`) or the recording rules (e.g. `type=record`). When the parameter is absent or empty, no filtering is done. +- `rules=alertName,RuleName`: return only the alerting and recording rules with the specified names. If we've filtered out all the rules of a group, the group is not returned. When the parameter is absent or empty, no filtering is done. ```json $ curl http://localhost:9090/api/v1/rules diff --git a/web/api/v1/api.go b/web/api/v1/api.go index aeea87ca70..d955958041 100644 --- a/web/api/v1/api.go +++ b/web/api/v1/api.go @@ -1296,6 +1296,16 @@ func (api *API) rules(r *http.Request) apiFuncResult { res := &RuleDiscovery{RuleGroups: make([]*RuleGroup, len(ruleGroups))} typ := strings.ToLower(r.URL.Query().Get("type")) + // Parse the rule names into a comma separated list of rule names, then create a set. + rulesQuery := r.URL.Query().Get("rules") + ruleNamesSet := map[string]struct{}{} + if rulesQuery != "" { + names := strings.Split(rulesQuery, ",") + for _, rn := range names { + ruleNamesSet[strings.TrimSpace(rn)] = struct{}{} + } + } + if typ != "" && typ != "alert" && typ != "record" { return invalidParamError(errors.Errorf("not supported value %q", typ), "type") } @@ -1313,14 +1323,20 @@ func (api *API) rules(r *http.Request) apiFuncResult { EvaluationTime: grp.GetEvaluationTime().Seconds(), LastEvaluation: grp.GetLastEvaluation(), } - for _, r := range grp.Rules() { + for _, rr := range grp.Rules() { var enrichedRule Rule - lastError := "" - if r.LastError() != nil { - lastError = r.LastError().Error() + if len(ruleNamesSet) > 0 { + if _, ok := ruleNamesSet[rr.Name()]; !ok { + continue + } } - switch rule := r.(type) { + + lastError := "" + if rr.LastError() != nil { + lastError = rr.LastError().Error() + } + switch rule := rr.(type) { case *rules.AlertingRule: if !returnAlerts { break @@ -1358,11 +1374,16 @@ func (api *API) rules(r *http.Request) apiFuncResult { err := errors.Errorf("failed to assert type of rule '%v'", rule.Name()) return apiFuncResult{nil, &apiError{errorInternal, err}, nil, nil} } + if enrichedRule != nil { apiRuleGroup.Rules = append(apiRuleGroup.Rules, enrichedRule) } } - res.RuleGroups[i] = apiRuleGroup + + // If the rule group response has no rules, skip it - this means we filtered all the rules of this group. + if len(apiRuleGroup.Rules) > 0 { + res.RuleGroups[i] = apiRuleGroup + } } return apiFuncResult{res, nil, nil, nil} } diff --git a/web/api/v1/api_test.go b/web/api/v1/api_test.go index efce042214..e354bf298f 100644 --- a/web/api/v1/api_test.go +++ b/web/api/v1/api_test.go @@ -1973,6 +1973,33 @@ func testEndpoints(t *testing.T, api *API, tr *testTargetRetriever, es storage.E }, }, }, + { + endpoint: api.rules, + query: url.Values{"rules": []string{"test_metric4"}}, + response: &RuleDiscovery{ + RuleGroups: []*RuleGroup{ + { + Name: "grp", + File: "/path/to/file", + Interval: 1, + Limit: 0, + Rules: []Rule{ + AlertingRule{ + State: "inactive", + Name: "test_metric4", + Query: "up == 1", + Duration: 1, + Labels: labels.Labels{}, + Annotations: labels.Labels{}, + Alerts: []*Alert{}, + Health: "unknown", + Type: "alerting", + }, + }, + }, + }, + }, + }, { endpoint: api.queryExemplars, query: url.Values{ From b02811233170bc40fd43fc8a65313f0b64ee6191 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C4=90urica=20Yuri=20Nikoli=C4=87?= Date: Tue, 18 Apr 2023 12:13:05 +0200 Subject: [PATCH 098/231] Making the number of CPU cores used for sorting postings lists editable (#12247) Signed-off-by: Yuri Nikolic --- tsdb/head.go | 2 +- tsdb/index/postings.go | 14 ++++++++++---- tsdb/index/postings_test.go | 4 ++-- 3 files changed, 13 insertions(+), 7 deletions(-) diff --git a/tsdb/head.go b/tsdb/head.go index 4696884f21..1d65d32297 100644 --- a/tsdb/head.go +++ b/tsdb/head.go @@ -574,7 +574,7 @@ const cardinalityCacheExpirationTime = time.Duration(30) * time.Second func (h *Head) Init(minValidTime int64) error { h.minValidTime.Store(minValidTime) defer func() { - h.postings.EnsureOrder() + h.postings.EnsureOrder(h.opts.WALReplayConcurrency) }() defer h.gc() // After loading the wal remove the obsolete data from the head. defer func() { diff --git a/tsdb/index/postings.go b/tsdb/index/postings.go index b55d70df03..15df374fc5 100644 --- a/tsdb/index/postings.go +++ b/tsdb/index/postings.go @@ -224,7 +224,10 @@ func (p *MemPostings) All() Postings { // EnsureOrder ensures that all postings lists are sorted. After it returns all further // calls to add and addFor will insert new IDs in a sorted manner. -func (p *MemPostings) EnsureOrder() { +// Parameter numberOfConcurrentProcesses is used to specify the maximal number of +// CPU cores used for this operation. If it is <= 0, GOMAXPROCS is used. +// GOMAXPROCS was the default before introducing this parameter. +func (p *MemPostings) EnsureOrder(numberOfConcurrentProcesses int) { p.mtx.Lock() defer p.mtx.Unlock() @@ -232,13 +235,16 @@ func (p *MemPostings) EnsureOrder() { return } - n := runtime.GOMAXPROCS(0) + concurrency := numberOfConcurrentProcesses + if concurrency <= 0 { + concurrency = runtime.GOMAXPROCS(0) + } workc := make(chan *[][]storage.SeriesRef) var wg sync.WaitGroup - wg.Add(n) + wg.Add(concurrency) - for i := 0; i < n; i++ { + for i := 0; i < concurrency; i++ { go func() { for job := range workc { for _, l := range *job { diff --git a/tsdb/index/postings_test.go b/tsdb/index/postings_test.go index 1b1ecd3c39..a34f3c12df 100644 --- a/tsdb/index/postings_test.go +++ b/tsdb/index/postings_test.go @@ -54,7 +54,7 @@ func TestMemPostings_ensureOrder(t *testing.T) { p.m["a"][v] = l } - p.EnsureOrder() + p.EnsureOrder(0) for _, e := range p.m { for _, l := range e { @@ -114,7 +114,7 @@ func BenchmarkMemPostings_ensureOrder(b *testing.B) { b.ResetTimer() for n := 0; n < b.N; n++ { - p.EnsureOrder() + p.EnsureOrder(0) p.ordered = false } }) From 96b6463f2587f8c3da1031bfd5bc6b9aca575733 Mon Sep 17 00:00:00 2001 From: gotjosh Date: Tue, 18 Apr 2023 16:26:21 +0100 Subject: [PATCH 099/231] review comments Signed-off-by: gotjosh --- web/api/v1/api.go | 42 ++++++++++++++++++++++++++++++------------ web/api/v1/api_test.go | 34 +++++++++++++++++++++++++++++++++- 2 files changed, 63 insertions(+), 13 deletions(-) diff --git a/web/api/v1/api.go b/web/api/v1/api.go index d955958041..9a13e09d98 100644 --- a/web/api/v1/api.go +++ b/web/api/v1/api.go @@ -1292,20 +1292,26 @@ type RecordingRule struct { } func (api *API) rules(r *http.Request) apiFuncResult { + if err := r.ParseForm(); err != nil { + return apiFuncResult{nil, &apiError{errorBadData, errors.Wrapf(err, "error parsing form values")}, nil, nil} + } + + queryFormToSet := func(values []string) map[string]struct{} { + set := make(map[string]struct{}, len(values)) + for _, v := range values { + set[v] = struct{}{} + } + return set + } + + rnSet := queryFormToSet(r.Form["rule_name[]"]) + rgSet := queryFormToSet(r.Form["rule_group[]"]) + fSet := queryFormToSet(r.Form["file[]"]) + ruleGroups := api.rulesRetriever(r.Context()).RuleGroups() res := &RuleDiscovery{RuleGroups: make([]*RuleGroup, len(ruleGroups))} typ := strings.ToLower(r.URL.Query().Get("type")) - // Parse the rule names into a comma separated list of rule names, then create a set. - rulesQuery := r.URL.Query().Get("rules") - ruleNamesSet := map[string]struct{}{} - if rulesQuery != "" { - names := strings.Split(rulesQuery, ",") - for _, rn := range names { - ruleNamesSet[strings.TrimSpace(rn)] = struct{}{} - } - } - if typ != "" && typ != "alert" && typ != "record" { return invalidParamError(errors.Errorf("not supported value %q", typ), "type") } @@ -1314,6 +1320,18 @@ func (api *API) rules(r *http.Request) apiFuncResult { returnRecording := typ == "" || typ == "record" for i, grp := range ruleGroups { + if len(rgSet) > 0 { + if _, ok := rgSet[grp.Name()]; !ok { + continue + } + } + + if len(fSet) > 0 { + if _, ok := fSet[grp.File()]; !ok { + continue + } + } + apiRuleGroup := &RuleGroup{ Name: grp.Name(), File: grp.File(), @@ -1326,8 +1344,8 @@ func (api *API) rules(r *http.Request) apiFuncResult { for _, rr := range grp.Rules() { var enrichedRule Rule - if len(ruleNamesSet) > 0 { - if _, ok := ruleNamesSet[rr.Name()]; !ok { + if len(rnSet) > 0 { + if _, ok := rnSet[rr.Name()]; !ok { continue } } diff --git a/web/api/v1/api_test.go b/web/api/v1/api_test.go index e354bf298f..c3e1bf59d7 100644 --- a/web/api/v1/api_test.go +++ b/web/api/v1/api_test.go @@ -1975,7 +1975,39 @@ func testEndpoints(t *testing.T, api *API, tr *testTargetRetriever, es storage.E }, { endpoint: api.rules, - query: url.Values{"rules": []string{"test_metric4"}}, + query: url.Values{"rule_name[]": []string{"test_metric4"}}, + response: &RuleDiscovery{ + RuleGroups: []*RuleGroup{ + { + Name: "grp", + File: "/path/to/file", + Interval: 1, + Limit: 0, + Rules: []Rule{ + AlertingRule{ + State: "inactive", + Name: "test_metric4", + Query: "up == 1", + Duration: 1, + Labels: labels.Labels{}, + Annotations: labels.Labels{}, + Alerts: []*Alert{}, + Health: "unknown", + Type: "alerting", + }, + }, + }, + }, + }, + }, + { + endpoint: api.rules, + query: url.Values{"rule_group[]": []string{"respond-with-nothing"}}, + response: &RuleDiscovery{RuleGroups: []*RuleGroup{nil}}, + }, + { + endpoint: api.rules, + query: url.Values{"file[]": []string{"/path/to/file"}, "rule_name[]": []string{"test_metric4"}}, response: &RuleDiscovery{ RuleGroups: []*RuleGroup{ { From e2a2790b2c830e903e28562d667b0a03adb3beeb Mon Sep 17 00:00:00 2001 From: gotjosh Date: Tue, 18 Apr 2023 16:50:16 +0100 Subject: [PATCH 100/231] add more docs Signed-off-by: gotjosh --- docs/querying/api.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/querying/api.md b/docs/querying/api.md index 0cc549b65e..bc0587dd0c 100644 --- a/docs/querying/api.md +++ b/docs/querying/api.md @@ -675,7 +675,9 @@ GET /api/v1/rules URL query parameters: - `type=alert|record`: return only the alerting rules (e.g. `type=alert`) or the recording rules (e.g. `type=record`). When the parameter is absent or empty, no filtering is done. -- `rules=alertName,RuleName`: return only the alerting and recording rules with the specified names. If we've filtered out all the rules of a group, the group is not returned. When the parameter is absent or empty, no filtering is done. +- `rule_name[]=`: return the groups and rules of the specified alerting and recording rules names, the parameter supports repetition. If we've filtered out all the rules of a group, the group is not returned. When the parameter is absent or empty, no filtering is done. +- `rule_group[]=`: return the group and rules of the specified rule group names, the parameter supports repetitions. When the parameter is absent or empty, no filtering is done. +- `file[]=`: return the group and rules of specified filepath for rule groups, the parameter supports repetition. When the parameter is absent or empty, no filtering is done. ```json $ curl http://localhost:9090/api/v1/rules From 28909a46362737c218039dbf952d13e200b31c1f Mon Sep 17 00:00:00 2001 From: gotjosh Date: Tue, 18 Apr 2023 16:51:35 +0100 Subject: [PATCH 101/231] more worthsmithing Signed-off-by: gotjosh --- docs/querying/api.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/querying/api.md b/docs/querying/api.md index bc0587dd0c..c6d8d2c836 100644 --- a/docs/querying/api.md +++ b/docs/querying/api.md @@ -675,9 +675,9 @@ GET /api/v1/rules URL query parameters: - `type=alert|record`: return only the alerting rules (e.g. `type=alert`) or the recording rules (e.g. `type=record`). When the parameter is absent or empty, no filtering is done. -- `rule_name[]=`: return the groups and rules of the specified alerting and recording rules names, the parameter supports repetition. If we've filtered out all the rules of a group, the group is not returned. When the parameter is absent or empty, no filtering is done. -- `rule_group[]=`: return the group and rules of the specified rule group names, the parameter supports repetitions. When the parameter is absent or empty, no filtering is done. -- `file[]=`: return the group and rules of specified filepath for rule groups, the parameter supports repetition. When the parameter is absent or empty, no filtering is done. +- `rule_name[]=`: return the groups and its rules of the specified alerting and recording rules names, the parameter supports repetition. If we've filtered out all the rules of a group, the group is not returned. When the parameter is absent or empty, no filtering is done. +- `rule_group[]=`: return the groups and its rules of the specified rule group names, the parameter supports repetitions. When the parameter is absent or empty, no filtering is done. +- `file[]=`: return the groups and its rules of specified filepath for rule groups, the parameter supports repetition. When the parameter is absent or empty, no filtering is done. ```json $ curl http://localhost:9090/api/v1/rules From cf230bcd18bbbb429a46985049b049abb3437140 Mon Sep 17 00:00:00 2001 From: gotjosh Date: Wed, 19 Apr 2023 09:49:34 +0100 Subject: [PATCH 102/231] more wordsmithing Signed-off-by: gotjosh --- docs/querying/api.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/querying/api.md b/docs/querying/api.md index c6d8d2c836..820414fb13 100644 --- a/docs/querying/api.md +++ b/docs/querying/api.md @@ -675,9 +675,9 @@ GET /api/v1/rules URL query parameters: - `type=alert|record`: return only the alerting rules (e.g. `type=alert`) or the recording rules (e.g. `type=record`). When the parameter is absent or empty, no filtering is done. -- `rule_name[]=`: return the groups and its rules of the specified alerting and recording rules names, the parameter supports repetition. If we've filtered out all the rules of a group, the group is not returned. When the parameter is absent or empty, no filtering is done. -- `rule_group[]=`: return the groups and its rules of the specified rule group names, the parameter supports repetitions. When the parameter is absent or empty, no filtering is done. -- `file[]=`: return the groups and its rules of specified filepath for rule groups, the parameter supports repetition. When the parameter is absent or empty, no filtering is done. +- `rule_name[]=`: only return rules with the given rule name. If the parameter is repeated, rules with any of provided names are returned. If we've filtered out all the rules of a group, the group is not returned. When the parameter is absent or empty, no filtering is done. +- `rule_group[]=`: only return rules with the given rule group name. If the parameter is repeated, rules with any of provided rule group names are returned. When the parameter is absent or empty, no filtering is done. +- `file[]=`: only return rules with the given filepath. If the parameter is repeated, rules with any of provided filepaths are returned. When the parameter is absent or empty, no filtering is done. ```json $ curl http://localhost:9090/api/v1/rules From c3c7d44d845e08969845133a8c64c631ddbbb437 Mon Sep 17 00:00:00 2001 From: beorn7 Date: Wed, 12 Apr 2023 13:05:41 +0200 Subject: [PATCH 103/231] lint: Adjust to the lint warnings raised by current versions of golint-ci MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit We haven't updated golint-ci in our CI yet, but this commit prepares for that. There are a lot of new warnings, and it is mostly because the "revive" linter got updated. I agree with most of the new warnings, mostly around not naming unused function parameters (although it is justified in some cases for documentation purposes – while things like mocks are a good example where not naming the parameter is clearer). I'm pretty upset about the "empty block" warning to include `for` loops. It's such a common pattern to do something in the head of the `for` loop and then have an empty block. There is still an open issue about this: https://github.com/mgechev/revive/issues/810 I have disabled "revive" altogether in files where empty blocks are used excessively, and I have made the effort to add individual `// nolint:revive` where empty blocks are used just once or twice. It's borderline noisy, though, but let's go with it for now. I should mention that none of the "empty block" warnings for `for` loop bodies were legitimate. Signed-off-by: beorn7 --- cmd/prometheus/main.go | 1 + cmd/promtool/backfill_test.go | 2 +- cmd/promtool/rules.go | 2 +- cmd/promtool/rules_test.go | 4 ++-- discovery/aws/ec2.go | 2 +- discovery/hetzner/hcloud.go | 2 +- discovery/hetzner/robot.go | 4 ++-- discovery/ionos/server.go | 2 +- discovery/kubernetes/client_metrics.go | 6 +++--- discovery/kubernetes/endpoints.go | 1 + discovery/kubernetes/endpointslice.go | 2 +- discovery/kubernetes/ingress.go | 2 +- discovery/kubernetes/node.go | 2 +- discovery/kubernetes/pod.go | 2 +- discovery/kubernetes/service.go | 2 +- discovery/legacymanager/manager_test.go | 13 +++---------- discovery/manager_test.go | 17 +++++------------ discovery/nomad/nomad.go | 2 +- discovery/ovhcloud/dedicated_server.go | 2 +- discovery/ovhcloud/vps.go | 2 +- discovery/vultr/vultr.go | 4 +--- discovery/zookeeper/zookeeper.go | 2 +- model/textparse/promparse.go | 7 ++++--- promql/bench_test.go | 2 +- promql/engine.go | 2 +- promql/functions.go | 1 + promql/parser/lex.go | 3 ++- promql/parser/parse.go | 6 +++--- rules/manager.go | 7 ++++--- rules/manager_test.go | 8 ++++---- rules/origin_test.go | 22 +++++++++++----------- scrape/scrape_test.go | 4 ++-- scrape/target_test.go | 2 +- storage/buffer_test.go | 4 ++-- storage/fanout_test.go | 2 +- storage/interface.go | 6 +++--- storage/memoized_iterator_test.go | 4 ++-- storage/merge.go | 11 +++++------ storage/remote/codec_test.go | 2 +- storage/remote/queue_manager_test.go | 10 +++++----- storage/remote/write_handler_test.go | 2 +- tsdb/agent/db.go | 16 ++++++++-------- tsdb/chunkenc/float_histogram.go | 2 +- tsdb/chunkenc/float_histogram_test.go | 2 +- tsdb/chunkenc/histogram.go | 2 +- tsdb/chunkenc/histogram_test.go | 2 +- tsdb/chunkenc/xor.go | 6 +++--- tsdb/chunks/head_chunks_test.go | 4 ++-- tsdb/compact_test.go | 4 ++-- tsdb/db_test.go | 6 +++--- tsdb/exemplar.go | 12 ++++++------ tsdb/goversion/goversion_test.go | 2 +- tsdb/head_test.go | 7 ++++--- tsdb/head_wal.go | 1 + tsdb/ooo_head_read.go | 1 + tsdb/querier.go | 2 +- tsdb/querier_bench_test.go | 2 +- tsdb/querier_test.go | 1 + tsdb/wal.go | 1 + tsdb/wlog/reader_test.go | 2 +- tsdb/wlog/wlog_test.go | 2 +- util/logging/dedupe_test.go | 2 +- util/testutil/context.go | 2 +- util/testutil/roundtrip.go | 2 +- util/treecache/treecache.go | 2 +- web/api/v1/api.go | 14 +++++++------- web/api/v1/api_test.go | 6 +++--- web/api/v1/errors_test.go | 1 + web/web.go | 6 +++--- 69 files changed, 145 insertions(+), 150 deletions(-) diff --git a/cmd/prometheus/main.go b/cmd/prometheus/main.go index f4f6af20df..5592036466 100644 --- a/cmd/prometheus/main.go +++ b/cmd/prometheus/main.go @@ -12,6 +12,7 @@ // limitations under the License. // The main package for the Prometheus server executable. +// nolint:revive // Many unsued function arguments in this file by design. package main import ( diff --git a/cmd/promtool/backfill_test.go b/cmd/promtool/backfill_test.go index 2c551abeb3..e6f7cad31b 100644 --- a/cmd/promtool/backfill_test.go +++ b/cmd/promtool/backfill_test.go @@ -44,7 +44,7 @@ func sortSamples(samples []backfillSample) { }) } -func queryAllSeries(t testing.TB, q storage.Querier, expectedMinTime, expectedMaxTime int64) []backfillSample { +func queryAllSeries(t testing.TB, q storage.Querier, expectedMinTime, expectedMaxTime int64) []backfillSample { // nolint:revive ss := q.Select(false, nil, labels.MustNewMatcher(labels.MatchRegexp, "", ".*")) samples := []backfillSample{} for ss.Next() { diff --git a/cmd/promtool/rules.go b/cmd/promtool/rules.go index aedc7bcb9d..43b76dbe4a 100644 --- a/cmd/promtool/rules.go +++ b/cmd/promtool/rules.go @@ -68,7 +68,7 @@ func newRuleImporter(logger log.Logger, config ruleImporterConfig, apiClient que } // loadGroups parses groups from a list of recording rule files. -func (importer *ruleImporter) loadGroups(ctx context.Context, filenames []string) (errs []error) { +func (importer *ruleImporter) loadGroups(_ context.Context, filenames []string) (errs []error) { groups, errs := importer.ruleManager.LoadGroups(importer.config.evalInterval, labels.Labels{}, "", nil, filenames...) if errs != nil { return errs diff --git a/cmd/promtool/rules_test.go b/cmd/promtool/rules_test.go index fb582ed0dd..213b7d2a01 100644 --- a/cmd/promtool/rules_test.go +++ b/cmd/promtool/rules_test.go @@ -35,7 +35,7 @@ type mockQueryRangeAPI struct { samples model.Matrix } -func (mockAPI mockQueryRangeAPI) QueryRange(ctx context.Context, query string, r v1.Range, opts ...v1.Option) (model.Value, v1.Warnings, error) { +func (mockAPI mockQueryRangeAPI) QueryRange(_ context.Context, query string, r v1.Range, opts ...v1.Option) (model.Value, v1.Warnings, error) { // nolint:revive return mockAPI.samples, v1.Warnings{}, nil } @@ -161,7 +161,7 @@ func TestBackfillRuleIntegration(t *testing.T) { } } -func newTestRuleImporter(ctx context.Context, start time.Time, tmpDir string, testSamples model.Matrix, maxBlockDuration time.Duration) (*ruleImporter, error) { +func newTestRuleImporter(_ context.Context, start time.Time, tmpDir string, testSamples model.Matrix, maxBlockDuration time.Duration) (*ruleImporter, error) { logger := log.NewNopLogger() cfg := ruleImporterConfig{ outputDir: tmpDir, diff --git a/discovery/aws/ec2.go b/discovery/aws/ec2.go index ca9921159d..86d76627e1 100644 --- a/discovery/aws/ec2.go +++ b/discovery/aws/ec2.go @@ -164,7 +164,7 @@ func NewEC2Discovery(conf *EC2SDConfig, logger log.Logger) *EC2Discovery { return d } -func (d *EC2Discovery) ec2Client(ctx context.Context) (*ec2.EC2, error) { +func (d *EC2Discovery) ec2Client(context.Context) (*ec2.EC2, error) { if d.ec2 != nil { return d.ec2, nil } diff --git a/discovery/hetzner/hcloud.go b/discovery/hetzner/hcloud.go index aa406a1a7a..50afdc1ec3 100644 --- a/discovery/hetzner/hcloud.go +++ b/discovery/hetzner/hcloud.go @@ -59,7 +59,7 @@ type hcloudDiscovery struct { } // newHcloudDiscovery returns a new hcloudDiscovery which periodically refreshes its targets. -func newHcloudDiscovery(conf *SDConfig, logger log.Logger) (*hcloudDiscovery, error) { +func newHcloudDiscovery(conf *SDConfig, _ log.Logger) (*hcloudDiscovery, error) { d := &hcloudDiscovery{ port: conf.Port, } diff --git a/discovery/hetzner/robot.go b/discovery/hetzner/robot.go index 4b7abaf77f..4960880289 100644 --- a/discovery/hetzner/robot.go +++ b/discovery/hetzner/robot.go @@ -51,7 +51,7 @@ type robotDiscovery struct { } // newRobotDiscovery returns a new robotDiscovery which periodically refreshes its targets. -func newRobotDiscovery(conf *SDConfig, logger log.Logger) (*robotDiscovery, error) { +func newRobotDiscovery(conf *SDConfig, _ log.Logger) (*robotDiscovery, error) { d := &robotDiscovery{ port: conf.Port, endpoint: conf.robotEndpoint, @@ -69,7 +69,7 @@ func newRobotDiscovery(conf *SDConfig, logger log.Logger) (*robotDiscovery, erro return d, nil } -func (d *robotDiscovery) refresh(ctx context.Context) ([]*targetgroup.Group, error) { +func (d *robotDiscovery) refresh(context.Context) ([]*targetgroup.Group, error) { req, err := http.NewRequest("GET", d.endpoint+"/server", nil) if err != nil { return nil, err diff --git a/discovery/ionos/server.go b/discovery/ionos/server.go index 8ac3639705..a850fbbfb4 100644 --- a/discovery/ionos/server.go +++ b/discovery/ionos/server.go @@ -60,7 +60,7 @@ type serverDiscovery struct { datacenterID string } -func newServerDiscovery(conf *SDConfig, logger log.Logger) (*serverDiscovery, error) { +func newServerDiscovery(conf *SDConfig, _ log.Logger) (*serverDiscovery, error) { d := &serverDiscovery{ port: conf.Port, datacenterID: conf.DatacenterID, diff --git a/discovery/kubernetes/client_metrics.go b/discovery/kubernetes/client_metrics.go index 3a33e3e8d5..b316f7d885 100644 --- a/discovery/kubernetes/client_metrics.go +++ b/discovery/kubernetes/client_metrics.go @@ -122,11 +122,11 @@ func (f *clientGoRequestMetricAdapter) Register(registerer prometheus.Registerer ) } -func (clientGoRequestMetricAdapter) Increment(ctx context.Context, code, method, host string) { +func (clientGoRequestMetricAdapter) Increment(_ context.Context, code, _, _ string) { clientGoRequestResultMetricVec.WithLabelValues(code).Inc() } -func (clientGoRequestMetricAdapter) Observe(ctx context.Context, verb string, u url.URL, latency time.Duration) { +func (clientGoRequestMetricAdapter) Observe(_ context.Context, _ string, u url.URL, latency time.Duration) { clientGoRequestLatencyMetricVec.WithLabelValues(u.EscapedPath()).Observe(latency.Seconds()) } @@ -169,7 +169,7 @@ func (f *clientGoWorkqueueMetricsProvider) NewLongestRunningProcessorSecondsMetr return clientGoWorkqueueLongestRunningProcessorMetricVec.WithLabelValues(name) } -func (clientGoWorkqueueMetricsProvider) NewRetriesMetric(name string) workqueue.CounterMetric { +func (clientGoWorkqueueMetricsProvider) NewRetriesMetric(string) workqueue.CounterMetric { // Retries are not used so the metric is omitted. return noopMetric{} } diff --git a/discovery/kubernetes/endpoints.go b/discovery/kubernetes/endpoints.go index 039daf4faf..27742ab464 100644 --- a/discovery/kubernetes/endpoints.go +++ b/discovery/kubernetes/endpoints.go @@ -11,6 +11,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +// nolint:revive // Many legitimately empty blocks in this file. package kubernetes import ( diff --git a/discovery/kubernetes/endpointslice.go b/discovery/kubernetes/endpointslice.go index 135735154c..29dc0be2f6 100644 --- a/discovery/kubernetes/endpointslice.go +++ b/discovery/kubernetes/endpointslice.go @@ -190,7 +190,7 @@ func (e *EndpointSlice) Run(ctx context.Context, ch chan<- []*targetgroup.Group) } go func() { - for e.process(ctx, ch) { + for e.process(ctx, ch) { // nolint:revive } }() diff --git a/discovery/kubernetes/ingress.go b/discovery/kubernetes/ingress.go index 8c9249f545..ad47c341a5 100644 --- a/discovery/kubernetes/ingress.go +++ b/discovery/kubernetes/ingress.go @@ -89,7 +89,7 @@ func (i *Ingress) Run(ctx context.Context, ch chan<- []*targetgroup.Group) { } go func() { - for i.process(ctx, ch) { + for i.process(ctx, ch) { // nolint:revive } }() diff --git a/discovery/kubernetes/node.go b/discovery/kubernetes/node.go index 93adf78252..16a06e7a0c 100644 --- a/discovery/kubernetes/node.go +++ b/discovery/kubernetes/node.go @@ -96,7 +96,7 @@ func (n *Node) Run(ctx context.Context, ch chan<- []*targetgroup.Group) { } go func() { - for n.process(ctx, ch) { + for n.process(ctx, ch) { // nolint:revive } }() diff --git a/discovery/kubernetes/pod.go b/discovery/kubernetes/pod.go index 396720c223..732cf52ad9 100644 --- a/discovery/kubernetes/pod.go +++ b/discovery/kubernetes/pod.go @@ -132,7 +132,7 @@ func (p *Pod) Run(ctx context.Context, ch chan<- []*targetgroup.Group) { } go func() { - for p.process(ctx, ch) { + for p.process(ctx, ch) { // nolint:revive } }() diff --git a/discovery/kubernetes/service.go b/discovery/kubernetes/service.go index a19f06e7d1..40e17679ee 100644 --- a/discovery/kubernetes/service.go +++ b/discovery/kubernetes/service.go @@ -92,7 +92,7 @@ func (s *Service) Run(ctx context.Context, ch chan<- []*targetgroup.Group) { } go func() { - for s.process(ctx, ch) { + for s.process(ctx, ch) { // nolint:revive } }() diff --git a/discovery/legacymanager/manager_test.go b/discovery/legacymanager/manager_test.go index 57c82b72a8..13b84e6e36 100644 --- a/discovery/legacymanager/manager_test.go +++ b/discovery/legacymanager/manager_test.go @@ -686,12 +686,7 @@ func TestTargetUpdatesOrder(t *testing.T) { case tgs := <-provUpdates: discoveryManager.updateGroup(poolKey{setName: strconv.Itoa(i), provider: tc.title}, tgs) for _, got := range discoveryManager.allGroups() { - assertEqualGroups(t, got, tc.expectedTargets[x], func(got, expected string) string { - return fmt.Sprintf("%d: \ntargets mismatch \ngot: %v \nexpected: %v", - x, - got, - expected) - }) + assertEqualGroups(t, got, tc.expectedTargets[x]) } } } @@ -699,7 +694,7 @@ func TestTargetUpdatesOrder(t *testing.T) { } } -func assertEqualGroups(t *testing.T, got, expected []*targetgroup.Group, msg func(got, expected string) string) { +func assertEqualGroups(t *testing.T, got, expected []*targetgroup.Group) { t.Helper() // Need to sort by the groups's source as the received order is not guaranteed. @@ -1079,9 +1074,7 @@ func TestCoordinationWithReceiver(t *testing.T) { if _, ok := tgs[k]; !ok { t.Fatalf("step %d: target group not found: %s\ngot: %#v", i, k, tgs) } - assertEqualGroups(t, tgs[k], expected.tgs[k], func(got, expected string) string { - return fmt.Sprintf("step %d: targets mismatch \ngot: %q \nexpected: %q", i, got, expected) - }) + assertEqualGroups(t, tgs[k], expected.tgs[k]) } } } diff --git a/discovery/manager_test.go b/discovery/manager_test.go index 970168b0f5..5371608112 100644 --- a/discovery/manager_test.go +++ b/discovery/manager_test.go @@ -686,12 +686,7 @@ func TestTargetUpdatesOrder(t *testing.T) { case tgs := <-provUpdates: discoveryManager.updateGroup(poolKey{setName: strconv.Itoa(i), provider: tc.title}, tgs) for _, got := range discoveryManager.allGroups() { - assertEqualGroups(t, got, tc.expectedTargets[x], func(got, expected string) string { - return fmt.Sprintf("%d: \ntargets mismatch \ngot: %v \nexpected: %v", - x, - got, - expected) - }) + assertEqualGroups(t, got, tc.expectedTargets[x]) } } } @@ -699,7 +694,7 @@ func TestTargetUpdatesOrder(t *testing.T) { } } -func assertEqualGroups(t *testing.T, got, expected []*targetgroup.Group, msg func(got, expected string) string) { +func assertEqualGroups(t *testing.T, got, expected []*targetgroup.Group) { t.Helper() // Need to sort by the groups's source as the received order is not guaranteed. @@ -1129,7 +1124,7 @@ type lockStaticConfig struct { } func (s lockStaticConfig) Name() string { return "lockstatic" } -func (s lockStaticConfig) NewDiscoverer(options DiscovererOptions) (Discoverer, error) { +func (s lockStaticConfig) NewDiscoverer(DiscovererOptions) (Discoverer, error) { return (lockStaticDiscoverer)(s), nil } @@ -1330,9 +1325,7 @@ func TestCoordinationWithReceiver(t *testing.T) { if _, ok := tgs[k]; !ok { t.Fatalf("step %d: target group not found: %s\ngot: %#v", i, k, tgs) } - assertEqualGroups(t, tgs[k], expected.tgs[k], func(got, expected string) string { - return fmt.Sprintf("step %d: targets mismatch \ngot: %q \nexpected: %q", i, got, expected) - }) + assertEqualGroups(t, tgs[k], expected.tgs[k]) } } } @@ -1399,7 +1392,7 @@ func (o onceProvider) Run(_ context.Context, ch chan<- []*targetgroup.Group) { // TestTargetSetTargetGroupsUpdateDuringApplyConfig is used to detect races when // ApplyConfig happens at the same time as targets update. -func TestTargetSetTargetGroupsUpdateDuringApplyConfig(t *testing.T) { +func TestTargetSetTargetGroupsUpdateDuringApplyConfig(*testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() discoveryManager := NewManager(ctx, log.NewNopLogger()) diff --git a/discovery/nomad/nomad.go b/discovery/nomad/nomad.go index c8d5130396..7013f0737c 100644 --- a/discovery/nomad/nomad.go +++ b/discovery/nomad/nomad.go @@ -161,7 +161,7 @@ func NewDiscovery(conf *SDConfig, logger log.Logger) (*Discovery, error) { return d, nil } -func (d *Discovery) refresh(ctx context.Context) ([]*targetgroup.Group, error) { +func (d *Discovery) refresh(context.Context) ([]*targetgroup.Group, error) { opts := &nomad.QueryOptions{ AllowStale: d.allowStale, } diff --git a/discovery/ovhcloud/dedicated_server.go b/discovery/ovhcloud/dedicated_server.go index aeb4eccbbf..bb5dadcd7b 100644 --- a/discovery/ovhcloud/dedicated_server.go +++ b/discovery/ovhcloud/dedicated_server.go @@ -102,7 +102,7 @@ func (d *dedicatedServerDiscovery) getSource() string { return fmt.Sprintf("%s_%s", d.config.Name(), d.getService()) } -func (d *dedicatedServerDiscovery) refresh(ctx context.Context) ([]*targetgroup.Group, error) { +func (d *dedicatedServerDiscovery) refresh(context.Context) ([]*targetgroup.Group, error) { client, err := createClient(d.config) if err != nil { return nil, err diff --git a/discovery/ovhcloud/vps.go b/discovery/ovhcloud/vps.go index 705b42b655..e2d1dee364 100644 --- a/discovery/ovhcloud/vps.go +++ b/discovery/ovhcloud/vps.go @@ -117,7 +117,7 @@ func (d *vpsDiscovery) getSource() string { return fmt.Sprintf("%s_%s", d.config.Name(), d.getService()) } -func (d *vpsDiscovery) refresh(ctx context.Context) ([]*targetgroup.Group, error) { +func (d *vpsDiscovery) refresh(context.Context) ([]*targetgroup.Group, error) { client, err := createClient(d.config) if err != nil { return nil, err diff --git a/discovery/vultr/vultr.go b/discovery/vultr/vultr.go index 2f489e7d45..42881d3c19 100644 --- a/discovery/vultr/vultr.go +++ b/discovery/vultr/vultr.go @@ -202,10 +202,8 @@ func (d *Discovery) listInstances(ctx context.Context) ([]govultr.Instance, erro if meta.Links.Next == "" { break - } else { - listOptions.Cursor = meta.Links.Next - continue } + listOptions.Cursor = meta.Links.Next } return instances, nil diff --git a/discovery/zookeeper/zookeeper.go b/discovery/zookeeper/zookeeper.go index 308d63a5fc..cadff5fd2e 100644 --- a/discovery/zookeeper/zookeeper.go +++ b/discovery/zookeeper/zookeeper.go @@ -193,7 +193,7 @@ func (d *Discovery) Run(ctx context.Context, ch chan<- []*targetgroup.Group) { } for _, pathUpdate := range d.pathUpdates { // Drain event channel in case the treecache leaks goroutines otherwise. - for range pathUpdate { + for range pathUpdate { // nolint:revive } } d.conn.Close() diff --git a/model/textparse/promparse.go b/model/textparse/promparse.go index 2c981f050e..94338a6660 100644 --- a/model/textparse/promparse.go +++ b/model/textparse/promparse.go @@ -238,9 +238,10 @@ func (p *PromParser) Metric(l *labels.Labels) string { return s } -// Exemplar writes the exemplar of the current sample into the passed -// exemplar. It returns if an exemplar exists. -func (p *PromParser) Exemplar(e *exemplar.Exemplar) bool { +// Exemplar implements the Parser interface. However, since the classic +// Prometheus text format does not support exemplars, this implementation simply +// returns false and does nothing else. +func (p *PromParser) Exemplar(*exemplar.Exemplar) bool { return false } diff --git a/promql/bench_test.go b/promql/bench_test.go index 3d35884477..4ece16bfe4 100644 --- a/promql/bench_test.go +++ b/promql/bench_test.go @@ -27,7 +27,7 @@ import ( "github.com/prometheus/prometheus/util/teststorage" ) -func setupRangeQueryTestData(stor *teststorage.TestStorage, engine *Engine, interval, numIntervals int) error { +func setupRangeQueryTestData(stor *teststorage.TestStorage, _ *Engine, interval, numIntervals int) error { metrics := []labels.Labels{} metrics = append(metrics, labels.FromStrings("__name__", "a_one")) metrics = append(metrics, labels.FromStrings("__name__", "b_one")) diff --git a/promql/engine.go b/promql/engine.go index fae4f09320..21b894b936 100644 --- a/promql/engine.go +++ b/promql/engine.go @@ -1957,7 +1957,7 @@ func (ev *evaluator) matrixIterSlice( // (b) the number of samples is relatively small. // so a linear search will be as fast as a binary search. var drop int - for drop = 0; floats[drop].T < mint; drop++ { + for drop = 0; floats[drop].T < mint; drop++ { // nolint:revive } ev.currentSamples -= drop copy(floats, floats[drop:]) diff --git a/promql/functions.go b/promql/functions.go index fd99703df2..0c22cb44c7 100644 --- a/promql/functions.go +++ b/promql/functions.go @@ -11,6 +11,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +// nolint:revive // Many unsued function arguments in this file by design. package promql import ( diff --git a/promql/parser/lex.go b/promql/parser/lex.go index 7b6a6b027b..657dc28095 100644 --- a/promql/parser/lex.go +++ b/promql/parser/lex.go @@ -11,6 +11,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +// nolint:revive // Many legitimately empty blocks in this file. package parser import ( @@ -293,7 +294,7 @@ func (l *Lexer) accept(valid string) bool { // acceptRun consumes a run of runes from the valid set. func (l *Lexer) acceptRun(valid string) { for strings.ContainsRune(valid, l.next()) { - // consume + // Consume. } l.backup() } diff --git a/promql/parser/parse.go b/promql/parser/parse.go index 6c37ce6fc6..3c32f3c05b 100644 --- a/promql/parser/parse.go +++ b/promql/parser/parse.go @@ -332,7 +332,7 @@ func (p *parser) Lex(lval *yySymType) int { // It is a no-op since the parsers error routines are triggered // by mechanisms that allow more fine-grained control // For more information, see https://pkg.go.dev/golang.org/x/tools/cmd/goyacc. -func (p *parser) Error(e string) { +func (p *parser) Error(string) { } // InjectItem allows injecting a single Item at the beginning of the token stream @@ -481,9 +481,9 @@ func (p *parser) checkAST(node Node) (typ ValueType) { // This is made a function instead of a variable, so it is lazily evaluated on demand. opRange := func() (r PositionRange) { // Remove whitespace at the beginning and end of the range. - for r.Start = n.LHS.PositionRange().End; isSpace(rune(p.lex.input[r.Start])); r.Start++ { + for r.Start = n.LHS.PositionRange().End; isSpace(rune(p.lex.input[r.Start])); r.Start++ { // nolint:revive } - for r.End = n.RHS.PositionRange().Start - 1; isSpace(rune(p.lex.input[r.End])); r.End-- { + for r.End = n.RHS.PositionRange().Start - 1; isSpace(rune(p.lex.input[r.End])); r.End-- { // nolint:revive } return } diff --git a/rules/manager.go b/rules/manager.go index b6513e82d4..31c90e9e9d 100644 --- a/rules/manager.go +++ b/rules/manager.go @@ -866,12 +866,13 @@ func (g *Group) RestoreForState(ts time.Time) { timeSpentPending := downAt.Sub(restoredActiveAt) timeRemainingPending := alertHoldDuration - timeSpentPending - if timeRemainingPending <= 0 { + switch { + case timeRemainingPending <= 0: // It means that alert was firing when prometheus went down. // In the next Eval, the state of this alert will be set back to // firing again if it's still firing in that Eval. // Nothing to be done in this case. - } else if timeRemainingPending < g.opts.ForGracePeriod { + case timeRemainingPending < g.opts.ForGracePeriod: // (new) restoredActiveAt = (ts + m.opts.ForGracePeriod) - alertHoldDuration // /* new firing time */ /* moving back by hold duration */ // @@ -884,7 +885,7 @@ func (g *Group) RestoreForState(ts time.Time) { // = (ts + m.opts.ForGracePeriod) - ts // = m.opts.ForGracePeriod restoredActiveAt = ts.Add(g.opts.ForGracePeriod).Add(-alertHoldDuration) - } else { + default: // By shifting ActiveAt to the future (ActiveAt + some_duration), // the total pending time from the original ActiveAt // would be `alertHoldDuration + some_duration`. diff --git a/rules/manager_test.go b/rules/manager_test.go index 440e06c9af..16bb080f55 100644 --- a/rules/manager_test.go +++ b/rules/manager_test.go @@ -779,13 +779,13 @@ func TestUpdate(t *testing.T) { rgs.Groups[i].Interval = model.Duration(10) } } - reloadAndValidate(rgs, t, tmpFile, ruleManager, expected, ogs) + reloadAndValidate(rgs, t, tmpFile, ruleManager, ogs) // Update limit and reload. for i := range rgs.Groups { rgs.Groups[i].Limit = 1 } - reloadAndValidate(rgs, t, tmpFile, ruleManager, expected, ogs) + reloadAndValidate(rgs, t, tmpFile, ruleManager, ogs) // Change group rules and reload. for i, g := range rgs.Groups { @@ -793,7 +793,7 @@ func TestUpdate(t *testing.T) { rgs.Groups[i].Rules[j].Expr.SetString(fmt.Sprintf("%s * 0", r.Expr.Value)) } } - reloadAndValidate(rgs, t, tmpFile, ruleManager, expected, ogs) + reloadAndValidate(rgs, t, tmpFile, ruleManager, ogs) } // ruleGroupsTest for running tests over rules. @@ -836,7 +836,7 @@ func formatRules(r *rulefmt.RuleGroups) ruleGroupsTest { } } -func reloadAndValidate(rgs *rulefmt.RuleGroups, t *testing.T, tmpFile *os.File, ruleManager *Manager, expected map[string]labels.Labels, ogs map[string]*Group) { +func reloadAndValidate(rgs *rulefmt.RuleGroups, t *testing.T, tmpFile *os.File, ruleManager *Manager, ogs map[string]*Group) { bs, err := yaml.Marshal(formatRules(rgs)) require.NoError(t, err) tmpFile.Seek(0, 0) diff --git a/rules/origin_test.go b/rules/origin_test.go index dd8e47f74c..ea4f4f905d 100644 --- a/rules/origin_test.go +++ b/rules/origin_test.go @@ -30,19 +30,19 @@ type unknownRule struct{} func (u unknownRule) Name() string { return "" } func (u unknownRule) Labels() labels.Labels { return labels.EmptyLabels() } -func (u unknownRule) Eval(ctx context.Context, time time.Time, queryFunc QueryFunc, url *url.URL, i int) (promql.Vector, error) { +func (u unknownRule) Eval(context.Context, time.Time, QueryFunc, *url.URL, int) (promql.Vector, error) { return nil, nil } -func (u unknownRule) String() string { return "" } -func (u unknownRule) Query() parser.Expr { return nil } -func (u unknownRule) SetLastError(err error) {} -func (u unknownRule) LastError() error { return nil } -func (u unknownRule) SetHealth(health RuleHealth) {} -func (u unknownRule) Health() RuleHealth { return "" } -func (u unknownRule) SetEvaluationDuration(duration time.Duration) {} -func (u unknownRule) GetEvaluationDuration() time.Duration { return 0 } -func (u unknownRule) SetEvaluationTimestamp(time time.Time) {} -func (u unknownRule) GetEvaluationTimestamp() time.Time { return time.Time{} } +func (u unknownRule) String() string { return "" } +func (u unknownRule) Query() parser.Expr { return nil } +func (u unknownRule) SetLastError(error) {} +func (u unknownRule) LastError() error { return nil } +func (u unknownRule) SetHealth(RuleHealth) {} +func (u unknownRule) Health() RuleHealth { return "" } +func (u unknownRule) SetEvaluationDuration(time.Duration) {} +func (u unknownRule) GetEvaluationDuration() time.Duration { return 0 } +func (u unknownRule) SetEvaluationTimestamp(time.Time) {} +func (u unknownRule) GetEvaluationTimestamp() time.Time { return time.Time{} } func TestNewRuleDetailPanics(t *testing.T) { require.PanicsWithValue(t, `unknown rule type "rules.unknownRule"`, func() { diff --git a/scrape/scrape_test.go b/scrape/scrape_test.go index dcb3b48c10..a8028b652a 100644 --- a/scrape/scrape_test.go +++ b/scrape/scrape_test.go @@ -2405,7 +2405,7 @@ type testScraper struct { scrapeFunc func(context.Context, io.Writer) error } -func (ts *testScraper) offset(interval time.Duration, jitterSeed uint64) time.Duration { +func (ts *testScraper) offset(time.Duration, uint64) time.Duration { return ts.offsetDur } @@ -2867,7 +2867,7 @@ func TestScrapeAddFast(t *testing.T) { require.NoError(t, slApp.Commit()) } -func TestReuseCacheRace(t *testing.T) { +func TestReuseCacheRace(*testing.T) { var ( app = &nopAppendable{} cfg = &config.ScrapeConfig{ diff --git a/scrape/target_test.go b/scrape/target_test.go index 2bc3f000c5..9d25df4149 100644 --- a/scrape/target_test.go +++ b/scrape/target_test.go @@ -134,7 +134,7 @@ func TestTargetURL(t *testing.T) { require.Equal(t, expectedURL, target.URL()) } -func newTestTarget(targetURL string, deadline time.Duration, lbls labels.Labels) *Target { +func newTestTarget(targetURL string, _ time.Duration, lbls labels.Labels) *Target { lb := labels.NewBuilder(lbls) lb.Set(model.SchemeLabel, "http") lb.Set(model.AddressLabel, strings.TrimPrefix(targetURL, "http://")) diff --git a/storage/buffer_test.go b/storage/buffer_test.go index ebe24d8df3..fcb43c2735 100644 --- a/storage/buffer_test.go +++ b/storage/buffer_test.go @@ -188,8 +188,8 @@ func BenchmarkBufferedSeriesIterator(b *testing.B) { b.ReportAllocs() b.ResetTimer() - for it.Next() != chunkenc.ValNone { - // scan everything + for it.Next() != chunkenc.ValNone { // nolint:revive + // Scan everything. } require.NoError(b, it.Err()) } diff --git a/storage/fanout_test.go b/storage/fanout_test.go index 4996e8f64a..b4490636df 100644 --- a/storage/fanout_test.go +++ b/storage/fanout_test.go @@ -233,7 +233,7 @@ func (errQuerier) Select(bool, *storage.SelectHints, ...*labels.Matcher) storage return storage.ErrSeriesSet(errSelect) } -func (errQuerier) LabelValues(name string, matchers ...*labels.Matcher) ([]string, storage.Warnings, error) { +func (errQuerier) LabelValues(string, ...*labels.Matcher) ([]string, storage.Warnings, error) { return nil, nil, errors.New("label values error") } diff --git a/storage/interface.go b/storage/interface.go index 5cf70a351b..b282f1fc62 100644 --- a/storage/interface.go +++ b/storage/interface.go @@ -99,7 +99,7 @@ type MockQueryable struct { MockQuerier Querier } -func (q *MockQueryable) Querier(ctx context.Context, mint, maxt int64) (Querier, error) { +func (q *MockQueryable) Querier(context.Context, int64, int64) (Querier, error) { return q.MockQuerier, nil } @@ -118,11 +118,11 @@ type MockQuerier struct { SelectMockFunction func(sortSeries bool, hints *SelectHints, matchers ...*labels.Matcher) SeriesSet } -func (q *MockQuerier) LabelValues(name string, matchers ...*labels.Matcher) ([]string, Warnings, error) { +func (q *MockQuerier) LabelValues(string, ...*labels.Matcher) ([]string, Warnings, error) { return nil, nil, nil } -func (q *MockQuerier) LabelNames(matchers ...*labels.Matcher) ([]string, Warnings, error) { +func (q *MockQuerier) LabelNames(...*labels.Matcher) ([]string, Warnings, error) { return nil, nil, nil } diff --git a/storage/memoized_iterator_test.go b/storage/memoized_iterator_test.go index 382c84e63b..d996436e00 100644 --- a/storage/memoized_iterator_test.go +++ b/storage/memoized_iterator_test.go @@ -82,8 +82,8 @@ func BenchmarkMemoizedSeriesIterator(b *testing.B) { b.ReportAllocs() b.ResetTimer() - for it.Next() != chunkenc.ValNone { - // scan everything + for it.Next() != chunkenc.ValNone { // nolint:revive + // Scan everything. } require.NoError(b, it.Err()) } diff --git a/storage/merge.go b/storage/merge.go index 8db1f7ae83..193a025227 100644 --- a/storage/merge.go +++ b/storage/merge.go @@ -722,12 +722,11 @@ func (c *compactChunkIterator) Next() bool { break } - if next.MinTime == prev.MinTime && - next.MaxTime == prev.MaxTime && - bytes.Equal(next.Chunk.Bytes(), prev.Chunk.Bytes()) { - // 1:1 duplicates, skip it. - } else { - // We operate on same series, so labels does not matter here. + // Only do something if it is not a perfect duplicate. + if next.MinTime != prev.MinTime || + next.MaxTime != prev.MaxTime || + !bytes.Equal(next.Chunk.Bytes(), prev.Chunk.Bytes()) { + // We operate on same series, so labels do not matter here. overlapping = append(overlapping, newChunkToSeriesDecoder(labels.EmptyLabels(), next)) if next.MaxTime > oMaxTime { oMaxTime = next.MaxTime diff --git a/storage/remote/codec_test.go b/storage/remote/codec_test.go index 36d0d7c31d..27e2cc704d 100644 --- a/storage/remote/codec_test.go +++ b/storage/remote/codec_test.go @@ -524,7 +524,7 @@ func TestDecodeWriteRequest(t *testing.T) { require.Equal(t, writeRequestFixture, actual) } -func TestNilHistogramProto(t *testing.T) { +func TestNilHistogramProto(*testing.T) { // This function will panic if it impromperly handles nil // values, causing the test to fail. HistogramProtoToHistogram(prompb.Histogram{}) diff --git a/storage/remote/queue_manager_test.go b/storage/remote/queue_manager_test.go index 5ec52de6b2..a57c3bf7b1 100644 --- a/storage/remote/queue_manager_test.go +++ b/storage/remote/queue_manager_test.go @@ -362,7 +362,7 @@ func TestReshard(t *testing.T) { c.waitForExpectedData(t) } -func TestReshardRaceWithStop(t *testing.T) { +func TestReshardRaceWithStop(*testing.T) { c := NewTestWriteClient() var m *QueueManager h := sync.Mutex{} @@ -864,10 +864,10 @@ func (c *TestBlockingWriteClient) Endpoint() string { // For benchmarking the send and not the receive side. type NopWriteClient struct{} -func NewNopWriteClient() *NopWriteClient { return &NopWriteClient{} } -func (c *NopWriteClient) Store(_ context.Context, req []byte) error { return nil } -func (c *NopWriteClient) Name() string { return "nopwriteclient" } -func (c *NopWriteClient) Endpoint() string { return "http://test-remote.com/1234" } +func NewNopWriteClient() *NopWriteClient { return &NopWriteClient{} } +func (c *NopWriteClient) Store(context.Context, []byte) error { return nil } +func (c *NopWriteClient) Name() string { return "nopwriteclient" } +func (c *NopWriteClient) Endpoint() string { return "http://test-remote.com/1234" } func BenchmarkSampleSend(b *testing.B) { // Send one sample per series, which is the typical remote_write case diff --git a/storage/remote/write_handler_test.go b/storage/remote/write_handler_test.go index 9c787f17e5..e7a88ddc23 100644 --- a/storage/remote/write_handler_test.go +++ b/storage/remote/write_handler_test.go @@ -294,7 +294,7 @@ func (m *mockAppendable) AppendExemplar(_ storage.SeriesRef, l labels.Labels, e return 0, nil } -func (m *mockAppendable) AppendHistogram(ref storage.SeriesRef, l labels.Labels, t int64, h *histogram.Histogram, fh *histogram.FloatHistogram) (storage.SeriesRef, error) { +func (m *mockAppendable) AppendHistogram(_ storage.SeriesRef, l labels.Labels, t int64, h *histogram.Histogram, fh *histogram.FloatHistogram) (storage.SeriesRef, error) { if t < m.latestHistogram { return 0, storage.ErrOutOfOrderSample } diff --git a/tsdb/agent/db.go b/tsdb/agent/db.go index a17e0d1b98..cb075f3060 100644 --- a/tsdb/agent/db.go +++ b/tsdb/agent/db.go @@ -732,22 +732,22 @@ func (db *DB) StartTime() (int64, error) { } // Querier implements the Storage interface. -func (db *DB) Querier(ctx context.Context, mint, maxt int64) (storage.Querier, error) { +func (db *DB) Querier(context.Context, int64, int64) (storage.Querier, error) { return nil, ErrUnsupported } // ChunkQuerier implements the Storage interface. -func (db *DB) ChunkQuerier(ctx context.Context, mint, maxt int64) (storage.ChunkQuerier, error) { +func (db *DB) ChunkQuerier(context.Context, int64, int64) (storage.ChunkQuerier, error) { return nil, ErrUnsupported } // ExemplarQuerier implements the Storage interface. -func (db *DB) ExemplarQuerier(ctx context.Context) (storage.ExemplarQuerier, error) { +func (db *DB) ExemplarQuerier(context.Context) (storage.ExemplarQuerier, error) { return nil, ErrUnsupported } // Appender implements storage.Storage. -func (db *DB) Appender(_ context.Context) storage.Appender { +func (db *DB) Appender(context.Context) storage.Appender { return db.appenderPool.Get().(storage.Appender) } @@ -823,7 +823,7 @@ func (a *appender) Append(ref storage.SeriesRef, l labels.Labels, t int64, v flo return 0, storage.ErrOutOfOrderSample } - // NOTE: always modify pendingSamples and sampleSeries together + // NOTE: always modify pendingSamples and sampleSeries together. a.pendingSamples = append(a.pendingSamples, record.RefSample{ Ref: series.ref, T: t, @@ -849,8 +849,8 @@ func (a *appender) getOrCreate(l labels.Labels) (series *memSeries, created bool return series, true } -func (a *appender) AppendExemplar(ref storage.SeriesRef, l labels.Labels, e exemplar.Exemplar) (storage.SeriesRef, error) { - // series references and chunk references are identical for agent mode. +func (a *appender) AppendExemplar(ref storage.SeriesRef, _ labels.Labels, e exemplar.Exemplar) (storage.SeriesRef, error) { + // Series references and chunk references are identical for agent mode. headRef := chunks.HeadSeriesRef(ref) s := a.series.GetByID(headRef) @@ -973,7 +973,7 @@ func (a *appender) AppendHistogram(ref storage.SeriesRef, l labels.Labels, t int return storage.SeriesRef(series.ref), nil } -func (a *appender) UpdateMetadata(ref storage.SeriesRef, l labels.Labels, m metadata.Metadata) (storage.SeriesRef, error) { +func (a *appender) UpdateMetadata(storage.SeriesRef, labels.Labels, metadata.Metadata) (storage.SeriesRef, error) { // TODO: Wire metadata in the Agent's appender. return 0, nil } diff --git a/tsdb/chunkenc/float_histogram.go b/tsdb/chunkenc/float_histogram.go index b462c6d9fd..67b706b2eb 100644 --- a/tsdb/chunkenc/float_histogram.go +++ b/tsdb/chunkenc/float_histogram.go @@ -107,7 +107,7 @@ func (c *FloatHistogramChunk) Appender() (Appender, error) { // To get an appender, we must know the state it would have if we had // appended all existing data from scratch. We iterate through the end // and populate via the iterator's state. - for it.Next() == ValFloatHistogram { + for it.Next() == ValFloatHistogram { // nolint:revive } if err := it.Err(); err != nil { return nil, err diff --git a/tsdb/chunkenc/float_histogram_test.go b/tsdb/chunkenc/float_histogram_test.go index 31d96ee7a9..90c16d1ea9 100644 --- a/tsdb/chunkenc/float_histogram_test.go +++ b/tsdb/chunkenc/float_histogram_test.go @@ -111,7 +111,7 @@ func TestFloatHistogramChunkSameBuckets(t *testing.T) { // 3. Now recycle an iterator that was never used to access anything. itX := c.Iterator(nil) - for itX.Next() == ValFloatHistogram { + for itX.Next() == ValFloatHistogram { // nolint:revive // Just iterate through without accessing anything. } it3 := c.iterator(itX) diff --git a/tsdb/chunkenc/histogram.go b/tsdb/chunkenc/histogram.go index 7b6a9cacb3..0d53f1f496 100644 --- a/tsdb/chunkenc/histogram.go +++ b/tsdb/chunkenc/histogram.go @@ -126,7 +126,7 @@ func (c *HistogramChunk) Appender() (Appender, error) { // To get an appender, we must know the state it would have if we had // appended all existing data from scratch. We iterate through the end // and populate via the iterator's state. - for it.Next() == ValHistogram { + for it.Next() == ValHistogram { // nolint:revive } if err := it.Err(); err != nil { return nil, err diff --git a/tsdb/chunkenc/histogram_test.go b/tsdb/chunkenc/histogram_test.go index 4bb146ccdb..45f31a3b4d 100644 --- a/tsdb/chunkenc/histogram_test.go +++ b/tsdb/chunkenc/histogram_test.go @@ -116,7 +116,7 @@ func TestHistogramChunkSameBuckets(t *testing.T) { // 3. Now recycle an iterator that was never used to access anything. itX := c.Iterator(nil) - for itX.Next() == ValHistogram { + for itX.Next() == ValHistogram { // nolint:revive // Just iterate through without accessing anything. } it3 := c.iterator(itX) diff --git a/tsdb/chunkenc/xor.go b/tsdb/chunkenc/xor.go index 2fa2f613cb..ba2d96d36b 100644 --- a/tsdb/chunkenc/xor.go +++ b/tsdb/chunkenc/xor.go @@ -99,7 +99,7 @@ func (c *XORChunk) Appender() (Appender, error) { // To get an appender we must know the state it would have if we had // appended all existing data from scratch. // We iterate through the end and populate via the iterator's state. - for it.Next() != ValNone { + for it.Next() != ValNone { // nolint:revive } if err := it.Err(); err != nil { return nil, err @@ -152,11 +152,11 @@ type xorAppender struct { trailing uint8 } -func (a *xorAppender) AppendHistogram(t int64, h *histogram.Histogram) { +func (a *xorAppender) AppendHistogram(int64, *histogram.Histogram) { panic("appended a histogram to an xor chunk") } -func (a *xorAppender) AppendFloatHistogram(t int64, h *histogram.FloatHistogram) { +func (a *xorAppender) AppendFloatHistogram(int64, *histogram.FloatHistogram) { panic("appended a float histogram to an xor chunk") } diff --git a/tsdb/chunks/head_chunks_test.go b/tsdb/chunks/head_chunks_test.go index ac89ae3e59..20a4c2064b 100644 --- a/tsdb/chunks/head_chunks_test.go +++ b/tsdb/chunks/head_chunks_test.go @@ -503,10 +503,10 @@ func createChunkDiskMapper(t *testing.T, dir string) *ChunkDiskMapper { func randomChunk(t *testing.T) chunkenc.Chunk { chunk := chunkenc.NewXORChunk() - len := rand.Int() % 120 + length := rand.Int() % 120 app, err := chunk.Appender() require.NoError(t, err) - for i := 0; i < len; i++ { + for i := 0; i < length; i++ { app.Append(rand.Int63(), rand.Float64()) } return chunk diff --git a/tsdb/compact_test.go b/tsdb/compact_test.go index d9a105f225..93bca179e0 100644 --- a/tsdb/compact_test.go +++ b/tsdb/compact_test.go @@ -467,8 +467,8 @@ func (erringBReader) Size() int64 { return 0 } type nopChunkWriter struct{} -func (nopChunkWriter) WriteChunks(chunks ...chunks.Meta) error { return nil } -func (nopChunkWriter) Close() error { return nil } +func (nopChunkWriter) WriteChunks(...chunks.Meta) error { return nil } +func (nopChunkWriter) Close() error { return nil } func samplesForRange(minTime, maxTime int64, maxSamplesPerChunk int) (ret [][]sample) { var curr []sample diff --git a/tsdb/db_test.go b/tsdb/db_test.go index c54fccf6f1..3e4d35f878 100644 --- a/tsdb/db_test.go +++ b/tsdb/db_test.go @@ -1426,11 +1426,11 @@ type mockCompactorFailing struct { max int } -func (*mockCompactorFailing) Plan(dir string) ([]string, error) { +func (*mockCompactorFailing) Plan(string) ([]string, error) { return nil, nil } -func (c *mockCompactorFailing) Write(dest string, b BlockReader, mint, maxt int64, parent *BlockMeta) (ulid.ULID, error) { +func (c *mockCompactorFailing) Write(dest string, _ BlockReader, _, _ int64, _ *BlockMeta) (ulid.ULID, error) { if len(c.blocks) >= c.max { return ulid.ULID{}, fmt.Errorf("the compactor already did the maximum allowed blocks so it is time to fail") } @@ -1458,7 +1458,7 @@ func (*mockCompactorFailing) Compact(string, []string, []*Block) (ulid.ULID, err return ulid.ULID{}, nil } -func (*mockCompactorFailing) CompactOOO(dest string, oooHead *OOOCompactionHead) (result []ulid.ULID, err error) { +func (*mockCompactorFailing) CompactOOO(string, *OOOCompactionHead) (result []ulid.ULID, err error) { return nil, fmt.Errorf("mock compaction failing CompactOOO") } diff --git a/tsdb/exemplar.go b/tsdb/exemplar.go index 5ba3567e41..ca108ae91c 100644 --- a/tsdb/exemplar.go +++ b/tsdb/exemplar.go @@ -115,17 +115,17 @@ func NewExemplarMetrics(reg prometheus.Registerer) *ExemplarMetrics { // 1GB of extra memory, accounting for the fact that this is heap allocated space. // If len <= 0, then the exemplar storage is essentially a noop storage but can later be // resized to store exemplars. -func NewCircularExemplarStorage(len int64, m *ExemplarMetrics) (ExemplarStorage, error) { - if len < 0 { - len = 0 +func NewCircularExemplarStorage(length int64, m *ExemplarMetrics) (ExemplarStorage, error) { + if length < 0 { + length = 0 } c := &CircularExemplarStorage{ - exemplars: make([]*circularBufferEntry, len), - index: make(map[string]*indexEntry, len/estimatedExemplarsPerSeries), + exemplars: make([]*circularBufferEntry, length), + index: make(map[string]*indexEntry, length/estimatedExemplarsPerSeries), metrics: m, } - c.metrics.maxExemplars.Set(float64(len)) + c.metrics.maxExemplars.Set(float64(length)) return c, nil } diff --git a/tsdb/goversion/goversion_test.go b/tsdb/goversion/goversion_test.go index 9a7486d66f..853844fb93 100644 --- a/tsdb/goversion/goversion_test.go +++ b/tsdb/goversion/goversion_test.go @@ -24,4 +24,4 @@ import ( // // The blank import above is actually what invokes the test of this package. If // the import succeeds (the code compiles), the test passed. -func Test(t *testing.T) {} +func Test(*testing.T) {} diff --git a/tsdb/head_test.go b/tsdb/head_test.go index e80c197b23..39bcf4c78b 100644 --- a/tsdb/head_test.go +++ b/tsdb/head_test.go @@ -11,6 +11,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +// nolint:revive // Many legitimately empty blocks in this file. package tsdb import ( @@ -103,7 +104,7 @@ func BenchmarkHeadAppender_Append_Commit_ExistingSeries(b *testing.B) { b.Cleanup(func() { require.NoError(b, h.Close()) }) ts := int64(1000) - append := func() error { + appendSamples := func() error { var err error app := h.Appender(context.Background()) for _, s := range series[:seriesCount] { @@ -120,13 +121,13 @@ func BenchmarkHeadAppender_Append_Commit_ExistingSeries(b *testing.B) { } // Init series, that's not what we're benchmarking here. - require.NoError(b, append()) + require.NoError(b, appendSamples()) b.ReportAllocs() b.ResetTimer() for i := 0; i < b.N; i++ { - require.NoError(b, append()) + require.NoError(b, appendSamples()) } }) } diff --git a/tsdb/head_wal.go b/tsdb/head_wal.go index 6a8a30d5ad..4da500a712 100644 --- a/tsdb/head_wal.go +++ b/tsdb/head_wal.go @@ -11,6 +11,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +// nolint:revive // Many legitimately empty blocks in this file. package tsdb import ( diff --git a/tsdb/ooo_head_read.go b/tsdb/ooo_head_read.go index 86d0e3b7b3..fd16c6ca8f 100644 --- a/tsdb/ooo_head_read.go +++ b/tsdb/ooo_head_read.go @@ -11,6 +11,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +// nolint:revive // Many unsued function arguments in this file by design. package tsdb import ( diff --git a/tsdb/querier.go b/tsdb/querier.go index 4b3144c71d..f3852d517b 100644 --- a/tsdb/querier.go +++ b/tsdb/querier.go @@ -1085,7 +1085,7 @@ func newNopChunkReader() ChunkReader { } } -func (cr nopChunkReader) Chunk(meta chunks.Meta) (chunkenc.Chunk, error) { +func (cr nopChunkReader) Chunk(chunks.Meta) (chunkenc.Chunk, error) { return cr.emptyChunk, nil } diff --git a/tsdb/querier_bench_test.go b/tsdb/querier_bench_test.go index 78993d105f..89758a1d31 100644 --- a/tsdb/querier_bench_test.go +++ b/tsdb/querier_bench_test.go @@ -250,7 +250,7 @@ func BenchmarkQuerierSelect(b *testing.B) { b.ResetTimer() for i := 0; i < b.N; i++ { ss := q.Select(sorted, nil, matcher) - for ss.Next() { + for ss.Next() { // nolint:revive } require.NoError(b, ss.Err()) } diff --git a/tsdb/querier_test.go b/tsdb/querier_test.go index fa3dd2418f..3a92e49516 100644 --- a/tsdb/querier_test.go +++ b/tsdb/querier_test.go @@ -11,6 +11,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +// nolint:revive // Many unsued function arguments in this file by design. package tsdb import ( diff --git a/tsdb/wal.go b/tsdb/wal.go index e0bc1ec699..93d445a126 100644 --- a/tsdb/wal.go +++ b/tsdb/wal.go @@ -11,6 +11,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +// nolint:revive // Many unsued function arguments in this file by design. package tsdb import ( diff --git a/tsdb/wlog/reader_test.go b/tsdb/wlog/reader_test.go index 97d251b3ae..737520e76a 100644 --- a/tsdb/wlog/reader_test.go +++ b/tsdb/wlog/reader_test.go @@ -533,7 +533,7 @@ func TestReaderData(t *testing.T) { require.NoError(t, err) reader := fn(sr) - for reader.Next() { + for reader.Next() { // nolint:revive } require.NoError(t, reader.Err()) diff --git a/tsdb/wlog/wlog_test.go b/tsdb/wlog/wlog_test.go index ed8a9df2e3..d36934bbd3 100644 --- a/tsdb/wlog/wlog_test.go +++ b/tsdb/wlog/wlog_test.go @@ -164,7 +164,7 @@ func TestWALRepair_ReadingError(t *testing.T) { sr := NewSegmentBufReader(s) require.NoError(t, err) r := NewReader(sr) - for r.Next() { + for r.Next() { // nolint:revive } // Close the segment so we don't break things on Windows. diff --git a/util/logging/dedupe_test.go b/util/logging/dedupe_test.go index ad234445b2..e05d6454c5 100644 --- a/util/logging/dedupe_test.go +++ b/util/logging/dedupe_test.go @@ -22,7 +22,7 @@ import ( type counter int -func (c *counter) Log(keyvals ...interface{}) error { +func (c *counter) Log(...interface{}) error { (*c)++ return nil } diff --git a/util/testutil/context.go b/util/testutil/context.go index cf730421b4..c1f4a831ce 100644 --- a/util/testutil/context.go +++ b/util/testutil/context.go @@ -37,6 +37,6 @@ func (c *MockContext) Err() error { } // Value ignores the Value and always returns nil -func (c *MockContext) Value(key interface{}) interface{} { +func (c *MockContext) Value(interface{}) interface{} { return nil } diff --git a/util/testutil/roundtrip.go b/util/testutil/roundtrip.go index a93991a13e..364e0c2642 100644 --- a/util/testutil/roundtrip.go +++ b/util/testutil/roundtrip.go @@ -22,7 +22,7 @@ type roundTrip struct { theError error } -func (rt *roundTrip) RoundTrip(r *http.Request) (*http.Response, error) { +func (rt *roundTrip) RoundTrip(*http.Request) (*http.Response, error) { return rt.theResponse, rt.theError } diff --git a/util/treecache/treecache.go b/util/treecache/treecache.go index 7dd41dcedd..acdd6f7bea 100644 --- a/util/treecache/treecache.go +++ b/util/treecache/treecache.go @@ -116,7 +116,7 @@ func (tc *ZookeeperTreeCache) Stop() { tc.stop <- struct{}{} go func() { // Drain tc.head.events so that go routines can make progress and exit. - for range tc.head.events { + for range tc.head.events { // nolint:revive } }() go func() { diff --git a/web/api/v1/api.go b/web/api/v1/api.go index dde814eb0f..0624cf2d8b 100644 --- a/web/api/v1/api.go +++ b/web/api/v1/api.go @@ -117,7 +117,7 @@ type RulesRetriever interface { type StatsRenderer func(context.Context, *stats.Statistics, string) stats.QueryStats -func defaultStatsRenderer(ctx context.Context, s *stats.Statistics, param string) stats.QueryStats { +func defaultStatsRenderer(_ context.Context, s *stats.Statistics, param string) stats.QueryStats { if param != "" { return stats.NewQueryStats(s) } @@ -392,7 +392,7 @@ func invalidParamError(err error, parameter string) apiFuncResult { }, nil, nil} } -func (api *API) options(r *http.Request) apiFuncResult { +func (api *API) options(*http.Request) apiFuncResult { return apiFuncResult{nil, nil, nil, nil} } @@ -1565,7 +1565,7 @@ func (api *API) snapshot(r *http.Request) apiFuncResult { }{name}, nil, nil, nil} } -func (api *API) cleanTombstones(r *http.Request) apiFuncResult { +func (api *API) cleanTombstones(*http.Request) apiFuncResult { if !api.enableAdmin { return apiFuncResult{nil, &apiError{errorUnavailable, errors.New("admin APIs disabled")}, nil, nil} } @@ -1764,7 +1764,7 @@ func marshalSeriesJSON(ptr unsafe.Pointer, stream *jsoniter.Stream) { stream.WriteObjectEnd() } -func marshalSeriesJSONIsEmpty(ptr unsafe.Pointer) bool { +func marshalSeriesJSONIsEmpty(unsafe.Pointer) bool { return false } @@ -1817,7 +1817,7 @@ func marshalSampleJSON(ptr unsafe.Pointer, stream *jsoniter.Stream) { stream.WriteObjectEnd() } -func marshalSampleJSONIsEmpty(ptr unsafe.Pointer) bool { +func marshalSampleJSONIsEmpty(unsafe.Pointer) bool { return false } @@ -1841,7 +1841,7 @@ func marshalHPointJSON(ptr unsafe.Pointer, stream *jsoniter.Stream) { stream.WriteArrayEnd() } -func marshalPointJSONIsEmpty(ptr unsafe.Pointer) bool { +func marshalPointJSONIsEmpty(unsafe.Pointer) bool { return false } @@ -1878,6 +1878,6 @@ func marshalExemplarJSON(ptr unsafe.Pointer, stream *jsoniter.Stream) { stream.WriteObjectEnd() } -func marshalExemplarJSONEmpty(ptr unsafe.Pointer) bool { +func marshalExemplarJSONEmpty(unsafe.Pointer) bool { return false } diff --git a/web/api/v1/api_test.go b/web/api/v1/api_test.go index efce042214..04aab31af4 100644 --- a/web/api/v1/api_test.go +++ b/web/api/v1/api_test.go @@ -2560,9 +2560,9 @@ type fakeDB struct { err error } -func (f *fakeDB) CleanTombstones() error { return f.err } -func (f *fakeDB) Delete(mint, maxt int64, ms ...*labels.Matcher) error { return f.err } -func (f *fakeDB) Snapshot(dir string, withHead bool) error { return f.err } +func (f *fakeDB) CleanTombstones() error { return f.err } +func (f *fakeDB) Delete(int64, int64, ...*labels.Matcher) error { return f.err } +func (f *fakeDB) Snapshot(string, bool) error { return f.err } func (f *fakeDB) Stats(statsByLabelName string) (_ *tsdb.Stats, retErr error) { dbDir, err := os.MkdirTemp("", "tsdb-api-ready") if err != nil { diff --git a/web/api/v1/errors_test.go b/web/api/v1/errors_test.go index da0d4b3f2e..4947afd813 100644 --- a/web/api/v1/errors_test.go +++ b/web/api/v1/errors_test.go @@ -11,6 +11,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +// nolint:revive // Many unsued function arguments in this file by design. package v1 import ( diff --git a/web/web.go b/web/web.go index 9d63094f69..cac41502d2 100644 --- a/web/web.go +++ b/web/web.go @@ -755,14 +755,14 @@ func toFloat64(f *io_prometheus_client.MetricFamily) float64 { return math.NaN() } -func (h *Handler) version(w http.ResponseWriter, r *http.Request) { +func (h *Handler) version(w http.ResponseWriter, _ *http.Request) { dec := json.NewEncoder(w) if err := dec.Encode(h.versionInfo); err != nil { http.Error(w, fmt.Sprintf("error encoding JSON: %s", err), http.StatusInternalServerError) } } -func (h *Handler) quit(w http.ResponseWriter, r *http.Request) { +func (h *Handler) quit(w http.ResponseWriter, _ *http.Request) { var closed bool h.quitOnce.Do(func() { closed = true @@ -774,7 +774,7 @@ func (h *Handler) quit(w http.ResponseWriter, r *http.Request) { } } -func (h *Handler) reload(w http.ResponseWriter, r *http.Request) { +func (h *Handler) reload(w http.ResponseWriter, _ *http.Request) { rc := make(chan error) h.reloadCh <- rc if err := <-rc; err != nil { From 5b53aa11083233963bd477d19134492355d9ed0f Mon Sep 17 00:00:00 2001 From: beorn7 Date: Wed, 12 Apr 2023 16:14:31 +0200 Subject: [PATCH 104/231] style: Replace `else if` cascades with `switch` Wiser coders than myself have come to the conclusion that a `switch` statement is almost always superior to a statement that includes any `else if`. The exceptions that I have found in our codebase are just these two: * The `if else` is followed by an additional statement before the next condition (separated by a `;`). * The whole thing is within a `for` loop and `break` statements are used. In this case, using `switch` would require tagging the `for` loop, which probably tips the balance. Why are `switch` statements more readable? For one, fewer curly braces. But more importantly, the conditions all have the same alignment, so the whole thing follows the natural flow of going down a list of conditions. With `else if`, in contrast, all conditions but the first are "hidden" behind `} else if `, harder to spot and (for no good reason) presented differently from the first condition. I'm sure the aforemention wise coders can list even more reasons. In any case, I like it so much that I have found myself recommending it in code reviews. I would like to make it a habit in our code base, without making it a hard requirement that we would test on the CI. But for that, there has to be a role model, so this commit eliminates all `if else` occurrences, unless it is autogenerated code or fits one of the exceptions above. Signed-off-by: beorn7 --- cmd/prometheus/main_unix_test.go | 6 +- cmd/promtool/tsdb.go | 5 +- discovery/dns/dns.go | 7 ++- discovery/kubernetes/kubernetes.go | 7 ++- discovery/linode/linode.go | 7 ++- discovery/marathon/marathon.go | 12 ++-- .../remote_storage_adapter/influxdb/client.go | 7 ++- model/histogram/float_histogram.go | 5 +- model/labels/labels.go | 14 +++-- promql/engine.go | 56 +++++++++++-------- promql/functions.go | 8 ++- promql/parser/ast.go | 9 +-- promql/parser/lex.go | 19 ++++--- promql/parser/parse.go | 28 +++++----- promql/parser/printer.go | 36 +++++++----- promql/quantile.go | 10 ++-- rules/alerting_test.go | 6 +- rules/manager_test.go | 7 ++- rules/recording_test.go | 6 +- scrape/manager.go | 5 +- scrape/scrape.go | 10 ++-- scrape/scrape_test.go | 36 +++++++----- storage/fanout.go | 5 +- storage/merge.go | 7 ++- storage/remote/codec.go | 23 ++++---- storage/remote/ewma.go | 5 +- storage/remote/queue_manager.go | 10 ++-- tsdb/agent/db.go | 5 +- tsdb/chunkenc/xor.go | 7 ++- tsdb/chunks/head_chunks.go | 5 +- tsdb/db.go | 10 ++-- tsdb/head_append.go | 36 +++++++----- tsdb/head_read.go | 13 +++-- tsdb/head_test.go | 29 +++++----- tsdb/index/postings.go | 22 ++++---- tsdb/ooo_head_read.go | 11 ++-- tsdb/querier.go | 25 +++++---- tsdb/tombstones/tombstones.go | 5 +- tsdb/wal.go | 20 ++++--- tsdb/wlog/live_reader.go | 5 +- tsdb/wlog/watcher.go | 10 ++-- util/treecache/treecache.go | 16 +++--- web/api/v1/api.go | 8 ++- web/federate_test.go | 10 ++-- 44 files changed, 340 insertions(+), 253 deletions(-) diff --git a/cmd/prometheus/main_unix_test.go b/cmd/prometheus/main_unix_test.go index b49110ea91..7224e25d70 100644 --- a/cmd/prometheus/main_unix_test.go +++ b/cmd/prometheus/main_unix_test.go @@ -72,9 +72,11 @@ Loop: if !startedOk { t.Fatal("prometheus didn't start in the specified timeout") } - if err := prom.Process.Kill(); err == nil { + switch err := prom.Process.Kill(); { + case err == nil: t.Errorf("prometheus didn't shutdown gracefully after sending the Interrupt signal") - } else if stoppedErr != nil && stoppedErr.Error() != "signal: interrupt" { // TODO - find a better way to detect when the process didn't exit as expected! + case stoppedErr != nil && stoppedErr.Error() != "signal: interrupt": + // TODO: find a better way to detect when the process didn't exit as expected! t.Errorf("prometheus exited with an unexpected error: %v", stoppedErr) } } diff --git a/cmd/promtool/tsdb.go b/cmd/promtool/tsdb.go index 0e0cdb863a..84aa43a9c4 100644 --- a/cmd/promtool/tsdb.go +++ b/cmd/promtool/tsdb.go @@ -403,14 +403,15 @@ func openBlock(path, blockID string) (*tsdb.DBReadOnly, tsdb.BlockReader, error) return nil, nil, err } var block tsdb.BlockReader - if blockID != "" { + switch { + case blockID != "": for _, b := range blocks { if b.Meta().ULID.String() == blockID { block = b break } } - } else if len(blocks) > 0 { + case len(blocks) > 0: block = blocks[len(blocks)-1] } if block == nil { diff --git a/discovery/dns/dns.go b/discovery/dns/dns.go index 2b11c242ab..96e07254f0 100644 --- a/discovery/dns/dns.go +++ b/discovery/dns/dns.go @@ -285,21 +285,22 @@ func lookupWithSearchPath(name string, qtype uint16, logger log.Logger) (*dns.Ms for _, lname := range conf.NameList(name) { response, err := lookupFromAnyServer(lname, qtype, conf, logger) - if err != nil { + switch { + case err != nil: // We can't go home yet, because a later name // may give us a valid, successful answer. However // we can no longer say "this name definitely doesn't // exist", because we did not get that answer for // at least one name. allResponsesValid = false - } else if response.Rcode == dns.RcodeSuccess { + case response.Rcode == dns.RcodeSuccess: // Outcome 1: GOLD! return response, nil } } if allResponsesValid { - // Outcome 2: everyone says NXDOMAIN, that's good enough for me + // Outcome 2: everyone says NXDOMAIN, that's good enough for me. return &dns.Msg{}, nil } // Outcome 3: boned. diff --git a/discovery/kubernetes/kubernetes.go b/discovery/kubernetes/kubernetes.go index 0f03e2cdb7..a44bd513ce 100644 --- a/discovery/kubernetes/kubernetes.go +++ b/discovery/kubernetes/kubernetes.go @@ -299,12 +299,13 @@ func New(l log.Logger, conf *SDConfig) (*Discovery, error) { err error ownNamespace string ) - if conf.KubeConfig != "" { + switch { + case conf.KubeConfig != "": kcfg, err = clientcmd.BuildConfigFromFlags("", conf.KubeConfig) if err != nil { return nil, err } - } else if conf.APIServer.URL == nil { + case conf.APIServer.URL == nil: // Use the Kubernetes provided pod service account // as described in https://kubernetes.io/docs/admin/service-accounts-admin/ kcfg, err = rest.InClusterConfig() @@ -324,7 +325,7 @@ func New(l log.Logger, conf *SDConfig) (*Discovery, error) { } level.Info(l).Log("msg", "Using pod service account via in-cluster config") - } else { + default: rt, err := config.NewRoundTripperFromConfig(conf.HTTPClientConfig, "kubernetes_sd") if err != nil { return nil, err diff --git a/discovery/linode/linode.go b/discovery/linode/linode.go index 0fd0a2c370..449e13cd89 100644 --- a/discovery/linode/linode.go +++ b/discovery/linode/linode.go @@ -250,19 +250,20 @@ func (d *Discovery) refreshData(ctx context.Context) ([]*targetgroup.Group, erro continue } - if detailedIP.Public && publicIPv4 == "" { + switch { + case detailedIP.Public && publicIPv4 == "": publicIPv4 = detailedIP.Address if detailedIP.RDNS != "" && detailedIP.RDNS != "null" { publicIPv4RDNS = detailedIP.RDNS } - } else if !detailedIP.Public && privateIPv4 == "" { + case !detailedIP.Public && privateIPv4 == "": privateIPv4 = detailedIP.Address if detailedIP.RDNS != "" && detailedIP.RDNS != "null" { privateIPv4RDNS = detailedIP.RDNS } - } else { + default: extraIPs = append(extraIPs, detailedIP.Address) } } diff --git a/discovery/marathon/marathon.go b/discovery/marathon/marathon.go index 079f93ad0b..cfd3e2c083 100644 --- a/discovery/marathon/marathon.go +++ b/discovery/marathon/marathon.go @@ -136,9 +136,10 @@ func NewDiscovery(conf SDConfig, logger log.Logger) (*Discovery, error) { return nil, err } - if len(conf.AuthToken) > 0 { + switch { + case len(conf.AuthToken) > 0: rt, err = newAuthTokenRoundTripper(conf.AuthToken, rt) - } else if len(conf.AuthTokenFile) > 0 { + case len(conf.AuthTokenFile) > 0: rt, err = newAuthTokenFileRoundTripper(conf.AuthTokenFile, rt) } if err != nil { @@ -400,19 +401,20 @@ func targetsForApp(app *app) []model.LabelSet { var labels []map[string]string var prefix string - if len(app.Container.PortMappings) != 0 { + switch { + case len(app.Container.PortMappings) != 0: // In Marathon 1.5.x the "container.docker.portMappings" object was moved // to "container.portMappings". ports, labels = extractPortMapping(app.Container.PortMappings, app.isContainerNet()) prefix = portMappingLabelPrefix - } else if len(app.Container.Docker.PortMappings) != 0 { + case len(app.Container.Docker.PortMappings) != 0: // Prior to Marathon 1.5 the port mappings could be found at the path // "container.docker.portMappings". ports, labels = extractPortMapping(app.Container.Docker.PortMappings, app.isContainerNet()) prefix = portMappingLabelPrefix - } else if len(app.PortDefinitions) != 0 { + case len(app.PortDefinitions) != 0: // PortDefinitions deprecates the "ports" array and can be used to specify // a list of ports with metadata in case a mapping is not required. ports = make([]uint32, len(app.PortDefinitions)) diff --git a/documentation/examples/remote_storage/remote_storage_adapter/influxdb/client.go b/documentation/examples/remote_storage/remote_storage_adapter/influxdb/client.go index fffbc9c2ae..959656aa8f 100644 --- a/documentation/examples/remote_storage/remote_storage_adapter/influxdb/client.go +++ b/documentation/examples/remote_storage/remote_storage_adapter/influxdb/client.go @@ -290,13 +290,14 @@ func mergeSamples(a, b []prompb.Sample) []prompb.Sample { result := make([]prompb.Sample, 0, len(a)+len(b)) i, j := 0, 0 for i < len(a) && j < len(b) { - if a[i].Timestamp < b[j].Timestamp { + switch { + case a[i].Timestamp < b[j].Timestamp: result = append(result, a[i]) i++ - } else if a[i].Timestamp > b[j].Timestamp { + case a[i].Timestamp > b[j].Timestamp: result = append(result, b[j]) j++ - } else { + default: result = append(result, a[i]) i++ j++ diff --git a/model/histogram/float_histogram.go b/model/histogram/float_histogram.go index cd73083bbd..f95f0051c9 100644 --- a/model/histogram/float_histogram.go +++ b/model/histogram/float_histogram.go @@ -824,10 +824,11 @@ mergeLoop: // Merge together all buckets from the original schema that fall into origIdx += span.Offset } currIdx := i.targetIdx(origIdx) - if firstPass { + switch { + case firstPass: i.currIdx = currIdx firstPass = false - } else if currIdx != i.currIdx { + case currIdx != i.currIdx: // Reached next bucket in targetSchema. // Do not actually forward to the next bucket, but break out. break mergeLoop diff --git a/model/labels/labels.go b/model/labels/labels.go index b7398d17f9..93524ddcfc 100644 --- a/model/labels/labels.go +++ b/model/labels/labels.go @@ -169,11 +169,12 @@ func (ls Labels) HashForLabels(b []byte, names ...string) (uint64, []byte) { b = b[:0] i, j := 0, 0 for i < len(ls) && j < len(names) { - if names[j] < ls[i].Name { + switch { + case names[j] < ls[i].Name: j++ - } else if ls[i].Name < names[j] { + case ls[i].Name < names[j]: i++ - } else { + default: b = append(b, ls[i].Name...) b = append(b, seps[0]) b = append(b, ls[i].Value...) @@ -213,11 +214,12 @@ func (ls Labels) BytesWithLabels(buf []byte, names ...string) []byte { b.WriteByte(labelSep) i, j := 0, 0 for i < len(ls) && j < len(names) { - if names[j] < ls[i].Name { + switch { + case names[j] < ls[i].Name: j++ - } else if ls[i].Name < names[j] { + case ls[i].Name < names[j]: i++ - } else { + default: if b.Len() > 1 { b.WriteByte(seps[0]) } diff --git a/promql/engine.go b/promql/engine.go index 21b894b936..1f493f129c 100644 --- a/promql/engine.go +++ b/promql/engine.go @@ -400,7 +400,7 @@ func (ng *Engine) SetQueryLogger(l QueryLogger) { } // NewInstantQuery returns an evaluation query for the given expression at the given time. -func (ng *Engine) NewInstantQuery(ctx context.Context, q storage.Queryable, opts *QueryOpts, qs string, ts time.Time) (Query, error) { +func (ng *Engine) NewInstantQuery(_ context.Context, q storage.Queryable, opts *QueryOpts, qs string, ts time.Time) (Query, error) { expr, err := parser.ParseExpr(qs) if err != nil { return nil, err @@ -416,7 +416,7 @@ func (ng *Engine) NewInstantQuery(ctx context.Context, q storage.Queryable, opts // NewRangeQuery returns an evaluation query for the given time range and with // the resolution set by the interval. -func (ng *Engine) NewRangeQuery(ctx context.Context, q storage.Queryable, opts *QueryOpts, qs string, start, end time.Time, interval time.Duration) (Query, error) { +func (ng *Engine) NewRangeQuery(_ context.Context, q storage.Queryable, opts *QueryOpts, qs string, start, end time.Time, interval time.Duration) (Query, error) { expr, err := parser.ParseExpr(qs) if err != nil { return nil, err @@ -1979,7 +1979,7 @@ func (ev *evaluator) matrixIterSlice( // (b) the number of samples is relatively small. // so a linear search will be as fast as a binary search. var drop int - for drop = 0; histograms[drop].T < mint; drop++ { + for drop = 0; histograms[drop].T < mint; drop++ { // nolint:revive } ev.currentSamples -= drop copy(histograms, histograms[drop:]) @@ -2096,13 +2096,13 @@ func (ev *evaluator) VectorAnd(lhs, rhs Vector, matching *parser.VectorMatching, } func (ev *evaluator) VectorOr(lhs, rhs Vector, matching *parser.VectorMatching, lhsh, rhsh []EvalSeriesHelper, enh *EvalNodeHelper) Vector { - if matching.Card != parser.CardManyToMany { + switch { + case matching.Card != parser.CardManyToMany: panic("set operations must only use many-to-many matching") - } - if len(lhs) == 0 { // Short-circuit. + case len(lhs) == 0: // Short-circuit. enh.Out = append(enh.Out, rhs...) return enh.Out - } else if len(rhs) == 0 { + case len(rhs) == 0: enh.Out = append(enh.Out, lhs...) return enh.Out } @@ -2221,13 +2221,14 @@ func (ev *evaluator) VectorBinop(op parser.ItemType, lhs, rhs Vector, matching * hl, hr = hr, hl } floatValue, histogramValue, keep := vectorElemBinop(op, fl, fr, hl, hr) - if returnBool { + switch { + case returnBool: if keep { floatValue = 1.0 } else { floatValue = 0.0 } - } else if !keep { + case !keep: continue } metric := resultMetric(ls.Metric, rs.Metric, op, matching, enh) @@ -2514,14 +2515,15 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without if !ok { var m labels.Labels enh.resetBuilder(metric) - if without { + switch { + case without: enh.lb.Del(grouping...) enh.lb.Del(labels.MetricName) m = enh.lb.Labels() - } else if len(grouping) > 0 { + case len(grouping) > 0: enh.lb.Keep(grouping...) m = enh.lb.Labels() - } else { + default: m = labels.EmptyLabels() } newAgg := &groupedAggregation{ @@ -2530,9 +2532,10 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without mean: s.F, groupCount: 1, } - if s.H == nil { + switch { + case s.H == nil: newAgg.hasFloat = true - } else if op == parser.SUM { + case op == parser.SUM: newAgg.histogramValue = s.H.Copy() newAgg.hasHistogram = true } @@ -2542,9 +2545,10 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without inputVecLen := int64(len(vec)) resultSize := k - if k > inputVecLen { + switch { + case k > inputVecLen: resultSize = inputVecLen - } else if k == 0 { + case k == 0: resultSize = 1 } switch op { @@ -2637,12 +2641,13 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without case parser.TOPK: // We build a heap of up to k elements, with the smallest element at heap[0]. - if int64(len(group.heap)) < k { + switch { + case int64(len(group.heap)) < k: heap.Push(&group.heap, &Sample{ F: s.F, Metric: s.Metric, }) - } else if group.heap[0].F < s.F || (math.IsNaN(group.heap[0].F) && !math.IsNaN(s.F)) { + case group.heap[0].F < s.F || (math.IsNaN(group.heap[0].F) && !math.IsNaN(s.F)): // This new element is bigger than the previous smallest element - overwrite that. group.heap[0] = Sample{ F: s.F, @@ -2655,12 +2660,13 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without case parser.BOTTOMK: // We build a heap of up to k elements, with the biggest element at heap[0]. - if int64(len(group.reverseHeap)) < k { + switch { + case int64(len(group.reverseHeap)) < k: heap.Push(&group.reverseHeap, &Sample{ F: s.F, Metric: s.Metric, }) - } else if group.reverseHeap[0].F > s.F || (math.IsNaN(group.reverseHeap[0].F) && !math.IsNaN(s.F)) { + case group.reverseHeap[0].F > s.F || (math.IsNaN(group.reverseHeap[0].F) && !math.IsNaN(s.F)): // This new element is smaller than the previous biggest element - overwrite that. group.reverseHeap[0] = Sample{ F: s.F, @@ -2819,9 +2825,10 @@ func PreprocessExpr(expr parser.Expr, start, end time.Time) parser.Expr { func preprocessExprHelper(expr parser.Expr, start, end time.Time) bool { switch n := expr.(type) { case *parser.VectorSelector: - if n.StartOrEnd == parser.START { + switch n.StartOrEnd { + case parser.START: n.Timestamp = makeInt64Pointer(timestamp.FromTime(start)) - } else if n.StartOrEnd == parser.END { + case parser.END: n.Timestamp = makeInt64Pointer(timestamp.FromTime(end)) } return n.Timestamp != nil @@ -2878,9 +2885,10 @@ func preprocessExprHelper(expr parser.Expr, start, end time.Time) bool { if isInvariant { n.Expr = newStepInvariantExpr(n.Expr) } - if n.StartOrEnd == parser.START { + switch n.StartOrEnd { + case parser.START: n.Timestamp = makeInt64Pointer(timestamp.FromTime(start)) - } else if n.StartOrEnd == parser.END { + case parser.END: n.Timestamp = makeInt64Pointer(timestamp.FromTime(end)) } return n.Timestamp != nil diff --git a/promql/functions.go b/promql/functions.go index 0c22cb44c7..2983de4fde 100644 --- a/promql/functions.go +++ b/promql/functions.go @@ -804,12 +804,14 @@ func funcPi(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) V // === sgn(Vector parser.ValueTypeVector) Vector === func funcSgn(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { return simpleFunc(vals, enh, func(v float64) float64 { - if v < 0 { + switch { + case v < 0: return -1 - } else if v > 0 { + case v > 0: return 1 + default: + return v } - return v }) } diff --git a/promql/parser/ast.go b/promql/parser/ast.go index 190af2d590..f156fc6024 100644 --- a/promql/parser/ast.go +++ b/promql/parser/ast.go @@ -368,13 +368,14 @@ func Children(node Node) []Node { case *AggregateExpr: // While this does not look nice, it should avoid unnecessary allocations // caused by slice resizing - if n.Expr == nil && n.Param == nil { + switch { + case n.Expr == nil && n.Param == nil: return nil - } else if n.Expr == nil { + case n.Expr == nil: return []Node{n.Param} - } else if n.Param == nil { + case n.Param == nil: return []Node{n.Expr} - } else { + default: return []Node{n.Expr, n.Param} } case *BinaryExpr: diff --git a/promql/parser/lex.go b/promql/parser/lex.go index 657dc28095..fe5a8abfeb 100644 --- a/promql/parser/lex.go +++ b/promql/parser/lex.go @@ -347,9 +347,10 @@ func lexStatements(l *Lexer) stateFn { switch r := l.next(); { case r == eof: - if l.parenDepth != 0 { + switch { + case l.parenDepth != 0: return l.errorf("unclosed left parenthesis") - } else if l.bracketOpen { + case l.bracketOpen: return l.errorf("unclosed left bracket") } l.emit(EOF) @@ -371,12 +372,13 @@ func lexStatements(l *Lexer) stateFn { case r == '^': l.emit(POW) case r == '=': - if t := l.peek(); t == '=' { + switch t := l.peek(); t { + case '=': l.next() l.emit(EQLC) - } else if t == '~' { + case '~': return l.errorf("unexpected character after '=': %q", t) - } else { + default: l.emit(EQL) } case r == '!': @@ -791,11 +793,12 @@ Loop: default: l.backup() word := l.input[l.start:l.pos] - if kw, ok := key[strings.ToLower(word)]; ok { + switch kw, ok := key[strings.ToLower(word)]; { + case ok: l.emit(kw) - } else if !strings.Contains(word, ":") { + case !strings.Contains(word, ":"): l.emit(IDENTIFIER) - } else { + default: l.emit(METRIC_IDENTIFIER) } break Loop diff --git a/promql/parser/parse.go b/promql/parser/parse.go index 3c32f3c05b..e69ed4595c 100644 --- a/promql/parser/parse.go +++ b/promql/parser/parse.go @@ -270,14 +270,15 @@ var errUnexpected = errors.New("unexpected error") // recover is the handler that turns panics into returns from the top level of Parse. func (p *parser) recover(errp *error) { e := recover() - if _, ok := e.(runtime.Error); ok { + switch _, ok := e.(runtime.Error); { + case ok: // Print the stack trace but do not inhibit the running application. buf := make([]byte, 64<<10) buf = buf[:runtime.Stack(buf, false)] fmt.Fprintf(os.Stderr, "parser panic: %v\n%s", e, buf) *errp = errUnexpected - } else if e != nil { + case e != nil: *errp = e.(error) } } @@ -518,20 +519,18 @@ func (p *parser) checkAST(node Node) (typ ValueType) { p.addParseErrf(n.RHS.PositionRange(), "binary expression must contain only scalar and instant vector types") } - if (lt != ValueTypeVector || rt != ValueTypeVector) && n.VectorMatching != nil { + switch { + case (lt != ValueTypeVector || rt != ValueTypeVector) && n.VectorMatching != nil: if len(n.VectorMatching.MatchingLabels) > 0 { p.addParseErrf(n.PositionRange(), "vector matching only allowed between instant vectors") } n.VectorMatching = nil - } else { - // Both operands are Vectors. - if n.Op.IsSetOperator() { - if n.VectorMatching.Card == CardOneToMany || n.VectorMatching.Card == CardManyToOne { - p.addParseErrf(n.PositionRange(), "no grouping allowed for %q operation", n.Op) - } - if n.VectorMatching.Card != CardManyToMany { - p.addParseErrf(n.PositionRange(), "set operations must always be many-to-many") - } + case n.Op.IsSetOperator(): // Both operands are Vectors. + if n.VectorMatching.Card == CardOneToMany || n.VectorMatching.Card == CardManyToOne { + p.addParseErrf(n.PositionRange(), "no grouping allowed for %q operation", n.Op) + } + if n.VectorMatching.Card != CardManyToMany { + p.addParseErrf(n.PositionRange(), "set operations must always be many-to-many") } } @@ -708,9 +707,10 @@ func (p *parser) addOffset(e Node, offset time.Duration) { } // it is already ensured by parseDuration func that there never will be a zero offset modifier - if *orgoffsetp != 0 { + switch { + case *orgoffsetp != 0: p.addParseErrf(e.PositionRange(), "offset may not be set multiple times") - } else if orgoffsetp != nil { + case orgoffsetp != nil: *orgoffsetp = offset } diff --git a/promql/parser/printer.go b/promql/parser/printer.go index 1f15eeef33..ff171f2152 100644 --- a/promql/parser/printer.go +++ b/promql/parser/printer.go @@ -124,17 +124,19 @@ func (node *MatrixSelector) String() string { // Copy the Vector selector before changing the offset vecSelector := *node.VectorSelector.(*VectorSelector) offset := "" - if vecSelector.OriginalOffset > time.Duration(0) { + switch { + case vecSelector.OriginalOffset > time.Duration(0): offset = fmt.Sprintf(" offset %s", model.Duration(vecSelector.OriginalOffset)) - } else if vecSelector.OriginalOffset < time.Duration(0) { + case vecSelector.OriginalOffset < time.Duration(0): offset = fmt.Sprintf(" offset -%s", model.Duration(-vecSelector.OriginalOffset)) } at := "" - if vecSelector.Timestamp != nil { + switch { + case vecSelector.Timestamp != nil: at = fmt.Sprintf(" @ %.3f", float64(*vecSelector.Timestamp)/1000.0) - } else if vecSelector.StartOrEnd == START { + case vecSelector.StartOrEnd == START: at = " @ start()" - } else if vecSelector.StartOrEnd == END { + case vecSelector.StartOrEnd == END: at = " @ end()" } @@ -162,17 +164,19 @@ func (node *SubqueryExpr) getSubqueryTimeSuffix() string { step = model.Duration(node.Step).String() } offset := "" - if node.OriginalOffset > time.Duration(0) { + switch { + case node.OriginalOffset > time.Duration(0): offset = fmt.Sprintf(" offset %s", model.Duration(node.OriginalOffset)) - } else if node.OriginalOffset < time.Duration(0) { + case node.OriginalOffset < time.Duration(0): offset = fmt.Sprintf(" offset -%s", model.Duration(-node.OriginalOffset)) } at := "" - if node.Timestamp != nil { + switch { + case node.Timestamp != nil: at = fmt.Sprintf(" @ %.3f", float64(*node.Timestamp)/1000.0) - } else if node.StartOrEnd == START { + case node.StartOrEnd == START: at = " @ start()" - } else if node.StartOrEnd == END { + case node.StartOrEnd == END: at = " @ end()" } return fmt.Sprintf("[%s:%s]%s%s", model.Duration(node.Range), step, at, offset) @@ -207,17 +211,19 @@ func (node *VectorSelector) String() string { labelStrings = append(labelStrings, matcher.String()) } offset := "" - if node.OriginalOffset > time.Duration(0) { + switch { + case node.OriginalOffset > time.Duration(0): offset = fmt.Sprintf(" offset %s", model.Duration(node.OriginalOffset)) - } else if node.OriginalOffset < time.Duration(0) { + case node.OriginalOffset < time.Duration(0): offset = fmt.Sprintf(" offset -%s", model.Duration(-node.OriginalOffset)) } at := "" - if node.Timestamp != nil { + switch { + case node.Timestamp != nil: at = fmt.Sprintf(" @ %.3f", float64(*node.Timestamp)/1000.0) - } else if node.StartOrEnd == START { + case node.StartOrEnd == START: at = " @ start()" - } else if node.StartOrEnd == END { + case node.StartOrEnd == END: at = " @ end()" } diff --git a/promql/quantile.go b/promql/quantile.go index aaead671c7..78d0bbaf0c 100644 --- a/promql/quantile.go +++ b/promql/quantile.go @@ -169,11 +169,12 @@ func histogramQuantile(q float64, h *histogram.FloatHistogram) float64 { } } if bucket.Lower < 0 && bucket.Upper > 0 { - if len(h.NegativeBuckets) == 0 && len(h.PositiveBuckets) > 0 { + switch { + case len(h.NegativeBuckets) == 0 && len(h.PositiveBuckets) > 0: // The result is in the zero bucket and the histogram has only // positive buckets. So we consider 0 to be the lower bound. bucket.Lower = 0 - } else if len(h.PositiveBuckets) == 0 && len(h.NegativeBuckets) > 0 { + case len(h.PositiveBuckets) == 0 && len(h.NegativeBuckets) > 0: // The result is in the zero bucket and the histogram has only // negative buckets. So we consider 0 to be the upper bound. bucket.Upper = 0 @@ -244,12 +245,13 @@ func histogramFraction(lower, upper float64, h *histogram.FloatHistogram) float6 for it.Next() { b := it.At() if b.Lower < 0 && b.Upper > 0 { - if len(h.NegativeBuckets) == 0 && len(h.PositiveBuckets) > 0 { + switch { + case len(h.NegativeBuckets) == 0 && len(h.PositiveBuckets) > 0: // This is the zero bucket and the histogram has only // positive buckets. So we consider 0 to be the lower // bound. b.Lower = 0 - } else if len(h.PositiveBuckets) == 0 && len(h.NegativeBuckets) > 0 { + case len(h.PositiveBuckets) == 0 && len(h.NegativeBuckets) > 0: // This is in the zero bucket and the histogram has only // negative buckets. So we consider 0 to be the upper // bound. diff --git a/rules/alerting_test.go b/rules/alerting_test.go index 13aab98048..8cd0da2815 100644 --- a/rules/alerting_test.go +++ b/rules/alerting_test.go @@ -587,10 +587,10 @@ func TestAlertingRuleLimit(t *testing.T) { evalTime := time.Unix(0, 0) for _, test := range tests { - _, err := rule.Eval(suite.Context(), evalTime, EngineQueryFunc(suite.QueryEngine(), suite.Storage()), nil, test.limit) - if err != nil { + switch _, err := rule.Eval(suite.Context(), evalTime, EngineQueryFunc(suite.QueryEngine(), suite.Storage()), nil, test.limit); { + case err != nil: require.EqualError(t, err, test.err) - } else if test.err != "" { + case test.err != "": t.Errorf("Expected errror %s, got none", test.err) } } diff --git a/rules/manager_test.go b/rules/manager_test.go index 16bb080f55..26a7909644 100644 --- a/rules/manager_test.go +++ b/rules/manager_test.go @@ -481,17 +481,18 @@ func TestForStateRestore(t *testing.T) { }) // Checking if we have restored it correctly. - if tst.noRestore { + switch { + case tst.noRestore: require.Equal(t, tst.num, len(got)) for _, e := range got { require.Equal(t, e.ActiveAt, restoreTime) } - } else if tst.gracePeriod { + case tst.gracePeriod: require.Equal(t, tst.num, len(got)) for _, e := range got { require.Equal(t, opts.ForGracePeriod, e.ActiveAt.Add(alertForDuration).Sub(restoreTime)) } - } else { + default: exp := tst.alerts require.Equal(t, len(exp), len(got)) sortAlerts(exp) diff --git a/rules/recording_test.go b/rules/recording_test.go index 61f47e0487..35a0b1a0bd 100644 --- a/rules/recording_test.go +++ b/rules/recording_test.go @@ -223,10 +223,10 @@ func TestRecordingRuleLimit(t *testing.T) { evalTime := time.Unix(0, 0) for _, test := range tests { - _, err := rule.Eval(suite.Context(), evalTime, EngineQueryFunc(suite.QueryEngine(), suite.Storage()), nil, test.limit) - if err != nil { + switch _, err := rule.Eval(suite.Context(), evalTime, EngineQueryFunc(suite.QueryEngine(), suite.Storage()), nil, test.limit); { + case err != nil: require.EqualError(t, err, test.err) - } else if test.err != "" { + case test.err != "": t.Errorf("Expected error %s, got none", test.err) } } diff --git a/scrape/manager.go b/scrape/manager.go index 69a0eaa1f7..d75fe30cf5 100644 --- a/scrape/manager.go +++ b/scrape/manager.go @@ -288,10 +288,11 @@ func (m *Manager) ApplyConfig(cfg *config.Config) error { // Cleanup and reload pool if the configuration has changed. var failed bool for name, sp := range m.scrapePools { - if cfg, ok := m.scrapeConfigs[name]; !ok { + switch cfg, ok := m.scrapeConfigs[name]; { + case !ok: sp.stop() delete(m.scrapePools, name) - } else if !reflect.DeepEqual(sp.config, cfg) { + case !reflect.DeepEqual(sp.config, cfg): err := sp.reload(cfg) if err != nil { level.Error(m.logger).Log("msg", "error reloading scrape pool", "err", err, "scrape_pool", name) diff --git a/scrape/scrape.go b/scrape/scrape.go index f38527ff30..680585af89 100644 --- a/scrape/scrape.go +++ b/scrape/scrape.go @@ -503,9 +503,10 @@ func (sp *scrapePool) Sync(tgs []*targetgroup.Group) { // Replicate .Labels().IsEmpty() with a loop here to avoid generating garbage. nonEmpty := false t.LabelsRange(func(l labels.Label) { nonEmpty = true }) - if nonEmpty { + switch { + case nonEmpty: all = append(all, t) - } else if !t.discoveredLabels.IsEmpty() { + case !t.discoveredLabels.IsEmpty(): sp.droppedTargets = append(sp.droppedTargets, t) } } @@ -946,9 +947,10 @@ func (c *scrapeCache) iterDone(flushCache bool) { count := len(c.series) + len(c.droppedSeries) + len(c.metadata) c.metaMtx.Unlock() - if flushCache { + switch { + case flushCache: c.successfulCount = count - } else if count > c.successfulCount*2+1000 { + case count > c.successfulCount*2+1000: // If a target had varying labels in scrapes that ultimately failed, // the caches would grow indefinitely. Force a flush when this happens. // We use the heuristic that this is a doubling of the cache size diff --git a/scrape/scrape_test.go b/scrape/scrape_test.go index a8028b652a..6b4b2d5f57 100644 --- a/scrape/scrape_test.go +++ b/scrape/scrape_test.go @@ -724,9 +724,10 @@ func TestScrapeLoopStop(t *testing.T) { // All samples in a scrape must have the same timestamp. var ts int64 for i, s := range appender.result { - if i%6 == 0 { + switch { + case i%6 == 0: ts = s.t - } else if s.t != ts { + case s.t != ts: t.Fatalf("Unexpected multiple timestamps within single scrape") } } @@ -1139,10 +1140,11 @@ func TestScrapeLoopRunCreatesStaleMarkersOnFailedScrape(t *testing.T) { scraper.scrapeFunc = func(ctx context.Context, w io.Writer) error { numScrapes++ - if numScrapes == 1 { + switch numScrapes { + case 1: w.Write([]byte("metric_a 42\n")) return nil - } else if numScrapes == 5 { + case 5: cancel() } return errors.New("scrape failed") @@ -1200,13 +1202,14 @@ func TestScrapeLoopRunCreatesStaleMarkersOnParseFailure(t *testing.T) { scraper.scrapeFunc = func(ctx context.Context, w io.Writer) error { numScrapes++ - if numScrapes == 1 { + switch numScrapes { + case 1: w.Write([]byte("metric_a 42\n")) return nil - } else if numScrapes == 2 { + case 2: w.Write([]byte("7&-\n")) return nil - } else if numScrapes == 3 { + case 3: cancel() } return errors.New("scrape failed") @@ -1265,14 +1268,15 @@ func TestScrapeLoopCache(t *testing.T) { numScrapes := 0 scraper.scrapeFunc = func(ctx context.Context, w io.Writer) error { - if numScrapes == 1 || numScrapes == 2 { + switch numScrapes { + case 1, 2: if _, ok := sl.cache.series["metric_a"]; !ok { t.Errorf("metric_a missing from cache after scrape %d", numScrapes) } if _, ok := sl.cache.series["metric_b"]; !ok { t.Errorf("metric_b missing from cache after scrape %d", numScrapes) } - } else if numScrapes == 3 { + case 3: if _, ok := sl.cache.series["metric_a"]; !ok { t.Errorf("metric_a missing from cache after scrape %d", numScrapes) } @@ -1283,13 +1287,14 @@ func TestScrapeLoopCache(t *testing.T) { numScrapes++ - if numScrapes == 1 { + switch numScrapes { + case 1: w.Write([]byte("metric_a 42\nmetric_b 43\n")) return nil - } else if numScrapes == 3 { + case 3: w.Write([]byte("metric_a 44\n")) return nil - } else if numScrapes == 4 { + case 4: cancel() } return fmt.Errorf("scrape failed") @@ -2280,11 +2285,12 @@ func TestTargetScrapeScrapeCancel(t *testing.T) { go func() { _, err := ts.scrape(ctx, io.Discard) - if err == nil { + switch { + case err == nil: errc <- errors.New("Expected error but got nil") - } else if ctx.Err() != context.Canceled { + case ctx.Err() != context.Canceled: errc <- errors.Errorf("Expected context cancellation error but got: %s", ctx.Err()) - } else { + default: close(errc) } }() diff --git a/storage/fanout.go b/storage/fanout.go index 4f995afbac..a9db4f6280 100644 --- a/storage/fanout.go +++ b/storage/fanout.go @@ -222,9 +222,10 @@ func (f *fanoutAppender) Rollback() (err error) { for _, appender := range f.secondaries { rollbackErr := appender.Rollback() - if err == nil { + switch { + case err == nil: err = rollbackErr - } else if rollbackErr != nil { + case rollbackErr != nil: level.Error(f.logger).Log("msg", "Squashed rollback error on rollback", "err", rollbackErr) } } diff --git a/storage/merge.go b/storage/merge.go index 193a025227..c0665d720b 100644 --- a/storage/merge.go +++ b/storage/merge.go @@ -197,13 +197,14 @@ func mergeStrings(a, b []string) []string { res := make([]string, 0, maxl*10/9) for len(a) > 0 && len(b) > 0 { - if a[0] == b[0] { + switch { + case a[0] == b[0]: res = append(res, a[0]) a, b = a[1:], b[1:] - } else if a[0] < b[0] { + case a[0] < b[0]: res = append(res, a[0]) a = a[1:] - } else { + default: res = append(res, b[0]) b = b[1:] } diff --git a/storage/remote/codec.go b/storage/remote/codec.go index bfbd08d24b..2ceed4de11 100644 --- a/storage/remote/codec.go +++ b/storage/remote/codec.go @@ -291,13 +291,14 @@ func MergeLabels(primary, secondary []prompb.Label) []prompb.Label { result := make([]prompb.Label, 0, len(primary)+len(secondary)) i, j := 0, 0 for i < len(primary) && j < len(secondary) { - if primary[i].Name < secondary[j].Name { + switch { + case primary[i].Name < secondary[j].Name: result = append(result, primary[i]) i++ - } else if primary[i].Name > secondary[j].Name { + case primary[i].Name > secondary[j].Name: result = append(result, secondary[j]) j++ - } else { + default: result = append(result, primary[i]) i++ j++ @@ -429,7 +430,8 @@ func (c *concreteSeriesIterator) Seek(t int64) chunkenc.ValueType { return c.series.histograms[n+c.histogramsCur].Timestamp >= t }) - if c.floatsCur < len(c.series.floats) && c.histogramsCur < len(c.series.histograms) { + switch { + case c.floatsCur < len(c.series.floats) && c.histogramsCur < len(c.series.histograms): // If float samples and histogram samples have overlapping timestamps prefer the float samples. if c.series.floats[c.floatsCur].Timestamp <= c.series.histograms[c.histogramsCur].Timestamp { c.curValType = chunkenc.ValFloat @@ -445,9 +447,9 @@ func (c *concreteSeriesIterator) Seek(t int64) chunkenc.ValueType { c.floatsCur-- } } - } else if c.floatsCur < len(c.series.floats) { + case c.floatsCur < len(c.series.floats): c.curValType = chunkenc.ValFloat - } else if c.histogramsCur < len(c.series.histograms) { + case c.histogramsCur < len(c.series.histograms): c.curValType = getHistogramValType(&c.series.histograms[c.histogramsCur]) } @@ -515,18 +517,19 @@ func (c *concreteSeriesIterator) Next() chunkenc.ValueType { } c.curValType = chunkenc.ValNone - if peekFloatTS < peekHistTS { + switch { + case peekFloatTS < peekHistTS: c.floatsCur++ c.curValType = chunkenc.ValFloat - } else if peekHistTS < peekFloatTS { + case peekHistTS < peekFloatTS: c.histogramsCur++ c.curValType = chunkenc.ValHistogram - } else if peekFloatTS == noTS && peekHistTS == noTS { + case peekFloatTS == noTS && peekHistTS == noTS: // This only happens when the iterator is exhausted; we set the cursors off the end to prevent // Seek() from returning anything afterwards. c.floatsCur = len(c.series.floats) c.histogramsCur = len(c.series.histograms) - } else { + default: // Prefer float samples to histogram samples if there's a conflict. We advance the cursor for histograms // anyway otherwise the histogram sample will get selected on the next call to Next(). c.floatsCur++ diff --git a/storage/remote/ewma.go b/storage/remote/ewma.go index c7fb0289b0..ea4472c494 100644 --- a/storage/remote/ewma.go +++ b/storage/remote/ewma.go @@ -55,9 +55,10 @@ func (r *ewmaRate) tick() { r.mutex.Lock() defer r.mutex.Unlock() - if r.init { + switch { + case r.init: r.lastRate += r.alpha * (instantRate - r.lastRate) - } else if newEvents > 0 { + case newEvents > 0: r.init = true r.lastRate = instantRate } diff --git a/storage/remote/queue_manager.go b/storage/remote/queue_manager.go index 62bd17a66d..10fb6d153d 100644 --- a/storage/remote/queue_manager.go +++ b/storage/remote/queue_manager.go @@ -1030,9 +1030,10 @@ func (t *QueueManager) calculateDesiredShards() int { return t.numShards } - if numShards > t.cfg.MaxShards { + switch { + case numShards > t.cfg.MaxShards: numShards = t.cfg.MaxShards - } else if numShards < t.cfg.MinShards { + case numShards < t.cfg.MinShards: numShards = t.cfg.MinShards } return numShards @@ -1575,10 +1576,11 @@ func sendWriteRequestWithBackoff(ctx context.Context, cfg config.QueueConfig, l } sleepDuration = backoff - if backoffErr.retryAfter > 0 { + switch { + case backoffErr.retryAfter > 0: sleepDuration = backoffErr.retryAfter level.Info(l).Log("msg", "Retrying after duration specified by Retry-After header", "duration", sleepDuration) - } else if backoffErr.retryAfter < 0 { + case backoffErr.retryAfter < 0: level.Debug(l).Log("msg", "retry-after cannot be in past, retrying using default backoff mechanism") } diff --git a/tsdb/agent/db.go b/tsdb/agent/db.go index cb075f3060..3343ee18ef 100644 --- a/tsdb/agent/db.go +++ b/tsdb/agent/db.go @@ -951,7 +951,8 @@ func (a *appender) AppendHistogram(ref storage.SeriesRef, l labels.Labels, t int return 0, storage.ErrOutOfOrderSample } - if h != nil { + switch { + case h != nil: // NOTE: always modify pendingHistograms and histogramSeries together a.pendingHistograms = append(a.pendingHistograms, record.RefHistogramSample{ Ref: series.ref, @@ -959,7 +960,7 @@ func (a *appender) AppendHistogram(ref storage.SeriesRef, l labels.Labels, t int H: h, }) a.histogramSeries = append(a.histogramSeries, series) - } else if fh != nil { + case fh != nil: // NOTE: always modify pendingFloatHistograms and floatHistogramSeries together a.pendingFloatHistograms = append(a.pendingFloatHistograms, record.RefFloatHistogramSample{ Ref: series.ref, diff --git a/tsdb/chunkenc/xor.go b/tsdb/chunkenc/xor.go index ba2d96d36b..133ed9952a 100644 --- a/tsdb/chunkenc/xor.go +++ b/tsdb/chunkenc/xor.go @@ -164,14 +164,15 @@ func (a *xorAppender) Append(t int64, v float64) { var tDelta uint64 num := binary.BigEndian.Uint16(a.b.bytes()) - if num == 0 { + switch num { + case 0: buf := make([]byte, binary.MaxVarintLen64) for _, b := range buf[:binary.PutVarint(buf, t)] { a.b.writeByte(b) } a.b.writeBits(math.Float64bits(v), 64) - } else if num == 1 { + case 1: tDelta = uint64(t - a.t) buf := make([]byte, binary.MaxVarintLen64) @@ -181,7 +182,7 @@ func (a *xorAppender) Append(t int64, v float64) { a.writeVDelta(v) - } else { + default: tDelta = uint64(t - a.t) dod := int64(tDelta - a.tDelta) diff --git a/tsdb/chunks/head_chunks.go b/tsdb/chunks/head_chunks.go index a7ff90475e..bcdab21254 100644 --- a/tsdb/chunks/head_chunks.go +++ b/tsdb/chunks/head_chunks.go @@ -999,9 +999,10 @@ func (cdm *ChunkDiskMapper) DeleteCorrupted(originalErr error) error { cdm.readPathMtx.RLock() lastSeq := 0 for seg := range cdm.mmappedChunkFiles { - if seg >= cerr.FileIndex { + switch { + case seg >= cerr.FileIndex: segs = append(segs, seg) - } else if seg > lastSeq { + case seg > lastSeq: lastSeq = seg } } diff --git a/tsdb/db.go b/tsdb/db.go index 659251c3ca..45c0771b1c 100644 --- a/tsdb/db.go +++ b/tsdb/db.go @@ -963,10 +963,11 @@ func (db *DB) ApplyConfig(conf *config.Config) error { // Create WBL if it was not present and if OOO is enabled with WAL enabled. var wblog *wlog.WL var err error - if db.head.wbl != nil { + switch { + case db.head.wbl != nil: // The existing WBL from the disk might have been replayed while OOO was disabled. wblog = db.head.wbl - } else if !db.oooWasEnabled.Load() && oooTimeWindow > 0 && db.opts.WALSegmentSize >= 0 { + case !db.oooWasEnabled.Load() && oooTimeWindow > 0 && db.opts.WALSegmentSize >= 0: segmentSize := wlog.DefaultSegmentSize // Wal is set to a custom size. if db.opts.WALSegmentSize > 0 { @@ -1532,10 +1533,11 @@ func (db *DB) deleteBlocks(blocks map[ulid.ULID]*Block) error { } toDelete := filepath.Join(db.dir, ulid.String()) - if _, err := os.Stat(toDelete); os.IsNotExist(err) { + switch _, err := os.Stat(toDelete); { + case os.IsNotExist(err): // Noop. continue - } else if err != nil { + case err != nil: return errors.Wrapf(err, "stat dir %v", toDelete) } diff --git a/tsdb/head_append.go b/tsdb/head_append.go index 6bc91ae06d..86cb097513 100644 --- a/tsdb/head_append.go +++ b/tsdb/head_append.go @@ -344,9 +344,10 @@ func (a *headAppender) Append(ref storage.SeriesRef, lset labels.Labels, t int64 } if value.IsStaleNaN(v) { - if s.lastHistogramValue != nil { + switch { + case s.lastHistogramValue != nil: return a.AppendHistogram(ref, lset, t, &histogram.Histogram{Sum: v}, nil) - } else if s.lastFloatHistogramValue != nil { + case s.lastFloatHistogramValue != nil: return a.AppendHistogram(ref, lset, t, nil, &histogram.FloatHistogram{Sum: v}) } } @@ -552,9 +553,10 @@ func (a *headAppender) AppendHistogram(ref storage.SeriesRef, lset labels.Labels return 0, err } if created { - if h != nil { + switch { + case h != nil: s.lastHistogramValue = &histogram.Histogram{} - } else if fh != nil { + case fh != nil: s.lastFloatHistogramValue = &histogram.FloatHistogram{} } a.series = append(a.series, record.RefSeries{ @@ -564,7 +566,8 @@ func (a *headAppender) AppendHistogram(ref storage.SeriesRef, lset labels.Labels } } - if h != nil { + switch { + case h != nil: s.Lock() if err := s.appendableHistogram(t, h); err != nil { s.Unlock() @@ -581,7 +584,7 @@ func (a *headAppender) AppendHistogram(ref storage.SeriesRef, lset labels.Labels H: h, }) a.histogramSeries = append(a.histogramSeries, s) - } else if fh != nil { + case fh != nil: s.Lock() if err := s.appendableFloatHistogram(t, fh); err != nil { s.Unlock() @@ -938,7 +941,10 @@ func (a *headAppender) Commit() (err error) { var ok, chunkCreated bool - if err == nil && oooSample { + switch { + case err != nil: + // Do nothing here. + case oooSample: // Sample is OOO and OOO handling is enabled // and the delta is within the OOO tolerance. var mmapRef chunks.ChunkDiskMapperRef @@ -976,7 +982,7 @@ func (a *headAppender) Commit() (err error) { // TODO(codesome): Add error reporting? It depends on addressing https://github.com/prometheus/prometheus/discussions/10305. samplesAppended-- } - } else if err == nil { + default: ok, chunkCreated = series.append(s.T, s.V, a.appendID, a.head.chunkDiskMapper, chunkRange) if ok { if s.T < inOrderMint { @@ -1177,14 +1183,15 @@ func (s *memSeries) appendHistogram(t int64, h *histogram.Histogram, appendID ui app.RecodeHistogram(h, pBackwardInserts, nBackwardInserts) } // We have 3 cases here - // - !okToAppend -> We need to cut a new chunk. + // - !okToAppend or counterReset -> We need to cut a new chunk. // - okToAppend but we have inserts → Existing chunk needs // recoding before we can append our histogram. // - okToAppend and no inserts → Chunk is ready to support our histogram. - if !okToAppend || counterReset { + switch { + case !okToAppend || counterReset: c = s.cutNewHeadChunk(t, chunkenc.EncHistogram, chunkDiskMapper, chunkRange) chunkCreated = true - } else if len(pForwardInserts) > 0 || len(nForwardInserts) > 0 { + case len(pForwardInserts) > 0 || len(nForwardInserts) > 0: // New buckets have appeared. We need to recode all // prior histogram samples within the chunk before we // can process this one. @@ -1270,14 +1277,15 @@ func (s *memSeries) appendFloatHistogram(t int64, fh *histogram.FloatHistogram, app.RecodeHistogramm(fh, pBackwardInserts, nBackwardInserts) } // We have 3 cases here - // - !okToAppend -> We need to cut a new chunk. + // - !okToAppend or counterReset -> We need to cut a new chunk. // - okToAppend but we have inserts → Existing chunk needs // recoding before we can append our histogram. // - okToAppend and no inserts → Chunk is ready to support our histogram. - if !okToAppend || counterReset { + switch { + case !okToAppend || counterReset: c = s.cutNewHeadChunk(t, chunkenc.EncFloatHistogram, chunkDiskMapper, chunkRange) chunkCreated = true - } else if len(pForwardInserts) > 0 || len(nForwardInserts) > 0 { + case len(pForwardInserts) > 0 || len(nForwardInserts) > 0: // New buckets have appeared. We need to recode all // prior histogram samples within the chunk before we // can process this one. diff --git a/tsdb/head_read.go b/tsdb/head_read.go index 9c40bcd7a8..9c546ab164 100644 --- a/tsdb/head_read.go +++ b/tsdb/head_read.go @@ -424,7 +424,8 @@ func (s *memSeries) oooMergedChunk(meta chunks.Meta, cdm *chunks.ChunkDiskMapper break } - if chunkRef == meta.OOOLastRef { + switch { + case chunkRef == meta.OOOLastRef: tmpChks = append(tmpChks, chunkMetaAndChunkDiskMapperRef{ meta: chunks.Meta{ MinTime: meta.OOOLastMinTime, @@ -435,7 +436,7 @@ func (s *memSeries) oooMergedChunk(meta chunks.Meta, cdm *chunks.ChunkDiskMapper origMinT: c.minTime, origMaxT: c.maxTime, }) - } else if c.OverlapsClosedInterval(mint, maxt) { + case c.OverlapsClosedInterval(mint, maxt): tmpChks = append(tmpChks, chunkMetaAndChunkDiskMapperRef{ meta: chunks.Meta{ MinTime: c.minTime, @@ -594,12 +595,14 @@ type boundedIterator struct { func (b boundedIterator) Next() chunkenc.ValueType { for b.Iterator.Next() == chunkenc.ValFloat { t, _ := b.Iterator.At() - if t < b.minT { + switch { + case t < b.minT: continue - } else if t > b.maxT { + case t > b.maxT: return chunkenc.ValNone + default: + return chunkenc.ValFloat } - return chunkenc.ValFloat } return chunkenc.ValNone } diff --git a/tsdb/head_test.go b/tsdb/head_test.go index 39bcf4c78b..a183e99c09 100644 --- a/tsdb/head_test.go +++ b/tsdb/head_test.go @@ -2960,10 +2960,11 @@ func TestAppendHistogram(t *testing.T) { actHistograms := make([]tsdbutil.Sample, 0, len(expHistograms)) actFloatHistograms := make([]tsdbutil.Sample, 0, len(expFloatHistograms)) for typ := it.Next(); typ != chunkenc.ValNone; typ = it.Next() { - if typ == chunkenc.ValHistogram { + switch typ { + case chunkenc.ValHistogram: ts, h := it.AtHistogram() actHistograms = append(actHistograms, sample{t: ts, h: h}) - } else if typ == chunkenc.ValFloatHistogram { + case chunkenc.ValFloatHistogram: ts, fh := it.AtFloatHistogram() actFloatHistograms = append(actFloatHistograms, sample{t: ts, fh: fh}) } @@ -3565,14 +3566,15 @@ func testHistogramStaleSampleHelper(t *testing.T, floatHistogram bool) { for i, eh := range expHistograms { ah := actHistograms[i] if floatHistogram { - if value.IsStaleNaN(eh.fh.Sum) { + switch { + case value.IsStaleNaN(eh.fh.Sum): actNumStale++ require.True(t, value.IsStaleNaN(ah.fh.Sum)) // To make require.Equal work. ah.fh.Sum = 0 eh.fh = eh.fh.Copy() eh.fh.Sum = 0 - } else if i > 0 { + case i > 0: prev := expHistograms[i-1] if prev.fh == nil || value.IsStaleNaN(prev.fh.Sum) { eh.fh.CounterResetHint = histogram.UnknownCounterReset @@ -3580,14 +3582,15 @@ func testHistogramStaleSampleHelper(t *testing.T, floatHistogram bool) { } require.Equal(t, eh, ah) } else { - if value.IsStaleNaN(eh.h.Sum) { + switch { + case value.IsStaleNaN(eh.h.Sum): actNumStale++ require.True(t, value.IsStaleNaN(ah.h.Sum)) // To make require.Equal work. ah.h.Sum = 0 eh.h = eh.h.Copy() eh.h.Sum = 0 - } else if i > 0 { + case i > 0: prev := expHistograms[i-1] if prev.h == nil || value.IsStaleNaN(prev.h.Sum) { eh.h.CounterResetHint = histogram.UnknownCounterReset @@ -4488,19 +4491,19 @@ func TestHistogramValidation(t *testing.T) { for testName, tc := range tests { t.Run(testName, func(t *testing.T) { - err := ValidateHistogram(tc.h) - if tc.errMsg != "" { + switch err := ValidateHistogram(tc.h); { + case tc.errMsg != "": require.ErrorContains(t, err, tc.errMsg) - } else { + default: require.NoError(t, err) } - err = ValidateFloatHistogram(tc.h.ToFloat()) - if tc.errMsgFloat != "" { + switch err := ValidateFloatHistogram(tc.h.ToFloat()); { + case tc.errMsgFloat != "": require.ErrorContains(t, err, tc.errMsgFloat) - } else if tc.errMsg != "" { + case tc.errMsg != "": require.ErrorContains(t, err, tc.errMsg) - } else { + default: require.NoError(t, err) } }) diff --git a/tsdb/index/postings.go b/tsdb/index/postings.go index 15df374fc5..514775210e 100644 --- a/tsdb/index/postings.go +++ b/tsdb/index/postings.go @@ -565,12 +565,11 @@ func newMergedPostings(p []Postings) (m *mergedPostings, nonEmpty bool) { for _, it := range p { // NOTE: mergedPostings struct requires the user to issue an initial Next. - if it.Next() { + switch { + case it.Next(): ph = append(ph, it) - } else { - if it.Err() != nil { - return &mergedPostings{err: it.Err()}, true - } + case it.Err() != nil: + return &mergedPostings{err: it.Err()}, true } } @@ -704,16 +703,16 @@ func (rp *removedPostings) Next() bool { return true } - fcur, rcur := rp.full.At(), rp.remove.At() - if fcur < rcur { + switch fcur, rcur := rp.full.At(), rp.remove.At(); { + case fcur < rcur: rp.cur = fcur rp.fok = rp.full.Next() return true - } else if rcur < fcur { + case rcur < fcur: // Forward the remove postings to the right position. rp.rok = rp.remove.Seek(fcur) - } else { + default: // Skip the current posting. rp.fok = rp.full.Next() } @@ -848,9 +847,10 @@ func (it *bigEndianPostings) Err() error { func FindIntersectingPostings(p Postings, candidates []Postings) (indexes []int, err error) { h := make(postingsWithIndexHeap, 0, len(candidates)) for idx, it := range candidates { - if it.Next() { + switch { + case it.Next(): h = append(h, postingsWithIndex{index: idx, p: it}) - } else if it.Err() != nil { + case it.Err() != nil: return nil, it.Err() } } diff --git a/tsdb/ooo_head_read.go b/tsdb/ooo_head_read.go index fd16c6ca8f..8ba3ea39af 100644 --- a/tsdb/ooo_head_read.go +++ b/tsdb/ooo_head_read.go @@ -123,7 +123,7 @@ func (oh *OOOHeadIndexReader) series(ref storage.SeriesRef, builder *labels.Scra } } - // There is nothing to do if we did not collect any chunk + // There is nothing to do if we did not collect any chunk. if len(tmpChks) == 0 { return nil } @@ -136,14 +136,15 @@ func (oh *OOOHeadIndexReader) series(ref storage.SeriesRef, builder *labels.Scra // chunks Meta the first chunk that overlaps with others. // Example chunks of a series: 5:(100, 200) 6:(500, 600) 7:(150, 250) 8:(550, 650) // In the example 5 overlaps with 7 and 6 overlaps with 8 so we only want to - // to return chunk Metas for chunk 5 and chunk 6 + // to return chunk Metas for chunk 5 and chunk 6e *chks = append(*chks, tmpChks[0]) - maxTime := tmpChks[0].MaxTime // tracks the maxTime of the previous "to be merged chunk" + maxTime := tmpChks[0].MaxTime // Tracks the maxTime of the previous "to be merged chunk". for _, c := range tmpChks[1:] { - if c.MinTime > maxTime { + switch { + case c.MinTime > maxTime: *chks = append(*chks, c) maxTime = c.MaxTime - } else if c.MaxTime > maxTime { + case c.MaxTime > maxTime: maxTime = c.MaxTime (*chks)[len(*chks)-1].MaxTime = c.MaxTime } diff --git a/tsdb/querier.go b/tsdb/querier.go index f3852d517b..abddebb870 100644 --- a/tsdb/querier.go +++ b/tsdb/querier.go @@ -239,18 +239,20 @@ func PostingsForMatchers(ix IndexReader, ms ...*labels.Matcher) (index.Postings, } for _, m := range ms { - if m.Name == "" && m.Value == "" { // Special-case for AllPostings, used in tests at least. + switch { + case m.Name == "" && m.Value == "": // Special-case for AllPostings, used in tests at least. k, v := index.AllPostingsKey() allPostings, err := ix.Postings(k, v) if err != nil { return nil, err } its = append(its, allPostings) - } else if labelMustBeSet[m.Name] { + case labelMustBeSet[m.Name]: // If this matcher must be non-empty, we can be smarter. matchesEmpty := m.Matches("") isNot := m.Type == labels.MatchNotEqual || m.Type == labels.MatchNotRegexp - if isNot && matchesEmpty { // l!="foo" + switch { + case isNot && matchesEmpty: // l!="foo" // If the label can't be empty and is a Not and the inner matcher // doesn't match empty, then subtract it out at the end. inverse, err := m.Inverse() @@ -263,7 +265,7 @@ func PostingsForMatchers(ix IndexReader, ms ...*labels.Matcher) (index.Postings, return nil, err } notIts = append(notIts, it) - } else if isNot && !matchesEmpty { // l!="" + case isNot && !matchesEmpty: // l!="" // If the label can't be empty and is a Not, but the inner matcher can // be empty we need to use inversePostingsForMatcher. inverse, err := m.Inverse() @@ -279,7 +281,7 @@ func PostingsForMatchers(ix IndexReader, ms ...*labels.Matcher) (index.Postings, return index.EmptyPostings(), nil } its = append(its, it) - } else { // l="a" + default: // l="a" // Non-Not matcher, use normal postingsForMatcher. it, err := postingsForMatcher(ix, m) if err != nil { @@ -290,7 +292,7 @@ func PostingsForMatchers(ix IndexReader, ms ...*labels.Matcher) (index.Postings, } its = append(its, it) } - } else { // l="" + default: // l="" // If the matchers for a labelname selects an empty value, it selects all // the series which don't have the label name set too. See: // https://github.com/prometheus/prometheus/issues/3575 and @@ -966,23 +968,24 @@ func (m *mergedStringIter) Next() bool { return false } - if !m.aok { + switch { + case !m.aok: m.cur = m.b.At() m.bok = m.b.Next() m.err = m.b.Err() - } else if !m.bok { + case !m.bok: m.cur = m.a.At() m.aok = m.a.Next() m.err = m.a.Err() - } else if m.b.At() > m.a.At() { + case m.b.At() > m.a.At(): m.cur = m.a.At() m.aok = m.a.Next() m.err = m.a.Err() - } else if m.a.At() > m.b.At() { + case m.a.At() > m.b.At(): m.cur = m.b.At() m.bok = m.b.Next() m.err = m.b.Err() - } else { // Equal. + default: // Equal. m.cur = m.b.At() m.aok = m.a.Next() m.err = m.a.Err() diff --git a/tsdb/tombstones/tombstones.go b/tsdb/tombstones/tombstones.go index f7e2a2a1e7..a52e1caa97 100644 --- a/tsdb/tombstones/tombstones.go +++ b/tsdb/tombstones/tombstones.go @@ -190,9 +190,10 @@ type Stone struct { func ReadTombstones(dir string) (Reader, int64, error) { b, err := os.ReadFile(filepath.Join(dir, TombstonesFilename)) - if os.IsNotExist(err) { + switch { + case os.IsNotExist(err): return NewMemTombstones(), 0, nil - } else if err != nil { + case err != nil: return nil, 0, err } diff --git a/tsdb/wal.go b/tsdb/wal.go index 93d445a126..0c57865e66 100644 --- a/tsdb/wal.go +++ b/tsdb/wal.go @@ -522,9 +522,10 @@ func (w *SegmentWAL) openSegmentFile(name string) (*os.File, error) { } }() - if n, err := f.Read(metab); err != nil { + switch n, err := f.Read(metab); { + case err != nil: return nil, errors.Wrapf(err, "validate meta %q", f.Name()) - } else if n != 8 { + case n != 8: return nil, errors.Errorf("invalid header size %d in %q", n, f.Name()) } @@ -1063,9 +1064,10 @@ func (r *walReader) entry(cr io.Reader) (WALEntryType, byte, []byte, error) { tr := io.TeeReader(cr, r.crc32) b := make([]byte, 6) - if n, err := tr.Read(b); err != nil { + switch n, err := tr.Read(b); { + case err != nil: return 0, 0, nil, err - } else if n != 6 { + case n != 6: return 0, 0, nil, r.corruptionErr("invalid entry header size %d", n) } @@ -1087,15 +1089,17 @@ func (r *walReader) entry(cr io.Reader) (WALEntryType, byte, []byte, error) { } buf := r.buf[:length] - if n, err := tr.Read(buf); err != nil { + switch n, err := tr.Read(buf); { + case err != nil: return 0, 0, nil, err - } else if n != length { + case n != length: return 0, 0, nil, r.corruptionErr("invalid entry body size %d", n) } - if n, err := cr.Read(b[:4]); err != nil { + switch n, err := cr.Read(b[:4]); { + case err != nil: return 0, 0, nil, err - } else if n != 4 { + case n != 4: return 0, 0, nil, r.corruptionErr("invalid checksum length %d", n) } if exp, has := binary.BigEndian.Uint32(b[:4]), r.crc32.Sum32(); has != exp { diff --git a/tsdb/wlog/live_reader.go b/tsdb/wlog/live_reader.go index 29467aef4a..0ca69093a5 100644 --- a/tsdb/wlog/live_reader.go +++ b/tsdb/wlog/live_reader.go @@ -126,9 +126,10 @@ func (r *LiveReader) Next() bool { // we return EOF and the user can try again later. If we have a full // page, buildRecord is guaranteed to return a record or a non-EOF; it // has checks the records fit in pages. - if ok, err := r.buildRecord(); ok { + switch ok, err := r.buildRecord(); { + case ok: return true - } else if err != nil && err != io.EOF { + case err != nil && err != io.EOF: r.err = err return false } diff --git a/tsdb/wlog/watcher.go b/tsdb/wlog/watcher.go index 72121283d8..b0c17dcbac 100644 --- a/tsdb/wlog/watcher.go +++ b/tsdb/wlog/watcher.go @@ -405,9 +405,10 @@ func (w *Watcher) watch(segmentNum int, tail bool) error { // Ignore errors reading to end of segment whilst replaying the WAL. if !tail { - if err != nil && errors.Cause(err) != io.EOF { + switch { + case err != nil && errors.Cause(err) != io.EOF: level.Warn(w.logger).Log("msg", "Ignoring error reading to end of segment, may have dropped data", "err", err) - } else if reader.Offset() != size { + case reader.Offset() != size: level.Warn(w.logger).Log("msg", "Expected to have read whole segment, may have dropped data", "segment", segmentNum, "read", reader.Offset(), "size", size) } return nil @@ -425,9 +426,10 @@ func (w *Watcher) watch(segmentNum int, tail bool) error { // Ignore all errors reading to end of segment whilst replaying the WAL. if !tail { - if err != nil && errors.Cause(err) != io.EOF { + switch { + case err != nil && errors.Cause(err) != io.EOF: level.Warn(w.logger).Log("msg", "Ignoring error reading to end of segment, may have dropped data", "segment", segmentNum, "err", err) - } else if reader.Offset() != size { + case reader.Offset() != size: level.Warn(w.logger).Log("msg", "Expected to have read whole segment, may have dropped data", "segment", segmentNum, "read", reader.Offset(), "size", size) } return nil diff --git a/util/treecache/treecache.go b/util/treecache/treecache.go index acdd6f7bea..bece9d5c83 100644 --- a/util/treecache/treecache.go +++ b/util/treecache/treecache.go @@ -176,11 +176,11 @@ func (tc *ZookeeperTreeCache) loop(path string) { node = childNode } - err := tc.recursiveNodeUpdate(ev.Path, node) - if err != nil { + switch err := tc.recursiveNodeUpdate(ev.Path, node); { + case err != nil: level.Error(tc.logger).Log("msg", "Error during processing of Zookeeper event", "err", err) failure() - } else if tc.head.data == nil { + case tc.head.data == nil: level.Error(tc.logger).Log("msg", "Error during processing of Zookeeper event", "err", "path no longer exists", "path", tc.prefix) failure() } @@ -214,13 +214,14 @@ func (tc *ZookeeperTreeCache) loop(path string) { func (tc *ZookeeperTreeCache) recursiveNodeUpdate(path string, node *zookeeperTreeCacheNode) error { data, _, dataWatcher, err := tc.conn.GetW(path) - if errors.Is(err, zk.ErrNoNode) { + switch { + case errors.Is(err, zk.ErrNoNode): tc.recursiveDelete(path, node) if node == tc.head { return fmt.Errorf("path %s does not exist", path) } return nil - } else if err != nil { + case err != nil: return err } @@ -230,10 +231,11 @@ func (tc *ZookeeperTreeCache) recursiveNodeUpdate(path string, node *zookeeperTr } children, _, childWatcher, err := tc.conn.ChildrenW(path) - if errors.Is(err, zk.ErrNoNode) { + switch { + case errors.Is(err, zk.ErrNoNode): tc.recursiveDelete(path, node) return nil - } else if err != nil { + case err != nil: return err } diff --git a/web/api/v1/api.go b/web/api/v1/api.go index 0624cf2d8b..f4d6d27da1 100644 --- a/web/api/v1/api.go +++ b/web/api/v1/api.go @@ -989,12 +989,14 @@ func (api *API) targets(r *http.Request) apiFuncResult { ScrapeURL: target.URL().String(), GlobalURL: globalURL.String(), LastError: func() string { - if err == nil && lastErrStr == "" { + switch { + case err == nil && lastErrStr == "": return "" - } else if err != nil { + case err != nil: return errors.Wrapf(err, lastErrStr).Error() + default: + return lastErrStr } - return lastErrStr }(), LastScrape: target.LastScrape(), LastScrapeDuration: target.LastScrapeDuration().Seconds(), diff --git a/web/federate_test.go b/web/federate_test.go index 76d4b9cf67..61ef62f46d 100644 --- a/web/federate_test.go +++ b/web/federate_test.go @@ -388,13 +388,13 @@ func TestFederationWithNativeHistograms(t *testing.T) { break } require.NoError(t, err) - if et == textparse.EntryHelp { - metricFamilies++ - } if et == textparse.EntryHistogram || et == textparse.EntrySeries { p.Metric(&l) } - if et == textparse.EntryHistogram { + switch et { + case textparse.EntryHelp: + metricFamilies++ + case textparse.EntryHistogram: _, parsedTimestamp, h, fh := p.Histogram() require.Nil(t, h) actVec = append(actVec, promql.Sample{ @@ -402,7 +402,7 @@ func TestFederationWithNativeHistograms(t *testing.T) { H: fh, Metric: l, }) - } else if et == textparse.EntrySeries { + case textparse.EntrySeries: _, parsedTimestamp, f := p.Series() actVec = append(actVec, promql.Sample{ T: *parsedTimestamp, From 1f044154940dfc380e9ce6dd70bae3dc772af14c Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Tue, 18 Apr 2023 11:14:54 +0200 Subject: [PATCH 105/231] Update many Go dependencies Ran `make update-go-deps` then hand-edited to remove any downgrades. Then backed out changes to: * Azure: still waiting someone to test this. * Kubernetes: needs update elsewhere to klog. * kube-openapi: it doesn't compile with previous Go version. Signed-off-by: Bryan Boreham --- go.mod | 50 +++++++++++++-------------- go.sum | 104 +++++++++++++++++++++++++++++---------------------------- 2 files changed, 78 insertions(+), 76 deletions(-) diff --git a/go.mod b/go.mod index 92cc3931c7..7b309b03f5 100644 --- a/go.mod +++ b/go.mod @@ -8,46 +8,46 @@ require ( github.com/Azure/go-autorest/autorest/adal v0.9.23 github.com/alecthomas/kingpin/v2 v2.3.2 github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 - github.com/aws/aws-sdk-go v1.44.217 + github.com/aws/aws-sdk-go v1.44.245 github.com/cespare/xxhash/v2 v2.2.0 github.com/dennwc/varint v1.0.0 github.com/digitalocean/godo v1.98.0 - github.com/docker/docker v23.0.1+incompatible + github.com/docker/docker v23.0.4+incompatible github.com/edsrzf/mmap-go v1.1.0 github.com/envoyproxy/go-control-plane v0.11.0 - github.com/envoyproxy/protoc-gen-validate v0.9.1 + github.com/envoyproxy/protoc-gen-validate v0.10.1 github.com/fsnotify/fsnotify v1.6.0 github.com/go-kit/log v0.2.1 github.com/go-logfmt/logfmt v0.6.0 - github.com/go-openapi/strfmt v0.21.3 + github.com/go-openapi/strfmt v0.21.7 github.com/go-zookeeper/zk v1.0.3 github.com/gogo/protobuf v1.3.2 github.com/golang/snappy v0.0.4 - github.com/google/pprof v0.0.0-20230228050547-1710fef4ab10 - github.com/gophercloud/gophercloud v1.2.0 + github.com/google/pprof v0.0.0-20230406165453-00490a63f317 + github.com/gophercloud/gophercloud v1.3.0 github.com/grafana/regexp v0.0.0-20221122212121-6b5c0a4cb7fd github.com/grpc-ecosystem/grpc-gateway v1.16.0 github.com/hashicorp/consul/api v1.20.0 - github.com/hashicorp/nomad/api v0.0.0-20230308192510-48e7d70fcd4b - github.com/hetznercloud/hcloud-go v1.41.0 - github.com/ionos-cloud/sdk-go/v6 v6.1.5 + github.com/hashicorp/nomad/api v0.0.0-20230418003350-3067191c5197 + github.com/hetznercloud/hcloud-go v1.42.0 + github.com/ionos-cloud/sdk-go/v6 v6.1.6 github.com/json-iterator/go v1.1.12 github.com/kolo/xmlrpc v0.0.0-20220921171641-a4b6fa1dd06b - github.com/linode/linodego v1.14.1 + github.com/linode/linodego v1.16.1 github.com/miekg/dns v1.1.53 github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f github.com/oklog/run v1.1.0 github.com/oklog/ulid v1.3.1 - github.com/ovh/go-ovh v1.3.0 + github.com/ovh/go-ovh v1.4.1 github.com/pkg/errors v0.9.1 github.com/prometheus/alertmanager v0.25.0 - github.com/prometheus/client_golang v1.14.0 + github.com/prometheus/client_golang v1.15.0 github.com/prometheus/client_model v0.3.0 github.com/prometheus/common v0.42.0 github.com/prometheus/common/assets v0.2.0 github.com/prometheus/common/sigv4 v0.1.0 github.com/prometheus/exporter-toolkit v0.9.1 - github.com/scaleway/scaleway-sdk-go v1.0.0-beta.14 + github.com/scaleway/scaleway-sdk-go v1.0.0-beta.15 github.com/shurcooL/httpfs v0.0.0-20190707220628-8d4bc4ba7749 github.com/stretchr/testify v1.8.2 github.com/vultr/govultr/v2 v2.17.2 @@ -59,18 +59,18 @@ require ( go.opentelemetry.io/otel/sdk v1.14.0 go.opentelemetry.io/otel/trace v1.14.0 go.uber.org/atomic v1.10.0 - go.uber.org/automaxprocs v1.5.1 + go.uber.org/automaxprocs v1.5.2 go.uber.org/goleak v1.2.1 - golang.org/x/net v0.8.0 - golang.org/x/oauth2 v0.6.0 + golang.org/x/net v0.9.0 + golang.org/x/oauth2 v0.7.0 golang.org/x/sync v0.1.0 - golang.org/x/sys v0.6.0 + golang.org/x/sys v0.7.0 golang.org/x/time v0.3.0 - golang.org/x/tools v0.7.0 + golang.org/x/tools v0.8.0 google.golang.org/api v0.114.0 google.golang.org/genproto v0.0.0-20230306155012-7f2fa6fef1f4 google.golang.org/grpc v1.53.0 - google.golang.org/protobuf v1.29.1 + google.golang.org/protobuf v1.30.0 gopkg.in/yaml.v2 v2.4.0 gopkg.in/yaml.v3 v3.0.1 k8s.io/api v0.26.2 @@ -85,7 +85,7 @@ require ( github.com/coreos/go-systemd/v22 v22.5.0 // indirect github.com/hashicorp/errwrap v1.1.0 // indirect github.com/hashicorp/go-multierror v1.1.1 // indirect - github.com/rogpeppe/go-internal v1.9.0 // indirect + github.com/rogpeppe/go-internal v1.10.0 // indirect github.com/xhit/go-str2duration/v2 v2.1.0 // indirect ) @@ -166,16 +166,16 @@ require ( github.com/pmezard/go-difflib v1.0.0 // indirect github.com/prometheus/procfs v0.9.0 // indirect github.com/spf13/pflag v1.0.5 // indirect - go.mongodb.org/mongo-driver v1.11.2 // indirect + go.mongodb.org/mongo-driver v1.11.3 // indirect go.opencensus.io v0.24.0 // indirect go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.14.0 // indirect go.opentelemetry.io/otel/metric v0.37.0 // indirect go.opentelemetry.io/proto/otlp v0.19.0 // indirect golang.org/x/crypto v0.7.0 // indirect - golang.org/x/exp v0.0.0-20230307190834-24139beb5833 - golang.org/x/mod v0.9.0 // indirect - golang.org/x/term v0.6.0 // indirect - golang.org/x/text v0.8.0 // indirect + golang.org/x/exp v0.0.0-20230321023759-10a507213a29 + golang.org/x/mod v0.10.0 // indirect + golang.org/x/term v0.7.0 // indirect + golang.org/x/text v0.9.0 // indirect google.golang.org/appengine v1.6.7 // indirect gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/ini.v1 v1.67.0 // indirect diff --git a/go.sum b/go.sum index 7ea1be034d..c699aa9dff 100644 --- a/go.sum +++ b/go.sum @@ -98,8 +98,8 @@ github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2/go.mod h1:W github.com/aws/aws-lambda-go v1.13.3/go.mod h1:4UKl9IzQMoD+QF79YdCuzCwp8VbmG4VAQwij/eHl5CU= github.com/aws/aws-sdk-go v1.27.0/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo= github.com/aws/aws-sdk-go v1.38.35/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro= -github.com/aws/aws-sdk-go v1.44.217 h1:FcWC56MRl+k756aH3qeMQTylSdeJ58WN0iFz3fkyRz0= -github.com/aws/aws-sdk-go v1.44.217/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI= +github.com/aws/aws-sdk-go v1.44.245 h1:KtY2s4q31/kn33AdV63R5t77mdxsI7rq3YT7Mgo805M= +github.com/aws/aws-sdk-go v1.44.245/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI= github.com/aws/aws-sdk-go-v2 v0.18.0/go.mod h1:JWVYvqSMppoMJC0x5wdwiImzgXTI9FuZwxzkQq9wy+g= github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8= @@ -154,8 +154,8 @@ github.com/digitalocean/godo v1.98.0/go.mod h1:NRpFznZFvhHjBoqZAaOD3khVzsJ3EibzK github.com/dnaeon/go-vcr v1.2.0 h1:zHCHvJYTMh1N7xnV7zf1m1GPBF9Ad0Jk/whtQ1663qI= github.com/docker/distribution v2.8.1+incompatible h1:Q50tZOPR6T/hjNsyc9g8/syEs6bk8XXApsHjKukMl68= github.com/docker/distribution v2.8.1+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w= -github.com/docker/docker v23.0.1+incompatible h1:vjgvJZxprTTE1A37nm+CLNAdwu6xZekyoiVlUZEINcY= -github.com/docker/docker v23.0.1+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= +github.com/docker/docker v23.0.4+incompatible h1:Kd3Bh9V/rO+XpTP/BLqM+gx8z7+Yb0AA2Ibj+nNo4ek= +github.com/docker/docker v23.0.4+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= github.com/docker/go-connections v0.4.0 h1:El9xVISelRB7BuFusrZozjnkIM5YnzCViNKohAFqRJQ= github.com/docker/go-connections v0.4.0/go.mod h1:Gbd7IOopHjR8Iph03tsViu4nIes5XhDvyHbTtUxmeec= github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4= @@ -180,8 +180,8 @@ github.com/envoyproxy/go-control-plane v0.9.10-0.20210907150352-cf90f659a021/go. github.com/envoyproxy/go-control-plane v0.11.0 h1:jtLewhRR2vMRNnq2ZZUoCjUlgut+Y0+sDDWPOfwOi1o= github.com/envoyproxy/go-control-plane v0.11.0/go.mod h1:VnHyVMpzcLvCFt9yUz1UnCwHLhwx1WguiVDV7pTG/tI= github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= -github.com/envoyproxy/protoc-gen-validate v0.9.1 h1:PS7VIOgmSVhWUEeZwTe7z7zouA22Cr590PzXKbZHOVY= -github.com/envoyproxy/protoc-gen-validate v0.9.1/go.mod h1:OKNgG7TCp5pF4d6XftA0++PMirau2/yoOwVac3AbF2w= +github.com/envoyproxy/protoc-gen-validate v0.10.1 h1:c0g45+xCJhdgFGw7a5QAfdS4byAbud7miNWJ1WwEVf8= +github.com/envoyproxy/protoc-gen-validate v0.10.1/go.mod h1:DRjgyB0I43LtJapqN6NiRwroiAU2PaFuvk/vjgh61ss= github.com/evanphx/json-patch v4.12.0+incompatible h1:4onqiflcdA9EOZ4RxV643DvftH5pOlLGNtQ5lPWQu84= github.com/evanphx/json-patch v4.12.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4= @@ -245,8 +245,9 @@ github.com/go-openapi/spec v0.20.8 h1:ubHmXNY3FCIOinT8RNrrPfGc9t7I1qhPtdOGoG2AxR github.com/go-openapi/spec v0.20.8/go.mod h1:2OpW+JddWPrpXSCIX8eOx7lZ5iyuWj3RYR6VaaBKcWA= github.com/go-openapi/strfmt v0.21.0/go.mod h1:ZRQ409bWMj+SOgXofQAGTIo2Ebu72Gs+WaRADcS5iNg= github.com/go-openapi/strfmt v0.21.1/go.mod h1:I/XVKeLc5+MM5oPNN7P6urMOpuLXEcNrCX/rPGuWb0k= -github.com/go-openapi/strfmt v0.21.3 h1:xwhj5X6CjXEZZHMWy1zKJxvW9AfHC9pkyUjLvHtKG7o= github.com/go-openapi/strfmt v0.21.3/go.mod h1:k+RzNO0Da+k3FrrynSNN8F7n/peCmQQqbbXjtDfvmGg= +github.com/go-openapi/strfmt v0.21.7 h1:rspiXgNWgeUzhjo1YU01do6qsahtJNByjLVbPLNHb8k= +github.com/go-openapi/strfmt v0.21.7/go.mod h1:adeGTkxE44sPyLk0JV235VQAO/ZXUr8KAzYjclFs3ew= github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk= github.com/go-openapi/swag v0.19.15/go.mod h1:QYRuS/SOXUCsnplDa677K7+DxSOj6IPNl/eQntq43wQ= github.com/go-openapi/swag v0.21.1/go.mod h1:QYRuS/SOXUCsnplDa677K7+DxSOj6IPNl/eQntq43wQ= @@ -366,8 +367,8 @@ github.com/google/pprof v0.0.0-20200212024743-f11f1df84d12/go.mod h1:ZgVRPoUq/hf github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= github.com/google/pprof v0.0.0-20200430221834-fc25d7d30c6d/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= github.com/google/pprof v0.0.0-20200708004538-1a94d8640e99/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= -github.com/google/pprof v0.0.0-20230228050547-1710fef4ab10 h1:CqYfpuYIjnlNxM3msdyPRKabhXZWbKjf3Q8BWROFBso= -github.com/google/pprof v0.0.0-20230228050547-1710fef4ab10/go.mod h1:79YE0hCXdHag9sBkw2o+N/YnZtTkXi0UT9Nnixa5eYk= +github.com/google/pprof v0.0.0-20230406165453-00490a63f317 h1:hFhpt7CTmR3DX+b4R19ydQFtofxT0Sv3QsKNMVQYTMQ= +github.com/google/pprof v0.0.0-20230406165453-00490a63f317/go.mod h1:79YE0hCXdHag9sBkw2o+N/YnZtTkXi0UT9Nnixa5eYk= github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= @@ -380,8 +381,8 @@ github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+ github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= github.com/googleapis/gax-go/v2 v2.7.1 h1:gF4c0zjUP2H/s/hEGyLA3I0fA2ZWjzYiONAD6cvPr8A= github.com/googleapis/gax-go/v2 v2.7.1/go.mod h1:4orTrqY6hXxxaUL4LHIPl6lGo8vAE38/qKbhSAKP6QI= -github.com/gophercloud/gophercloud v1.2.0 h1:1oXyj4g54KBg/kFtCdMM6jtxSzeIyg8wv4z1HoGPp1E= -github.com/gophercloud/gophercloud v1.2.0/go.mod h1:aAVqcocTSXh2vYFZ1JTvx4EQmfgzxRcNupUfxZbBNDM= +github.com/gophercloud/gophercloud v1.3.0 h1:RUKyCMiZoQR3VlVR5E3K7PK1AC3/qppsWYo6dtBiqs8= +github.com/gophercloud/gophercloud v1.3.0/go.mod h1:aAVqcocTSXh2vYFZ1JTvx4EQmfgzxRcNupUfxZbBNDM= github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY= github.com/gorilla/context v1.1.1/go.mod h1:kBGZzfjB9CEq2AlWe17Uuf7NDRt0dE0s8S51q0aT7Yg= github.com/gorilla/mux v1.6.2/go.mod h1:1lud6UwP+6orDFRuTfBEV8e9/aOM/c4fVVCaMa2zaAs= @@ -450,13 +451,13 @@ github.com/hashicorp/mdns v1.0.4/go.mod h1:mtBihi+LeNXGtG8L9dX59gAEa12BDtBQSp4v/ github.com/hashicorp/memberlist v0.1.3/go.mod h1:ajVTdAv/9Im8oMAAj5G31PhhMCZJV2pPBoIllUwCN7I= github.com/hashicorp/memberlist v0.5.0 h1:EtYPN8DpAURiapus508I4n9CzHs2W+8NZGbmmR/prTM= github.com/hashicorp/memberlist v0.5.0/go.mod h1:yvyXLpo0QaGE59Y7hDTsTzDD25JYBZ4mHgHUZ8lrOI0= -github.com/hashicorp/nomad/api v0.0.0-20230308192510-48e7d70fcd4b h1:EkuSTU8c/63q4LMayj8ilgg/4I5PXDFVcnqKfs9qcwI= -github.com/hashicorp/nomad/api v0.0.0-20230308192510-48e7d70fcd4b/go.mod h1:bKUb1ytds5KwUioHdvdq9jmrDqCThv95si0Ub7iNeBg= +github.com/hashicorp/nomad/api v0.0.0-20230418003350-3067191c5197 h1:I5xhKLePXpXgM6pZ4xZNTiurLLS3sGuZrZFFzAbM67A= +github.com/hashicorp/nomad/api v0.0.0-20230418003350-3067191c5197/go.mod h1:2TCrNvonL09r7EiQ6M2rNt+Cmjbn1QbzchFoTWJFpj4= github.com/hashicorp/serf v0.8.2/go.mod h1:6hOLApaqBFA1NXqRQAsxw9QxuDEvNxSQRwA/JwenrHc= github.com/hashicorp/serf v0.10.1 h1:Z1H2J60yRKvfDYAOZLd2MU0ND4AH/WDz7xYHDWQsIPY= github.com/hashicorp/serf v0.10.1/go.mod h1:yL2t6BqATOLGc5HF7qbFkTfXoPIY0WZdWHfEvMqbG+4= -github.com/hetznercloud/hcloud-go v1.41.0 h1:KJGFRRc68QiVu4PrEP5BmCQVveCP2CM26UGQUKGpIUs= -github.com/hetznercloud/hcloud-go v1.41.0/go.mod h1:NaHg47L6C77mngZhwBG652dTAztYrsZ2/iITJKhQkHA= +github.com/hetznercloud/hcloud-go v1.42.0 h1:Es/CDOForQN3nOOP5Vxh1N/YHjpCg386iYEX5zCgi+A= +github.com/hetznercloud/hcloud-go v1.42.0/go.mod h1:YADL8AbmQYH0Eo+1lkuyoc8LutT0UeMvaKP47nNUb+Y= github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= github.com/hudl/fargo v1.3.0/go.mod h1:y3CKSmjA+wD2gak7sUSXTAoopbhU08POFhmITJgmKTg= github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc= @@ -464,8 +465,9 @@ github.com/imdario/mergo v0.3.13 h1:lFzP57bqS/wsqKssCGmtLAb8A0wKjLGrve2q3PPVcBk= github.com/imdario/mergo v0.3.13/go.mod h1:4lJ1jqUDcsbIECGy0RUJAXNIhg+6ocWgb1ALK2O4oXg= github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8= github.com/influxdata/influxdb1-client v0.0.0-20191209144304-8bf82d3c094d/go.mod h1:qj24IKcXYK6Iy9ceXlo3Tc+vtHo9lIhSX5JddghvEPo= -github.com/ionos-cloud/sdk-go/v6 v6.1.5 h1:BFqThLOgrGJWeo7w6UDyYuNxyi/GqEmNPl7C/YcQ8Fw= -github.com/ionos-cloud/sdk-go/v6 v6.1.5/go.mod h1:EzEgRIDxBELvfoa/uBN0kOQaqovLjUWEB7iW4/Q+t4k= +github.com/ionos-cloud/sdk-go/v6 v6.1.6 h1:0n4irdqNska+1s3YMCRhrAqKbibEgQ7SwwhAlHzYT5A= +github.com/ionos-cloud/sdk-go/v6 v6.1.6/go.mod h1:EzEgRIDxBELvfoa/uBN0kOQaqovLjUWEB7iW4/Q+t4k= +github.com/jarcoal/httpmock v1.3.0 h1:2RJ8GP0IIaWwcC9Fp2BmVi8Kog3v2Hn7VXM3fTd+nuc= github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k= github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg= github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo= @@ -513,8 +515,8 @@ github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= github.com/lightstep/lightstep-tracer-common/golang/gogo v0.0.0-20190605223551-bc2310a04743/go.mod h1:qklhhLq1aX+mtWk9cPHPzaBjWImj5ULL6C7HFJtXQMM= github.com/lightstep/lightstep-tracer-go v0.18.1/go.mod h1:jlF1pusYV4pidLvZ+XD0UBX0ZE6WURAspgAczcDHrL4= -github.com/linode/linodego v1.14.1 h1:uGxQyy0BidoEpLGdvfi4cPgEW+0YUFsEGrLEhcTfjNc= -github.com/linode/linodego v1.14.1/go.mod h1:NJlzvlNtdMRRkXb0oN6UWzUkj6t+IBsyveHgZ5Ppjyk= +github.com/linode/linodego v1.16.1 h1:5otq57M4PdHycPERRfSFZ0s1yz1ETVWGjCp3hh7+F9w= +github.com/linode/linodego v1.16.1/go.mod h1:aESRAbpLY9R6IA1WGAWHikRI9DU9Lhesapv1MhKmPHM= github.com/lyft/protoc-gen-validate v0.0.13/go.mod h1:XbGvPuh87YZc5TdIa2/I4pLk0QoUACkjt2znoq26NVQ= github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= @@ -543,6 +545,7 @@ github.com/mattn/go-runewidth v0.0.2/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzp github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0= github.com/matttproud/golang_protobuf_extensions v1.0.4 h1:mmDVorXM7PCGKw94cs5zkfA9PSy5pEvNWRP0ET0TIVo= github.com/matttproud/golang_protobuf_extensions v1.0.4/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4= +github.com/maxatome/go-testdeep v1.12.0 h1:Ql7Go8Tg0C1D/uMMX59LAoYK7LffeJQ6X2T04nTH68g= github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg= github.com/miekg/dns v1.1.26/go.mod h1:bPDLeHnStXmXAq1m/Ch/hvfNHr14JKNPMBo3VZKjuso= github.com/miekg/dns v1.1.41/go.mod h1:p6aan82bvRIyn+zDIv9xYNUpwa73JcSh9BKwknJysuI= @@ -614,8 +617,8 @@ github.com/openzipkin-contrib/zipkin-go-opentracing v0.4.5/go.mod h1:/wsWhb9smxS github.com/openzipkin/zipkin-go v0.1.6/go.mod h1:QgAqvLzwWbR/WpD4A3cGpPtJrZXNIiJc5AZX7/PBEpw= github.com/openzipkin/zipkin-go v0.2.1/go.mod h1:NaW6tEwdmWMaCDZzg8sh+IBNOxHMPnhQw8ySjnjRyN4= github.com/openzipkin/zipkin-go v0.2.2/go.mod h1:NaW6tEwdmWMaCDZzg8sh+IBNOxHMPnhQw8ySjnjRyN4= -github.com/ovh/go-ovh v1.3.0 h1:mvZaddk4E4kLcXhzb+cxBsMPYp2pHqiQpWYkInsuZPQ= -github.com/ovh/go-ovh v1.3.0/go.mod h1:AxitLZ5HBRPyUd+Zl60Ajaag+rNTdVXWIkzfrVuTXWA= +github.com/ovh/go-ovh v1.4.1 h1:VBGa5wMyQtTP7Zb+w97zRCh9sLtM/2YKRyy+MEJmWaM= +github.com/ovh/go-ovh v1.4.1/go.mod h1:6bL6pPyUT7tBfI0pqOegJgRjgjuO+mOo+MyXd1EEC0M= github.com/pact-foundation/pact-go v1.0.4/go.mod h1:uExwJY4kCzNPcHRj+hCR/HBbOOIwwtUjcrb0b5/5kLM= github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc= github.com/pascaldekloe/goe v0.1.0 h1:cBOtyMzM9HTpWjXfbbunk26uA6nG3a8n06Wieeh0MwY= @@ -644,8 +647,8 @@ github.com/prometheus/client_golang v1.3.0/go.mod h1:hJaj2vgQTGQmVCsAACORcieXFeD github.com/prometheus/client_golang v1.4.0/go.mod h1:e9GMxYsXl05ICDXkRhurwBS4Q3OK1iX/F2sw+iXX5zU= github.com/prometheus/client_golang v1.7.1/go.mod h1:PY5Wy2awLA44sXw4AOSfFBetzPP4j5+D6mVACh+pe2M= github.com/prometheus/client_golang v1.11.0/go.mod h1:Z6t4BnS23TR94PD6BsDNk8yVqroYurpAkEiz0P2BEV0= -github.com/prometheus/client_golang v1.14.0 h1:nJdhIvne2eSX/XRAFV9PcvFFRbrjbcTUj0VP62TMhnw= -github.com/prometheus/client_golang v1.14.0/go.mod h1:8vpkKitgIVNcqrRBWh1C4TIUQgYNtG/XQE4E/Zae36Y= +github.com/prometheus/client_golang v1.15.0 h1:5fCgGYogn0hFdhyhLbw7hEsWxufKtY9klyvdNfFlFhM= +github.com/prometheus/client_golang v1.15.0/go.mod h1:e9yaBhRPU2pPNsZwE+JdQl0KEt1N9XgF6zxWmaC0xOk= github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo= github.com/prometheus/client_model v0.0.0-20190115171406-56726106282f/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo= github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= @@ -683,16 +686,16 @@ github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6L github.com/rogpeppe/go-internal v1.1.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= github.com/rogpeppe/go-internal v1.2.2/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= -github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8= -github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs= +github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= +github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog= github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts= github.com/samuel/go-zookeeper v0.0.0-20190923202752-2cc03de413da/go.mod h1:gi+0XIa01GRL2eRQVjQkKGqKF3SF9vZR/HnPullcV2E= -github.com/scaleway/scaleway-sdk-go v1.0.0-beta.14 h1:yFl3jyaSVLNYXlnNYM5z2pagEk1dYQhfr1p20T1NyKY= -github.com/scaleway/scaleway-sdk-go v1.0.0-beta.14/go.mod h1:fCa7OJZ/9DRTnOKmxvT6pn+LPWUptQAmHF/SBJUGEcg= +github.com/scaleway/scaleway-sdk-go v1.0.0-beta.15 h1:Y7xOFbD+3jaPw+VN7lkakNJ/pa+ZSQVFp1ONtJaBxns= +github.com/scaleway/scaleway-sdk-go v1.0.0-beta.15/go.mod h1:fCa7OJZ/9DRTnOKmxvT6pn+LPWUptQAmHF/SBJUGEcg= github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529 h1:nn5Wsu0esKSJiIVhscUtVbo7ada43DJhG55ua/hjS5I= github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc= -github.com/shoenig/test v0.6.2 h1:tdq+WGnznwE5xcOMXkqqXuudK75RkSGBazBGcP1lX6w= +github.com/shoenig/test v0.6.3 h1:GVXWJFk9PiOjN0KoJ7VrJGH6uLPnqxR7/fe3HUPfE0c= github.com/shurcooL/httpfs v0.0.0-20190707220628-8d4bc4ba7749 h1:bUGsEnyNbVPw06Bs80sCeARAlK8lhwqGyi6UT8ymuGk= github.com/shurcooL/httpfs v0.0.0-20190707220628-8d4bc4ba7749/go.mod h1:ZY1cvUeJuFPAdZ/B6v7RHavJWZn2YPVFQ1OSXhCGOkg= github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc= @@ -766,8 +769,8 @@ go.etcd.io/etcd v0.0.0-20191023171146-3cf2f69b5738/go.mod h1:dnLIgRNXwCJa5e+c6mI go.mongodb.org/mongo-driver v1.7.3/go.mod h1:NqaYOwnXWr5Pm7AOpO5QFxKJ503nbMse/R79oO62zWg= go.mongodb.org/mongo-driver v1.7.5/go.mod h1:VXEWRZ6URJIkUq2SCAyapmhH0ZLRBP+FT4xhp5Zvxng= go.mongodb.org/mongo-driver v1.10.0/go.mod h1:wsihk0Kdgv8Kqu1Anit4sfK+22vSFbUrAVEYRhCXrA8= -go.mongodb.org/mongo-driver v1.11.2 h1:+1v2rDQUWNcGW7/7E0Jvdz51V38XXxJfhzbV17aNHCw= -go.mongodb.org/mongo-driver v1.11.2/go.mod h1:s7p5vEtfbeR1gYi6pnj3c3/urpbLv2T5Sfd6Rp2HBB8= +go.mongodb.org/mongo-driver v1.11.3 h1:Ql6K6qYHEzB6xvu4+AU0BoRoqf9vFPcc4o7MUIdPW8Y= +go.mongodb.org/mongo-driver v1.11.3/go.mod h1:PTSz5yu21bkT/wXpkS7WR5f0ddqw5quethTUn9WM+2g= go.opencensus.io v0.20.1/go.mod h1:6WKK9ahsWS3RSO+PY9ZHZUfv2irvY6gN279GOPZjmmk= go.opencensus.io v0.20.2/go.mod h1:6WKK9ahsWS3RSO+PY9ZHZUfv2irvY6gN279GOPZjmmk= go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= @@ -802,8 +805,8 @@ go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= go.uber.org/atomic v1.5.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ= go.uber.org/atomic v1.10.0 h1:9qC72Qh0+3MqyJbAn8YU5xVq1frD8bn3JtD2oXtafVQ= go.uber.org/atomic v1.10.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0= -go.uber.org/automaxprocs v1.5.1 h1:e1YG66Lrk73dn4qhg8WFSvhF0JuFQF0ERIp4rpuV8Qk= -go.uber.org/automaxprocs v1.5.1/go.mod h1:BF4eumQw0P9GtnuxxovUd06vwm1o18oMzFtK66vU6XU= +go.uber.org/automaxprocs v1.5.2 h1:2LxUOGiR3O6tw8ui5sZa2LAaHnsviZdVOUZw4fvbnME= +go.uber.org/automaxprocs v1.5.2/go.mod h1:eRbA25aqJrxAbsLO0xy5jVwPt7FQnRgjW+efnwa1WM0= go.uber.org/goleak v1.2.1 h1:NBol2c7O1ZokfZ0LEU9K6Whx/KnwvepVetCUhtKja4A= go.uber.org/goleak v1.2.1/go.mod h1:qlT2yGI9QafXHhZZLxlSuNsMw3FFLxBr+tBRlmO1xH4= go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0= @@ -839,8 +842,8 @@ golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u0 golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM= golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU= -golang.org/x/exp v0.0.0-20230307190834-24139beb5833 h1:SChBja7BCQewoTAU7IgvucQKMIXrEpFxNMs0spT3/5s= -golang.org/x/exp v0.0.0-20230307190834-24139beb5833/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc= +golang.org/x/exp v0.0.0-20230321023759-10a507213a29 h1:ooxPy7fPvB4kwsA2h+iBNHkAbp/4JxTSwCmvdjEYmug= +golang.org/x/exp v0.0.0-20230321023759-10a507213a29/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc= golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js= golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= @@ -862,8 +865,8 @@ golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzB golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= -golang.org/x/mod v0.9.0 h1:KENHtAZL2y3NLMYZeHY9DW8HW8V+kQyJsY/V9JlKvCs= -golang.org/x/mod v0.9.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= +golang.org/x/mod v0.10.0 h1:lFO9qtOdlre5W1jxS3r/4szv2/6iXxScdzjoBMXNhYk= +golang.org/x/mod v0.10.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -911,8 +914,8 @@ golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qx golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.1.0/go.mod h1:Cx3nUiGt4eDBEyega/BKRp+/AlGL8hYe7U9odMt2Cco= golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= -golang.org/x/net v0.8.0 h1:Zrh2ngAOFYneWTAIAPethzeaQLuHwhuBkuV6ZiRnUaQ= -golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc= +golang.org/x/net v0.9.0 h1:aWJ/m6xSmxWBx+V0XRHTlrYrPG56jKsLdTFmsSsCzOM= +golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= @@ -920,8 +923,8 @@ golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4Iltr golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.6.0 h1:Lh8GPgSKBfWSwFvtuWOfeI3aAAnbXTSutYxJiOJFgIw= -golang.org/x/oauth2 v0.6.0/go.mod h1:ycmewcwgD4Rpr3eZJLSB4Kyyljb3qDh40vJ8STE5HKw= +golang.org/x/oauth2 v0.7.0 h1:qe6s0zUXlPX80/dITx3440hWZ7GwMwgDDyrSGTPJG/g= +golang.org/x/oauth2 v0.7.0/go.mod h1:hPLQkd9LyjfXTiRohC/41GhcFqxisoUQ99sCUOHO9x4= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -1003,14 +1006,14 @@ golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.6.0 h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ= -golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.7.0 h1:3jlCCIQZPdOYu1h8BkNvLz8Kgwtae2cagcG/VamtZRU= +golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.1.0/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= -golang.org/x/term v0.6.0 h1:clScbb1cHjoCkyRbWwBEUZ5H/tIFu5TAXIqaZD0Gcjw= -golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U= +golang.org/x/term v0.7.0 h1:BEvjmm5fURWqcfbSKTdpkDXYBrUS1c0m8agp14W48vQ= +golang.org/x/term v0.7.0/go.mod h1:P32HKFT3hSsZrRxla30E9HqToFYAQPCMs/zFMBUFqPY= golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= @@ -1021,8 +1024,8 @@ golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= -golang.org/x/text v0.8.0 h1:57P1ETyNKtuIjB4SRd15iJxuhj8Gc416Y78H3qgMh68= -golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= +golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE= +golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= @@ -1082,8 +1085,8 @@ golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roY golang.org/x/tools v0.0.0-20200825202427-b303f430e36d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA= golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= -golang.org/x/tools v0.7.0 h1:W4OVu8VVOaIO0yzWMNdepAulS7YfoS3Zabrm8DOXXU4= -golang.org/x/tools v0.7.0/go.mod h1:4pg6aUX35JBAogB10C9AtvVL+qowtN4pT3CGSQex14s= +golang.org/x/tools v0.8.0 h1:vSDcovVPld282ceKgDimkRSC8kpaH1dgyc9UMzlt84Y= +golang.org/x/tools v0.8.0/go.mod h1:JxBZ99ISMI5ViVkT1tr6tdNmXeTrcpVSD3vZ1RsRdN4= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= @@ -1183,8 +1186,8 @@ google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlba google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= -google.golang.org/protobuf v1.29.1 h1:7QBf+IK2gx70Ap/hDsOmam3GE0v9HicjfEdAxE62UoM= -google.golang.org/protobuf v1.29.1/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= +google.golang.org/protobuf v1.30.0 h1:kPPoIgf3TsEvrm0PFe15JQ+570QVxYzEvvHqChK+cng= +google.golang.org/protobuf v1.30.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= @@ -1198,7 +1201,6 @@ gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMy gopkg.in/gcfg.v1 v1.2.3/go.mod h1:yesOnuUOFQAhST5vPY4nbZsb/huCgGGXlipJsBn0b3o= gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc= gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= -gopkg.in/ini.v1 v1.57.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA= gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= gopkg.in/resty.v1 v1.12.0/go.mod h1:mDo4pnntr5jdWRML875a/NmxYqAlA73dVijT2AXvQQo= From 74e6668e87adc1a0d0833a40f96f7f064c2012ae Mon Sep 17 00:00:00 2001 From: gotjosh Date: Thu, 20 Apr 2023 09:34:05 +0100 Subject: [PATCH 106/231] update docs Signed-off-by: gotjosh --- docs/querying/api.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/querying/api.md b/docs/querying/api.md index 820414fb13..ef7fa54c67 100644 --- a/docs/querying/api.md +++ b/docs/querying/api.md @@ -675,9 +675,9 @@ GET /api/v1/rules URL query parameters: - `type=alert|record`: return only the alerting rules (e.g. `type=alert`) or the recording rules (e.g. `type=record`). When the parameter is absent or empty, no filtering is done. -- `rule_name[]=`: only return rules with the given rule name. If the parameter is repeated, rules with any of provided names are returned. If we've filtered out all the rules of a group, the group is not returned. When the parameter is absent or empty, no filtering is done. -- `rule_group[]=`: only return rules with the given rule group name. If the parameter is repeated, rules with any of provided rule group names are returned. When the parameter is absent or empty, no filtering is done. -- `file[]=`: only return rules with the given filepath. If the parameter is repeated, rules with any of provided filepaths are returned. When the parameter is absent or empty, no filtering is done. +- `rule_name[]=`: only return rules with the given rule name. If the parameter is repeated, rules with any of the provided names are returned. If we've filtered out all the rules of a group, the group is not returned. When the parameter is absent or empty, no filtering is done. +- `rule_group[]=`: only return rules with the given rule group name. If the parameter is repeated, rules with any of the provided rule group names are returned. When the parameter is absent or empty, no filtering is done. +- `file[]=`: only return rules with the given filepath. If the parameter is repeated, rules with any of the provided filepaths are returned. When the parameter is absent or empty, no filtering is done. ```json $ curl http://localhost:9090/api/v1/rules From e78be38cc08231c0a151f4672b3a049f66c83f11 Mon Sep 17 00:00:00 2001 From: gotjosh Date: Thu, 20 Apr 2023 11:20:10 +0100 Subject: [PATCH 107/231] don't show empty groups Signed-off-by: gotjosh --- web/api/v1/api.go | 8 +++++--- web/api/v1/api_test.go | 2 +- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/web/api/v1/api.go b/web/api/v1/api.go index 9a13e09d98..e700f71205 100644 --- a/web/api/v1/api.go +++ b/web/api/v1/api.go @@ -1309,7 +1309,7 @@ func (api *API) rules(r *http.Request) apiFuncResult { fSet := queryFormToSet(r.Form["file[]"]) ruleGroups := api.rulesRetriever(r.Context()).RuleGroups() - res := &RuleDiscovery{RuleGroups: make([]*RuleGroup, len(ruleGroups))} + res := &RuleDiscovery{RuleGroups: make([]*RuleGroup, 0, len(ruleGroups))} typ := strings.ToLower(r.URL.Query().Get("type")) if typ != "" && typ != "alert" && typ != "record" { @@ -1319,7 +1319,8 @@ func (api *API) rules(r *http.Request) apiFuncResult { returnAlerts := typ == "" || typ == "alert" returnRecording := typ == "" || typ == "record" - for i, grp := range ruleGroups { + rgs := make([]*RuleGroup, 0, len(ruleGroups)) + for _, grp := range ruleGroups { if len(rgSet) > 0 { if _, ok := rgSet[grp.Name()]; !ok { continue @@ -1400,9 +1401,10 @@ func (api *API) rules(r *http.Request) apiFuncResult { // If the rule group response has no rules, skip it - this means we filtered all the rules of this group. if len(apiRuleGroup.Rules) > 0 { - res.RuleGroups[i] = apiRuleGroup + rgs = append(rgs, apiRuleGroup) } } + res.RuleGroups = rgs return apiFuncResult{res, nil, nil, nil} } diff --git a/web/api/v1/api_test.go b/web/api/v1/api_test.go index c3e1bf59d7..53ea21182d 100644 --- a/web/api/v1/api_test.go +++ b/web/api/v1/api_test.go @@ -2003,7 +2003,7 @@ func testEndpoints(t *testing.T, api *API, tr *testTargetRetriever, es storage.E { endpoint: api.rules, query: url.Values{"rule_group[]": []string{"respond-with-nothing"}}, - response: &RuleDiscovery{RuleGroups: []*RuleGroup{nil}}, + response: &RuleDiscovery{RuleGroups: []*RuleGroup{}}, }, { endpoint: api.rules, From 637235f0a6af483ddf45e16c5ad800fee5bb946b Mon Sep 17 00:00:00 2001 From: Julien Pivotto Date: Thu, 20 Apr 2023 16:19:50 +0200 Subject: [PATCH 108/231] Revert type casting removal This reverts the removal of type casting due to an error in the dragonfly integration. The change in the type casting introduced by the commit causes a type mismatch, resulting in the following errors: util/runtime/limits_default.go:42:57: cannot use rlimit.Cur (variable of type int64) as type uint64 in argument to limitToString util/runtime/limits_default.go:42:90: cannot use rlimit.Max (variable of type int64) as type uint64 in argument to limitToString Reverting this commit to resolve the type mismatch error and maintain compatibility with the dragonfly integration. Signed-off-by: Julien Pivotto --- util/runtime/limits_default.go | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/util/runtime/limits_default.go b/util/runtime/limits_default.go index 1588a93c79..cd0ce732f9 100644 --- a/util/runtime/limits_default.go +++ b/util/runtime/limits_default.go @@ -39,7 +39,9 @@ func getLimits(resource int, unit string) string { if err != nil { panic("syscall.Getrlimit failed: " + err.Error()) } - return fmt.Sprintf("(soft=%s, hard=%s)", limitToString(rlimit.Cur, unit), limitToString(rlimit.Max, unit)) + // rlimit.Cur and rlimit.Max are int64 on some platforms, such as dragonfly. + // We need to cast them explicitly to uint64. + return fmt.Sprintf("(soft=%s, hard=%s)", limitToString(uint64(rlimit.Cur), unit), limitToString(uint64(rlimit.Max), unit)) //nolint:unconvert } // FdLimits returns the soft and hard limits for file descriptors. From 7e9acc2e46452da9b259d2253ba6fc8ae9bd7e95 Mon Sep 17 00:00:00 2001 From: Matthieu MOREL Date: Thu, 20 Apr 2023 18:43:51 +0200 Subject: [PATCH 109/231] golangci-lint: remove skip-cache and restore singleCaseSwitch rule Signed-off-by: Matthieu MOREL --- .github/workflows/ci.yml | 1 - .golangci.yml | 3 --- promql/engine.go | 9 +++------ 3 files changed, 3 insertions(+), 10 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 07b1242c28..ee63ee300d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -150,7 +150,6 @@ jobs: uses: golangci/golangci-lint-action@v3.4.0 with: args: --verbose - skip-cache: true version: v1.51.2 fuzzing: uses: ./.github/workflows/fuzzing.yml diff --git a/.golangci.yml b/.golangci.yml index c0c20d425a..d1cd86ed59 100644 --- a/.golangci.yml +++ b/.golangci.yml @@ -24,9 +24,6 @@ issues: - linters: - gocritic text: "appendAssign" - - linters: - - gocritic - text: "singleCaseSwitch" - path: _test.go linters: - errcheck diff --git a/promql/engine.go b/promql/engine.go index 910601a88d..cbeeb82a1a 100644 --- a/promql/engine.go +++ b/promql/engine.go @@ -757,8 +757,7 @@ func subqueryTimes(path []parser.Node) (time.Duration, time.Duration, *int64) { ts int64 = math.MaxInt64 ) for _, node := range path { - switch n := node.(type) { - case *parser.SubqueryExpr: + if n, ok := node.(*parser.SubqueryExpr); ok { subqOffset += n.OriginalOffset subqRange += n.Range if n.Timestamp != nil { @@ -847,8 +846,7 @@ func (ng *Engine) getTimeRangesForSelector(s *parser.EvalStmt, n *parser.VectorS func (ng *Engine) getLastSubqueryInterval(path []parser.Node) time.Duration { var interval time.Duration for _, node := range path { - switch n := node.(type) { - case *parser.SubqueryExpr: + if n, ok := node.(*parser.SubqueryExpr); ok { interval = n.Step if n.Step == 0 { interval = time.Duration(ng.noStepSubqueryIntervalFn(durationMilliseconds(n.Range))) * time.Millisecond @@ -914,8 +912,7 @@ func extractGroupsFromPath(p []parser.Node) (bool, []string) { if len(p) == 0 { return false, nil } - switch n := p[len(p)-1].(type) { - case *parser.AggregateExpr: + if n, ok := p[len(p)-1].(*parser.AggregateExpr); ok { return !n.Without, n.Grouping } return false, nil From ecf6bfa619009255d3cb615d649cb66ca4c217fc Mon Sep 17 00:00:00 2001 From: John Losito Date: Fri, 21 Apr 2023 09:26:16 -0400 Subject: [PATCH 110/231] Update configuration.md Signed-off-by: John Losito --- docs/configuration/configuration.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/configuration/configuration.md b/docs/configuration/configuration.md index f27f8256a5..c5ecc292cf 100644 --- a/docs/configuration/configuration.md +++ b/docs/configuration/configuration.md @@ -350,6 +350,7 @@ metric_relabel_configs: # This is an experimental feature, this behaviour could # change or be removed in the future. [ body_size_limit: | default = 0 ] + # Per-scrape limit on number of scraped samples that will be accepted. # If more than this number of samples are present after metric relabeling # the entire scrape will be treated as failed. 0 means no limit. From e9a1e26ab700c1206c749b0b069fff99c8426aac Mon Sep 17 00:00:00 2001 From: Jeanette Tan Date: Sat, 22 Apr 2023 02:27:15 +0800 Subject: [PATCH 111/231] Perform integer/float histogram type checking on conversions, and use a consistent method for determining integer vs float histogram Signed-off-by: Jeanette Tan --- prompb/custom.go | 5 +++++ storage/remote/codec.go | 24 +++++++++++++++--------- storage/remote/codec_test.go | 2 +- storage/remote/queue_manager_test.go | 2 +- storage/remote/write_handler.go | 2 +- storage/remote/write_handler_test.go | 2 +- 6 files changed, 24 insertions(+), 13 deletions(-) diff --git a/prompb/custom.go b/prompb/custom.go index 4b07187bd2..13d6e0f0cd 100644 --- a/prompb/custom.go +++ b/prompb/custom.go @@ -20,6 +20,11 @@ import ( func (m Sample) T() int64 { return m.Timestamp } func (m Sample) V() float64 { return m.Value } +func (h Histogram) IsFloatHistogram() bool { + _, ok := h.GetCount().(*Histogram_CountFloat) + return ok +} + func (r *ChunkedReadResponse) PooledMarshal(p *sync.Pool) ([]byte, error) { size := r.Size() data, ok := p.Get().(*[]byte) diff --git a/storage/remote/codec.go b/storage/remote/codec.go index 02c84a3e6c..9a683b908f 100644 --- a/storage/remote/codec.go +++ b/storage/remote/codec.go @@ -455,10 +455,10 @@ func (c *concreteSeriesIterator) Seek(t int64) chunkenc.ValueType { } func getHistogramValType(h *prompb.Histogram) chunkenc.ValueType { - if _, isInt := h.GetCount().(*prompb.Histogram_CountInt); isInt { - return chunkenc.ValHistogram + if h.IsFloatHistogram() { + return chunkenc.ValFloatHistogram } - return chunkenc.ValFloatHistogram + return chunkenc.ValHistogram } // At implements chunkenc.Iterator. @@ -624,9 +624,11 @@ func exemplarProtoToExemplar(ep prompb.Exemplar) exemplar.Exemplar { } // HistogramProtoToHistogram extracts a (normal integer) Histogram from the -// provided proto message. The caller has to make sure that the proto message -// represents an integer histogram and not a float histogram. +// provided proto message. func HistogramProtoToHistogram(hp prompb.Histogram) *histogram.Histogram { + if hp.IsFloatHistogram() { + panic("don't call HistogramProtoToHistogram on a float histogram") + } return &histogram.Histogram{ CounterResetHint: histogram.CounterResetHint(hp.ResetHint), Schema: hp.Schema, @@ -642,9 +644,11 @@ func HistogramProtoToHistogram(hp prompb.Histogram) *histogram.Histogram { } // FloatHistogramProtoToFloatHistogram extracts a float Histogram from the -// provided proto message to a Float Histogram. The caller has to make sure that -// the proto message represents a float histogram and not an integer histogram. +// provided proto message to a Float Histogram. func FloatHistogramProtoToFloatHistogram(hp prompb.Histogram) *histogram.FloatHistogram { + if !hp.IsFloatHistogram() { + panic("don't call FloatHistogramProtoToFloatHistogram on an integer histogram") + } return &histogram.FloatHistogram{ CounterResetHint: histogram.CounterResetHint(hp.ResetHint), Schema: hp.Schema, @@ -660,9 +664,11 @@ func FloatHistogramProtoToFloatHistogram(hp prompb.Histogram) *histogram.FloatHi } // HistogramProtoToFloatHistogram extracts and converts a (normal integer) histogram from the provided proto message -// to a float histogram. The caller has to make sure that the proto message represents an integer histogram and not a -// float histogram. +// to a float histogram. func HistogramProtoToFloatHistogram(hp prompb.Histogram) *histogram.FloatHistogram { + if hp.IsFloatHistogram() { + panic("don't call HistogramProtoToFloatHistogram on a float histogram") + } return &histogram.FloatHistogram{ CounterResetHint: histogram.CounterResetHint(hp.ResetHint), Schema: hp.Schema, diff --git a/storage/remote/codec_test.go b/storage/remote/codec_test.go index 27e2cc704d..dbd5cec219 100644 --- a/storage/remote/codec_test.go +++ b/storage/remote/codec_test.go @@ -528,7 +528,7 @@ func TestNilHistogramProto(*testing.T) { // This function will panic if it impromperly handles nil // values, causing the test to fail. HistogramProtoToHistogram(prompb.Histogram{}) - FloatHistogramProtoToFloatHistogram(prompb.Histogram{}) + HistogramProtoToFloatHistogram(prompb.Histogram{}) } func exampleHistogram() histogram.Histogram { diff --git a/storage/remote/queue_manager_test.go b/storage/remote/queue_manager_test.go index a57c3bf7b1..b43258ff06 100644 --- a/storage/remote/queue_manager_test.go +++ b/storage/remote/queue_manager_test.go @@ -802,7 +802,7 @@ func (c *TestWriteClient) Store(_ context.Context, req []byte) error { for _, histogram := range ts.Histograms { count++ - if histogram.GetCountFloat() > 0 || histogram.GetZeroCountFloat() > 0 { + if histogram.IsFloatHistogram() { c.receivedFloatHistograms[seriesName] = append(c.receivedFloatHistograms[seriesName], histogram) } else { c.receivedHistograms[seriesName] = append(c.receivedHistograms[seriesName], histogram) diff --git a/storage/remote/write_handler.go b/storage/remote/write_handler.go index 5aa1130888..1f4c43e595 100644 --- a/storage/remote/write_handler.go +++ b/storage/remote/write_handler.go @@ -125,7 +125,7 @@ func (h *writeHandler) write(ctx context.Context, req *prompb.WriteRequest) (err } for _, hp := range ts.Histograms { - if hp.GetCountFloat() > 0 || hp.GetZeroCountFloat() > 0 { // It is a float histogram. + if hp.IsFloatHistogram() { fhs := FloatHistogramProtoToFloatHistogram(hp) _, err = app.AppendHistogram(0, labels, hp.Timestamp, nil, fhs) } else { diff --git a/storage/remote/write_handler_test.go b/storage/remote/write_handler_test.go index e7a88ddc23..3bce5f1d88 100644 --- a/storage/remote/write_handler_test.go +++ b/storage/remote/write_handler_test.go @@ -68,7 +68,7 @@ func TestRemoteWriteHandler(t *testing.T) { } for _, hp := range ts.Histograms { - if hp.GetCountFloat() > 0 || hp.GetZeroCountFloat() > 0 { // It is a float histogram. + if hp.IsFloatHistogram() { fh := FloatHistogramProtoToFloatHistogram(hp) require.Equal(t, mockHistogram{labels, hp.Timestamp, nil, fh}, appendable.histograms[k]) } else { From 1102ffd188b4a9bc16c2c120c3eeb3dc09a92a99 Mon Sep 17 00:00:00 2001 From: Jeanette Tan Date: Sat, 22 Apr 2023 02:27:15 +0800 Subject: [PATCH 112/231] Fix according to code review Signed-off-by: Jeanette Tan --- storage/remote/codec.go | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/storage/remote/codec.go b/storage/remote/codec.go index 9a683b908f..6a58ec4ac6 100644 --- a/storage/remote/codec.go +++ b/storage/remote/codec.go @@ -624,10 +624,11 @@ func exemplarProtoToExemplar(ep prompb.Exemplar) exemplar.Exemplar { } // HistogramProtoToHistogram extracts a (normal integer) Histogram from the -// provided proto message. +// provided proto message. The caller has to make sure that the proto message +// represents an integer histogram and not a float histogram, or it panics. func HistogramProtoToHistogram(hp prompb.Histogram) *histogram.Histogram { if hp.IsFloatHistogram() { - panic("don't call HistogramProtoToHistogram on a float histogram") + panic("HistogramProtoToHistogram called with a float histogram") } return &histogram.Histogram{ CounterResetHint: histogram.CounterResetHint(hp.ResetHint), @@ -644,10 +645,12 @@ func HistogramProtoToHistogram(hp prompb.Histogram) *histogram.Histogram { } // FloatHistogramProtoToFloatHistogram extracts a float Histogram from the -// provided proto message to a Float Histogram. +// provided proto message to a Float Histogram. The caller has to make sure that +// the proto message represents a float histogram and not an integer histogram, +// or it panics. func FloatHistogramProtoToFloatHistogram(hp prompb.Histogram) *histogram.FloatHistogram { if !hp.IsFloatHistogram() { - panic("don't call FloatHistogramProtoToFloatHistogram on an integer histogram") + panic("FloatHistogramProtoToFloatHistogram called with an integer histogram") } return &histogram.FloatHistogram{ CounterResetHint: histogram.CounterResetHint(hp.ResetHint), @@ -664,10 +667,11 @@ func FloatHistogramProtoToFloatHistogram(hp prompb.Histogram) *histogram.FloatHi } // HistogramProtoToFloatHistogram extracts and converts a (normal integer) histogram from the provided proto message -// to a float histogram. +// to a float histogram. The caller has to make sure that the proto message represents an integer histogram and not a +// float histogram, or it panics. func HistogramProtoToFloatHistogram(hp prompb.Histogram) *histogram.FloatHistogram { if hp.IsFloatHistogram() { - panic("don't call HistogramProtoToFloatHistogram on a float histogram") + panic("HistogramProtoToFloatHistogram called with a float histogram") } return &histogram.FloatHistogram{ CounterResetHint: histogram.CounterResetHint(hp.ResetHint), From 4d21ac23e642391a5c7f75962cd5c939679f7dd2 Mon Sep 17 00:00:00 2001 From: Jeanette Tan Date: Sat, 22 Apr 2023 03:14:19 +0800 Subject: [PATCH 113/231] Implement bucket limit for native histograms Signed-off-by: Jeanette Tan --- config/config.go | 3 ++ scrape/scrape.go | 44 +++++++++++++++++++---- scrape/scrape_test.go | 82 +++++++++++++++++++++++++------------------ scrape/target.go | 26 ++++++++++++++ storage/interface.go | 1 + 5 files changed, 116 insertions(+), 40 deletions(-) diff --git a/config/config.go b/config/config.go index 5c51d5a0d8..31ebb288e3 100644 --- a/config/config.go +++ b/config/config.go @@ -489,6 +489,9 @@ type ScrapeConfig struct { // More than this label value length post metric-relabeling will cause the // scrape to fail. LabelValueLengthLimit uint `yaml:"label_value_length_limit,omitempty"` + // More than this many buckets in a native histogram will cause the scrape to + // fail. + NativeHistogramBucketLimit uint `yaml:"bucket_limit,omitempty"` // We cannot do proper Go type embedding below as the parser will then parse // values arbitrarily into the overflow maps of further-down types. diff --git a/scrape/scrape.go b/scrape/scrape.go index 5c649e729a..7c24c1070b 100644 --- a/scrape/scrape.go +++ b/scrape/scrape.go @@ -191,6 +191,12 @@ var ( }, []string{"scrape_job"}, ) + targetScrapeNativeHistogramBucketLimit = prometheus.NewCounter( + prometheus.CounterOpts{ + Name: "prometheus_target_scrapes_histogram_exceeded_bucket_limit_total", + Help: "Total number of native histograms rejected due to exceeding the bucket limit.", + }, + ) ) func init() { @@ -216,6 +222,7 @@ func init() { targetScrapeExemplarOutOfOrder, targetScrapePoolExceededLabelLimits, targetSyncFailed, + targetScrapeNativeHistogramBucketLimit, ) } @@ -256,6 +263,7 @@ type scrapeLoopOptions struct { target *Target scraper scraper sampleLimit int + bucketLimit int labelLimits *labelLimits honorLabels bool honorTimestamps bool @@ -319,6 +327,7 @@ func newScrapePool(cfg *config.ScrapeConfig, app storage.Appendable, jitterSeed jitterSeed, opts.honorTimestamps, opts.sampleLimit, + opts.bucketLimit, opts.labelLimits, opts.interval, opts.timeout, @@ -412,6 +421,7 @@ func (sp *scrapePool) reload(cfg *config.ScrapeConfig) error { timeout = time.Duration(sp.config.ScrapeTimeout) bodySizeLimit = int64(sp.config.BodySizeLimit) sampleLimit = int(sp.config.SampleLimit) + bucketLimit = int(sp.config.NativeHistogramBucketLimit) labelLimits = &labelLimits{ labelLimit: int(sp.config.LabelLimit), labelNameLengthLimit: int(sp.config.LabelNameLengthLimit), @@ -446,6 +456,7 @@ func (sp *scrapePool) reload(cfg *config.ScrapeConfig) error { target: t, scraper: s, sampleLimit: sampleLimit, + bucketLimit: bucketLimit, labelLimits: labelLimits, honorLabels: honorLabels, honorTimestamps: honorTimestamps, @@ -530,6 +541,7 @@ func (sp *scrapePool) sync(targets []*Target) { timeout = time.Duration(sp.config.ScrapeTimeout) bodySizeLimit = int64(sp.config.BodySizeLimit) sampleLimit = int(sp.config.SampleLimit) + bucketLimit = int(sp.config.NativeHistogramBucketLimit) labelLimits = &labelLimits{ labelLimit: int(sp.config.LabelLimit), labelNameLengthLimit: int(sp.config.LabelNameLengthLimit), @@ -559,6 +571,7 @@ func (sp *scrapePool) sync(targets []*Target) { target: t, scraper: s, sampleLimit: sampleLimit, + bucketLimit: bucketLimit, labelLimits: labelLimits, honorLabels: honorLabels, honorTimestamps: honorTimestamps, @@ -731,7 +744,7 @@ func mutateReportSampleLabels(lset labels.Labels, target *Target) labels.Labels } // appender returns an appender for ingested samples from the target. -func appender(app storage.Appender, limit int) storage.Appender { +func appender(app storage.Appender, limit, bucketLimit int) storage.Appender { app = &timeLimitAppender{ Appender: app, maxTime: timestamp.FromTime(time.Now().Add(maxAheadTime)), @@ -744,6 +757,13 @@ func appender(app storage.Appender, limit int) storage.Appender { limit: limit, } } + + if bucketLimit > 0 { + app = &bucketLimitAppender{ + Appender: app, + limit: bucketLimit, + } + } return app } @@ -872,6 +892,7 @@ type scrapeLoop struct { forcedErr error forcedErrMtx sync.Mutex sampleLimit int + bucketLimit int labelLimits *labelLimits interval time.Duration timeout time.Duration @@ -1152,6 +1173,7 @@ func newScrapeLoop(ctx context.Context, jitterSeed uint64, honorTimestamps bool, sampleLimit int, + bucketLimit int, labelLimits *labelLimits, interval time.Duration, timeout time.Duration, @@ -1195,6 +1217,7 @@ func newScrapeLoop(ctx context.Context, appenderCtx: appenderCtx, honorTimestamps: honorTimestamps, sampleLimit: sampleLimit, + bucketLimit: bucketLimit, labelLimits: labelLimits, interval: interval, timeout: timeout, @@ -1462,10 +1485,11 @@ func (sl *scrapeLoop) getCache() *scrapeCache { } type appendErrors struct { - numOutOfOrder int - numDuplicates int - numOutOfBounds int - numExemplarOutOfOrder int + numOutOfOrder int + numDuplicates int + numOutOfBounds int + numExemplarOutOfOrder int + numHistogramBucketLimit int } func (sl *scrapeLoop) append(app storage.Appender, b []byte, contentType string, ts time.Time) (total, added, seriesAdded int, err error) { @@ -1510,7 +1534,7 @@ func (sl *scrapeLoop) append(app storage.Appender, b []byte, contentType string, } // Take an appender with limits. - app = appender(app, sl.sampleLimit) + app = appender(app, sl.sampleLimit, sl.bucketLimit) defer func() { if err != nil { @@ -1693,6 +1717,9 @@ loop: if appErrs.numExemplarOutOfOrder > 0 { level.Warn(sl.l).Log("msg", "Error on ingesting out-of-order exemplars", "num_dropped", appErrs.numExemplarOutOfOrder) } + if appErrs.numHistogramBucketLimit > 0 { + level.Warn(sl.l).Log("msg", "Error on ingesting native histograms that exceeded bucket limit", "num_dropped", appErrs.numHistogramBucketLimit) + } if err == nil { sl.cache.forEachStale(func(lset labels.Labels) bool { // Series no longer exposed, mark it stale. @@ -1735,6 +1762,11 @@ func (sl *scrapeLoop) checkAddError(ce *cacheEntry, met []byte, tp *int64, err e level.Debug(sl.l).Log("msg", "Out of bounds metric", "series", string(met)) targetScrapeSampleOutOfBounds.Inc() return false, nil + case storage.ErrHistogramBucketLimit: + appErrs.numHistogramBucketLimit++ + level.Debug(sl.l).Log("msg", "Exceeded bucket limit for native histograms", "series", string(met)) + targetScrapeNativeHistogramBucketLimit.Inc() + return false, nil case errSampleLimit: // Keep on parsing output if we hit the limit, so we report the correct // total number of samples scraped. diff --git a/scrape/scrape_test.go b/scrape/scrape_test.go index 2a2ca09485..c815bf5c60 100644 --- a/scrape/scrape_test.go +++ b/scrape/scrape_test.go @@ -487,7 +487,7 @@ func TestScrapePoolAppender(t *testing.T) { appl, ok := loop.(*scrapeLoop) require.True(t, ok, "Expected scrapeLoop but got %T", loop) - wrapped := appender(appl.appender(context.Background()), 0) + wrapped := appender(appl.appender(context.Background()), 0, 0) tl, ok := wrapped.(*timeLimitAppender) require.True(t, ok, "Expected timeLimitAppender but got %T", wrapped) @@ -503,7 +503,7 @@ func TestScrapePoolAppender(t *testing.T) { appl, ok = loop.(*scrapeLoop) require.True(t, ok, "Expected scrapeLoop but got %T", loop) - wrapped = appender(appl.appender(context.Background()), sampleLimit) + wrapped = appender(appl.appender(context.Background()), sampleLimit, 0) sl, ok := wrapped.(*limitAppender) require.True(t, ok, "Expected limitAppender but got %T", wrapped) @@ -513,6 +513,20 @@ func TestScrapePoolAppender(t *testing.T) { _, ok = tl.Appender.(nopAppender) require.True(t, ok, "Expected base appender but got %T", tl.Appender) + + wrapped = appender(appl.appender(context.Background()), sampleLimit, 100) + + bl, ok := wrapped.(*bucketLimitAppender) + require.True(t, ok, "Expected bucketLimitAppender but got %T", wrapped) + + sl, ok = bl.Appender.(*limitAppender) + require.True(t, ok, "Expected limitAppender but got %T", bl) + + tl, ok = sl.Appender.(*timeLimitAppender) + require.True(t, ok, "Expected timeLimitAppender but got %T", sl.Appender) + + _, ok = tl.Appender.(nopAppender) + require.True(t, ok, "Expected base appender but got %T", tl.Appender) } func TestScrapePoolRaces(t *testing.T) { @@ -610,7 +624,7 @@ func TestScrapeLoopStopBeforeRun(t *testing.T) { nopMutator, nil, nil, 0, true, - 0, + 0, 0, nil, 1, 0, @@ -682,7 +696,7 @@ func TestScrapeLoopStop(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, 10*time.Millisecond, time.Hour, @@ -758,7 +772,7 @@ func TestScrapeLoopRun(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, time.Second, time.Hour, @@ -813,7 +827,7 @@ func TestScrapeLoopRun(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, time.Second, 100*time.Millisecond, @@ -872,7 +886,7 @@ func TestScrapeLoopForcedErr(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, time.Second, time.Hour, @@ -930,7 +944,7 @@ func TestScrapeLoopMetadata(t *testing.T) { cache, 0, true, - 0, + 0, 0, nil, 0, 0, @@ -987,7 +1001,7 @@ func simpleTestScrapeLoop(t testing.TB) (context.Context, *scrapeLoop) { nil, 0, true, - 0, + 0, 0, nil, 0, 0, @@ -1047,7 +1061,7 @@ func TestScrapeLoopFailWithInvalidLabelsAfterRelabel(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, 0, 0, @@ -1125,7 +1139,7 @@ func TestScrapeLoopRunCreatesStaleMarkersOnFailedScrape(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, 10*time.Millisecond, time.Hour, @@ -1188,7 +1202,7 @@ func TestScrapeLoopRunCreatesStaleMarkersOnParseFailure(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, 10*time.Millisecond, time.Hour, @@ -1254,7 +1268,7 @@ func TestScrapeLoopCache(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, 10*time.Millisecond, time.Hour, @@ -1337,7 +1351,7 @@ func TestScrapeLoopCacheMemoryExhaustionProtection(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, 10*time.Millisecond, time.Hour, @@ -1451,7 +1465,7 @@ func TestScrapeLoopAppend(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, 0, 0, @@ -1546,7 +1560,7 @@ func TestScrapeLoopAppendForConflictingPrefixedLabels(t *testing.T) { return mutateSampleLabels(l, &Target{labels: labels.FromStrings(tc.targetLabels...)}, false, nil) }, nil, - func(ctx context.Context) storage.Appender { return app }, nil, 0, true, 0, nil, 0, 0, false, false, nil, false, + func(ctx context.Context) storage.Appender { return app }, nil, 0, true, 0, 0, nil, 0, 0, false, false, nil, false, ) slApp := sl.appender(context.Background()) _, _, _, err := sl.append(slApp, []byte(tc.exposedLabels), "", time.Date(2000, 1, 1, 1, 0, 0, 0, time.UTC)) @@ -1577,7 +1591,7 @@ func TestScrapeLoopAppendCacheEntryButErrNotFound(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, 0, 0, @@ -1635,7 +1649,7 @@ func TestScrapeLoopAppendSampleLimit(t *testing.T) { nil, 0, true, - app.limit, + app.limit, 0, nil, 0, 0, @@ -1712,7 +1726,7 @@ func TestScrapeLoop_ChangingMetricString(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, 0, 0, @@ -1760,7 +1774,7 @@ func TestScrapeLoopAppendStaleness(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, 0, 0, @@ -1811,7 +1825,7 @@ func TestScrapeLoopAppendNoStalenessIfTimestamp(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, 0, 0, @@ -1922,7 +1936,7 @@ metric_total{n="2"} 2 # {t="2"} 2.0 20000 nil, 0, true, - 0, + 0, 0, nil, 0, 0, @@ -1987,7 +2001,7 @@ func TestScrapeLoopAppendExemplarSeries(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, 0, 0, @@ -2039,7 +2053,7 @@ func TestScrapeLoopRunReportsTargetDownOnScrapeError(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, 10*time.Millisecond, time.Hour, @@ -2075,7 +2089,7 @@ func TestScrapeLoopRunReportsTargetDownOnInvalidUTF8(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, 10*time.Millisecond, time.Hour, @@ -2124,7 +2138,7 @@ func TestScrapeLoopAppendGracefullyIfAmendOrOutOfOrderOrOutOfBounds(t *testing.T nil, 0, true, - 0, + 0, 0, nil, 0, 0, @@ -2169,7 +2183,7 @@ func TestScrapeLoopOutOfBoundsTimeError(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, 0, 0, @@ -2441,7 +2455,7 @@ func TestScrapeLoop_RespectTimestamps(t *testing.T) { func(ctx context.Context) storage.Appender { return capp }, nil, 0, true, - 0, + 0, 0, nil, 0, 0, @@ -2482,7 +2496,7 @@ func TestScrapeLoop_DiscardTimestamps(t *testing.T) { func(ctx context.Context) storage.Appender { return capp }, nil, 0, false, - 0, + 0, 0, nil, 0, 0, @@ -2522,7 +2536,7 @@ func TestScrapeLoopDiscardDuplicateLabels(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, 0, 0, @@ -2580,7 +2594,7 @@ func TestScrapeLoopDiscardUnnamedMetrics(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, 0, 0, @@ -2843,7 +2857,7 @@ func TestScrapeAddFast(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, 0, 0, @@ -2929,7 +2943,7 @@ func TestScrapeReportSingleAppender(t *testing.T) { nil, 0, true, - 0, + 0, 0, nil, 10*time.Millisecond, time.Hour, @@ -3131,7 +3145,7 @@ func TestScrapeLoopLabelLimit(t *testing.T) { nil, 0, true, - 0, + 0, 0, &test.labelLimits, 0, 0, diff --git a/scrape/target.go b/scrape/target.go index 6c47031186..f916e45490 100644 --- a/scrape/target.go +++ b/scrape/target.go @@ -27,6 +27,7 @@ import ( "github.com/prometheus/prometheus/config" "github.com/prometheus/prometheus/discovery/targetgroup" + "github.com/prometheus/prometheus/model/histogram" "github.com/prometheus/prometheus/model/labels" "github.com/prometheus/prometheus/model/relabel" "github.com/prometheus/prometheus/model/textparse" @@ -355,6 +356,31 @@ func (app *timeLimitAppender) Append(ref storage.SeriesRef, lset labels.Labels, return ref, nil } +// bucketLimitAppender limits the number of total appended samples in a batch. +type bucketLimitAppender struct { + storage.Appender + + limit int +} + +func (app *bucketLimitAppender) AppendHistogram(ref storage.SeriesRef, lset labels.Labels, t int64, h *histogram.Histogram, fh *histogram.FloatHistogram) (storage.SeriesRef, error) { + if h != nil { + if len(h.PositiveBuckets)+len(h.NegativeBuckets) > app.limit { + return 0, storage.ErrHistogramBucketLimit + } + } + if fh != nil { + if len(fh.PositiveBuckets)+len(fh.NegativeBuckets) > app.limit { + return 0, storage.ErrHistogramBucketLimit + } + } + ref, err := app.Appender.AppendHistogram(ref, lset, t, h, fh) + if err != nil { + return 0, err + } + return ref, nil +} + // PopulateLabels builds a label set from the given label set and scrape configuration. // It returns a label set before relabeling was applied as the second return value. // Returns the original discovered label set found before relabelling was applied if the target is dropped during relabeling. diff --git a/storage/interface.go b/storage/interface.go index b282f1fc62..18119f2547 100644 --- a/storage/interface.go +++ b/storage/interface.go @@ -46,6 +46,7 @@ var ( ErrHistogramNegativeBucketCount = errors.New("histogram has a bucket whose observation count is negative") ErrHistogramSpanNegativeOffset = errors.New("histogram has a span whose offset is negative") ErrHistogramSpansBucketsMismatch = errors.New("histogram spans specify different number of buckets than provided") + ErrHistogramBucketLimit = errors.New("histogram bucket limit exceeded") ) // SeriesRef is a generic series reference. In prometheus it is either a From d3ad158a660de06201d28a384cc2175e0488bab2 Mon Sep 17 00:00:00 2001 From: Jeanette Tan Date: Sat, 22 Apr 2023 03:14:19 +0800 Subject: [PATCH 114/231] Update docs and comments Signed-off-by: Jeanette Tan --- config/config.go | 4 ++-- docs/configuration/configuration.md | 5 +++++ 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/config/config.go b/config/config.go index 31ebb288e3..956e1c6c65 100644 --- a/config/config.go +++ b/config/config.go @@ -489,8 +489,8 @@ type ScrapeConfig struct { // More than this label value length post metric-relabeling will cause the // scrape to fail. LabelValueLengthLimit uint `yaml:"label_value_length_limit,omitempty"` - // More than this many buckets in a native histogram will cause the scrape to - // fail. + // More than this many buckets in a native histogram will cause the histogram + // to be ignored, but it will not make the whole scrape fail. NativeHistogramBucketLimit uint `yaml:"bucket_limit,omitempty"` // We cannot do proper Go type embedding below as the parser will then parse diff --git a/docs/configuration/configuration.md b/docs/configuration/configuration.md index f27f8256a5..f51227320d 100644 --- a/docs/configuration/configuration.md +++ b/docs/configuration/configuration.md @@ -376,6 +376,11 @@ metric_relabel_configs: # 0 means no limit. This is an experimental feature, this behaviour could # change in the future. [ target_limit: | default = 0 ] + +# Limit on total number of positive and negative buckets allowed in a native +# histogram. If this is exceeded, the histogram will be ignored, but this will +# not make the scrape fail. 0 means no limit. +[ sample_limit: | default = 0 ] ``` Where `` must be unique across all scrape configurations. From 071426f72f3546149b8d6d6ae3c77bf044e8fc93 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gy=C3=B6rgy=20Krajcsovits?= Date: Sat, 22 Apr 2023 03:14:19 +0800 Subject: [PATCH 115/231] Add unit test for bucket limit appender MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Refactors textparser test to use a common test utility to create protobuf representation from MetricFamily Signed-off-by: György Krajcsovits --- model/textparse/protobufparse_test.go | 12 +--- scrape/scrape_test.go | 90 +++++++++++++++++++++++++++ util/testutil/protobuf.go | 50 +++++++++++++++ 3 files changed, 143 insertions(+), 9 deletions(-) create mode 100644 util/testutil/protobuf.go diff --git a/model/textparse/protobufparse_test.go b/model/textparse/protobufparse_test.go index 90c6a90f32..fb8669197e 100644 --- a/model/textparse/protobufparse_test.go +++ b/model/textparse/protobufparse_test.go @@ -15,7 +15,6 @@ package textparse import ( "bytes" - "encoding/binary" "errors" "io" "testing" @@ -26,8 +25,9 @@ import ( "github.com/prometheus/prometheus/model/exemplar" "github.com/prometheus/prometheus/model/histogram" "github.com/prometheus/prometheus/model/labels" + "github.com/prometheus/prometheus/util/testutil" - dto "github.com/prometheus/prometheus/prompb/io/prometheus/client" + dto "github.com/prometheus/client_model/go" ) func TestProtobufParse(t *testing.T) { @@ -449,7 +449,6 @@ metric: < `, } - varintBuf := make([]byte, binary.MaxVarintLen32) inputBuf := &bytes.Buffer{} for _, tmf := range textMetricFamilies { @@ -457,13 +456,8 @@ metric: < // From text to proto message. require.NoError(t, proto.UnmarshalText(tmf, pb)) // From proto message to binary protobuf. - protoBuf, err := proto.Marshal(pb) + err := testutil.AddMetricFamilyToProtobuf(inputBuf, pb) require.NoError(t, err) - - // Write first length, then binary protobuf. - varintLength := binary.PutUvarint(varintBuf, uint64(len(protoBuf))) - inputBuf.Write(varintBuf[:varintLength]) - inputBuf.Write(protoBuf) } exp := []struct { diff --git a/scrape/scrape_test.go b/scrape/scrape_test.go index c815bf5c60..b5cabea52e 100644 --- a/scrape/scrape_test.go +++ b/scrape/scrape_test.go @@ -30,6 +30,7 @@ import ( "github.com/go-kit/log" "github.com/pkg/errors" + "github.com/prometheus/client_golang/prometheus" dto "github.com/prometheus/client_model/go" config_util "github.com/prometheus/common/config" "github.com/prometheus/common/model" @@ -1709,6 +1710,95 @@ func TestScrapeLoopAppendSampleLimit(t *testing.T) { require.Equal(t, 0, seriesAdded) } +func TestScrapeLoop_HistogramBucketLimit(t *testing.T) { + resApp := &collectResultAppender{} + app := &bucketLimitAppender{Appender: resApp, limit: 2} + + sl := newScrapeLoop(context.Background(), + nil, nil, nil, + func(l labels.Labels) labels.Labels { + if l.Has("deleteme") { + return labels.EmptyLabels() + } + return l + }, + nopMutator, + func(ctx context.Context) storage.Appender { return app }, + nil, + 0, + true, + app.limit, 0, + nil, + 0, + 0, + false, + false, + nil, + false, + ) + + metric := dto.Metric{} + err := targetScrapeNativeHistogramBucketLimit.Write(&metric) + require.NoError(t, err) + beforeMetricValue := metric.GetCounter().GetValue() + + nativeHistogram := prometheus.NewHistogram(prometheus.HistogramOpts{ + Namespace: "testing", + Name: "example_native_histogram", + Help: "This is used for testing", + ConstLabels: map[string]string{"some": "value"}, + NativeHistogramBucketFactor: 1.1, // 10% increase from bucket to bucket + NativeHistogramMaxBucketNumber: 100, // intentionally higher than the limit we'll use in the scraper + }) + registry := prometheus.NewRegistry() + registry.Register(nativeHistogram) + nativeHistogram.Observe(1.0) + nativeHistogram.Observe(10.0) // in different bucket since > 1*1.1 + + gathered, err := registry.Gather() + require.NoError(t, err) + require.NotEmpty(t, gathered) + + histogramMetricFamily := gathered[0] + msg, err := testutil.MetricFamilyToProtobuf(histogramMetricFamily) + require.NoError(t, err) + + now := time.Now() + total, added, seriesAdded, err := sl.append(app, msg, "application/vnd.google.protobuf", now) + require.NoError(t, err) + require.Equal(t, 1, total) + require.Equal(t, 1, added) + require.Equal(t, 1, seriesAdded) + + err = targetScrapeNativeHistogramBucketLimit.Write(&metric) + require.NoError(t, err) + metricValue := metric.GetCounter().GetValue() + require.Equal(t, beforeMetricValue, metricValue) + beforeMetricValue = metricValue + + nativeHistogram.Observe(100.0) // in different bucket since > 10*1.1 + + gathered, err = registry.Gather() + require.NoError(t, err) + require.NotEmpty(t, gathered) + + histogramMetricFamily = gathered[0] + msg, err = testutil.MetricFamilyToProtobuf(histogramMetricFamily) + require.NoError(t, err) + + now = time.Now() + total, added, seriesAdded, err = sl.append(app, msg, "application/vnd.google.protobuf", now) + require.NoError(t, err) + require.Equal(t, 1, total) + require.Equal(t, 1, added) + require.Equal(t, 0, seriesAdded) + + err = targetScrapeNativeHistogramBucketLimit.Write(&metric) + require.NoError(t, err) + metricValue = metric.GetCounter().GetValue() + require.Equal(t, beforeMetricValue+1, metricValue) +} + func TestScrapeLoop_ChangingMetricString(t *testing.T) { // This is a regression test for the scrape loop cache not properly maintaining // IDs when the string representation of a metric changes across a scrape. Thus diff --git a/util/testutil/protobuf.go b/util/testutil/protobuf.go new file mode 100644 index 0000000000..8650f13730 --- /dev/null +++ b/util/testutil/protobuf.go @@ -0,0 +1,50 @@ +// Copyright 2023 The Prometheus Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package testutil + +import ( + "bytes" + "encoding/binary" + + "github.com/gogo/protobuf/proto" + dto "github.com/prometheus/client_model/go" +) + +// Write a MetricFamily into a protobuf +func MetricFamilyToProtobuf(metricFamily *dto.MetricFamily) ([]byte, error) { + buffer := &bytes.Buffer{} + err := AddMetricFamilyToProtobuf(buffer, metricFamily) + if err != nil { + return nil, err + } + return buffer.Bytes(), nil +} + +// Append a MetricFamily protobuf representation to a buffer +func AddMetricFamilyToProtobuf(buffer *bytes.Buffer, metricFamily *dto.MetricFamily) error { + protoBuf, err := proto.Marshal(metricFamily) + if err != nil { + return err + } + + varintBuf := make([]byte, binary.MaxVarintLen32) + varintLength := binary.PutUvarint(varintBuf, uint64(len(protoBuf))) + + _, err = buffer.Write(varintBuf[:varintLength]) + if err != nil { + return err + } + _, err = buffer.Write(protoBuf) + return err +} From 2ad39baa726958d9688ae2094bcedf7002c46c56 Mon Sep 17 00:00:00 2001 From: Jeanette Tan Date: Sat, 22 Apr 2023 03:14:19 +0800 Subject: [PATCH 116/231] Treat bucket limit like sample limit and make it fail the whole scrape and return an error Signed-off-by: Jeanette Tan --- config/config.go | 4 +-- docs/configuration/configuration.md | 4 +-- scrape/scrape.go | 40 ++++++++++++++++------------- scrape/scrape_test.go | 7 +++-- scrape/target.go | 9 ++++--- storage/interface.go | 1 - 6 files changed, 37 insertions(+), 28 deletions(-) diff --git a/config/config.go b/config/config.go index 956e1c6c65..31ebb288e3 100644 --- a/config/config.go +++ b/config/config.go @@ -489,8 +489,8 @@ type ScrapeConfig struct { // More than this label value length post metric-relabeling will cause the // scrape to fail. LabelValueLengthLimit uint `yaml:"label_value_length_limit,omitempty"` - // More than this many buckets in a native histogram will cause the histogram - // to be ignored, but it will not make the whole scrape fail. + // More than this many buckets in a native histogram will cause the scrape to + // fail. NativeHistogramBucketLimit uint `yaml:"bucket_limit,omitempty"` // We cannot do proper Go type embedding below as the parser will then parse diff --git a/docs/configuration/configuration.md b/docs/configuration/configuration.md index f51227320d..cfb11e21e4 100644 --- a/docs/configuration/configuration.md +++ b/docs/configuration/configuration.md @@ -378,8 +378,8 @@ metric_relabel_configs: [ target_limit: | default = 0 ] # Limit on total number of positive and negative buckets allowed in a native -# histogram. If this is exceeded, the histogram will be ignored, but this will -# not make the scrape fail. 0 means no limit. +# histogram. If this is exceeded, the entire scrape will be treated as failed. +# 0 means no limit. [ sample_limit: | default = 0 ] ``` diff --git a/scrape/scrape.go b/scrape/scrape.go index 7c24c1070b..179adbb2a2 100644 --- a/scrape/scrape.go +++ b/scrape/scrape.go @@ -194,7 +194,7 @@ var ( targetScrapeNativeHistogramBucketLimit = prometheus.NewCounter( prometheus.CounterOpts{ Name: "prometheus_target_scrapes_histogram_exceeded_bucket_limit_total", - Help: "Total number of native histograms rejected due to exceeding the bucket limit.", + Help: "Total number of scrapes that hit the native histograms bucket limit and were rejected.", }, ) ) @@ -1485,11 +1485,10 @@ func (sl *scrapeLoop) getCache() *scrapeCache { } type appendErrors struct { - numOutOfOrder int - numDuplicates int - numOutOfBounds int - numExemplarOutOfOrder int - numHistogramBucketLimit int + numOutOfOrder int + numDuplicates int + numOutOfBounds int + numExemplarOutOfOrder int } func (sl *scrapeLoop) append(app storage.Appender, b []byte, contentType string, ts time.Time) (total, added, seriesAdded int, err error) { @@ -1506,6 +1505,7 @@ func (sl *scrapeLoop) append(app storage.Appender, b []byte, contentType string, defTime = timestamp.FromTime(ts) appErrs = appendErrors{} sampleLimitErr error + bucketLimitErr error e exemplar.Exemplar // escapes to heap so hoisted out of loop meta metadata.Metadata metadataChanged bool @@ -1655,7 +1655,7 @@ loop: } else { ref, err = app.Append(ref, lset, t, val) } - sampleAdded, err = sl.checkAddError(ce, met, parsedTimestamp, err, &sampleLimitErr, &appErrs) + sampleAdded, err = sl.checkAddError(ce, met, parsedTimestamp, err, &sampleLimitErr, &bucketLimitErr, &appErrs) if err != nil { if err != storage.ErrNotFound { level.Debug(sl.l).Log("msg", "Unexpected error", "series", string(met), "err", err) @@ -1669,7 +1669,7 @@ loop: sl.cache.trackStaleness(hash, lset) } sl.cache.addRef(met, ref, lset, hash) - if sampleAdded && sampleLimitErr == nil { + if sampleAdded && sampleLimitErr == nil && bucketLimitErr == nil { seriesAdded++ } } @@ -1705,6 +1705,13 @@ loop: // We only want to increment this once per scrape, so this is Inc'd outside the loop. targetScrapeSampleLimit.Inc() } + if bucketLimitErr != nil { + if err == nil { + err = bucketLimitErr // if sample limit is hit, that error takes precedence + } + // We only want to increment this once per scrape, so this is Inc'd outside the loop. + targetScrapeNativeHistogramBucketLimit.Inc() + } if appErrs.numOutOfOrder > 0 { level.Warn(sl.l).Log("msg", "Error on ingesting out-of-order samples", "num_dropped", appErrs.numOutOfOrder) } @@ -1717,9 +1724,6 @@ loop: if appErrs.numExemplarOutOfOrder > 0 { level.Warn(sl.l).Log("msg", "Error on ingesting out-of-order exemplars", "num_dropped", appErrs.numExemplarOutOfOrder) } - if appErrs.numHistogramBucketLimit > 0 { - level.Warn(sl.l).Log("msg", "Error on ingesting native histograms that exceeded bucket limit", "num_dropped", appErrs.numHistogramBucketLimit) - } if err == nil { sl.cache.forEachStale(func(lset labels.Labels) bool { // Series no longer exposed, mark it stale. @@ -1737,8 +1741,8 @@ loop: } // Adds samples to the appender, checking the error, and then returns the # of samples added, -// whether the caller should continue to process more samples, and any sample limit errors. -func (sl *scrapeLoop) checkAddError(ce *cacheEntry, met []byte, tp *int64, err error, sampleLimitErr *error, appErrs *appendErrors) (bool, error) { +// whether the caller should continue to process more samples, and any sample or bucket limit errors. +func (sl *scrapeLoop) checkAddError(ce *cacheEntry, met []byte, tp *int64, err error, sampleLimitErr, bucketLimitErr *error, appErrs *appendErrors) (bool, error) { switch errors.Cause(err) { case nil: if tp == nil && ce != nil { @@ -1762,16 +1766,16 @@ func (sl *scrapeLoop) checkAddError(ce *cacheEntry, met []byte, tp *int64, err e level.Debug(sl.l).Log("msg", "Out of bounds metric", "series", string(met)) targetScrapeSampleOutOfBounds.Inc() return false, nil - case storage.ErrHistogramBucketLimit: - appErrs.numHistogramBucketLimit++ - level.Debug(sl.l).Log("msg", "Exceeded bucket limit for native histograms", "series", string(met)) - targetScrapeNativeHistogramBucketLimit.Inc() - return false, nil case errSampleLimit: // Keep on parsing output if we hit the limit, so we report the correct // total number of samples scraped. *sampleLimitErr = err return false, nil + case errBucketLimit: + // Keep on parsing output if we hit the limit, so we report the correct + // total number of samples scraped. + *bucketLimitErr = err + return false, nil default: return false, err } diff --git a/scrape/scrape_test.go b/scrape/scrape_test.go index b5cabea52e..8822ed96de 100644 --- a/scrape/scrape_test.go +++ b/scrape/scrape_test.go @@ -1788,7 +1788,10 @@ func TestScrapeLoop_HistogramBucketLimit(t *testing.T) { now = time.Now() total, added, seriesAdded, err = sl.append(app, msg, "application/vnd.google.protobuf", now) - require.NoError(t, err) + if err != errBucketLimit { + t.Fatalf("Did not see expected histogram bucket limit error: %s", err) + } + require.NoError(t, app.Rollback()) require.Equal(t, 1, total) require.Equal(t, 1, added) require.Equal(t, 0, seriesAdded) @@ -3010,7 +3013,7 @@ func TestReuseCacheRace(*testing.T) { func TestCheckAddError(t *testing.T) { var appErrs appendErrors sl := scrapeLoop{l: log.NewNopLogger()} - sl.checkAddError(nil, nil, nil, storage.ErrOutOfOrderSample, nil, &appErrs) + sl.checkAddError(nil, nil, nil, storage.ErrOutOfOrderSample, nil, nil, &appErrs) require.Equal(t, 1, appErrs.numOutOfOrder) } diff --git a/scrape/target.go b/scrape/target.go index f916e45490..a655e85413 100644 --- a/scrape/target.go +++ b/scrape/target.go @@ -314,7 +314,10 @@ func (ts Targets) Len() int { return len(ts) } func (ts Targets) Less(i, j int) bool { return ts[i].URL().String() < ts[j].URL().String() } func (ts Targets) Swap(i, j int) { ts[i], ts[j] = ts[j], ts[i] } -var errSampleLimit = errors.New("sample limit exceeded") +var ( + errSampleLimit = errors.New("sample limit exceeded") + errBucketLimit = errors.New("histogram bucket limit exceeded") +) // limitAppender limits the number of total appended samples in a batch. type limitAppender struct { @@ -366,12 +369,12 @@ type bucketLimitAppender struct { func (app *bucketLimitAppender) AppendHistogram(ref storage.SeriesRef, lset labels.Labels, t int64, h *histogram.Histogram, fh *histogram.FloatHistogram) (storage.SeriesRef, error) { if h != nil { if len(h.PositiveBuckets)+len(h.NegativeBuckets) > app.limit { - return 0, storage.ErrHistogramBucketLimit + return 0, errBucketLimit } } if fh != nil { if len(fh.PositiveBuckets)+len(fh.NegativeBuckets) > app.limit { - return 0, storage.ErrHistogramBucketLimit + return 0, errBucketLimit } } ref, err := app.Appender.AppendHistogram(ref, lset, t, h, fh) diff --git a/storage/interface.go b/storage/interface.go index 18119f2547..b282f1fc62 100644 --- a/storage/interface.go +++ b/storage/interface.go @@ -46,7 +46,6 @@ var ( ErrHistogramNegativeBucketCount = errors.New("histogram has a bucket whose observation count is negative") ErrHistogramSpanNegativeOffset = errors.New("histogram has a span whose offset is negative") ErrHistogramSpansBucketsMismatch = errors.New("histogram spans specify different number of buckets than provided") - ErrHistogramBucketLimit = errors.New("histogram bucket limit exceeded") ) // SeriesRef is a generic series reference. In prometheus it is either a From 13938d0ccc74b6e094cdf9cd88b921285e670081 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Sat, 22 Apr 2023 10:16:49 +0200 Subject: [PATCH 117/231] Update test_golang_oldest to 1.19 Since test_go is on 1.20, and it should use the previous version. Signed-off-by: Bryan Boreham --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ee63ee300d..bfa1ac00d0 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -66,7 +66,7 @@ jobs: runs-on: ubuntu-latest # The go verson in this image should be N-1 wrt test_go. container: - image: quay.io/prometheus/golang-builder:1.18-base + image: quay.io/prometheus/golang-builder:1.19-base steps: - uses: actions/checkout@v3 - run: make build From d281ebb178a9c1b51f74cd61afbdf16c1ca5f1f8 Mon Sep 17 00:00:00 2001 From: Vladimir Varankin Date: Tue, 28 Mar 2023 20:47:18 +0200 Subject: [PATCH 118/231] web: display GOMEMLIMIT in runtime info Signed-off-by: Vladimir Varankin --- web/api/v1/api.go | 1 + web/web.go | 2 ++ 2 files changed, 3 insertions(+) diff --git a/web/api/v1/api.go b/web/api/v1/api.go index c43f4573ab..e1168e1a66 100644 --- a/web/api/v1/api.go +++ b/web/api/v1/api.go @@ -143,6 +143,7 @@ type RuntimeInfo struct { CorruptionCount int64 `json:"corruptionCount"` GoroutineCount int `json:"goroutineCount"` GOMAXPROCS int `json:"GOMAXPROCS"` + GOMEMLIMIT int64 `json:"GOMEMLIMIT"` GOGC string `json:"GOGC"` GODEBUG string `json:"GODEBUG"` StorageRetention string `json:"storageRetention"` diff --git a/web/web.go b/web/web.go index c8a32e0a66..27378b3b81 100644 --- a/web/web.go +++ b/web/web.go @@ -29,6 +29,7 @@ import ( "path" "path/filepath" "runtime" + "runtime/debug" "strconv" "strings" "sync" @@ -710,6 +711,7 @@ func (h *Handler) runtimeInfo() (api_v1.RuntimeInfo, error) { CWD: h.cwd, GoroutineCount: runtime.NumGoroutine(), GOMAXPROCS: runtime.GOMAXPROCS(0), + GOMEMLIMIT: debug.SetMemoryLimit(-1), GOGC: os.Getenv("GOGC"), GODEBUG: os.Getenv("GODEBUG"), } From aeccf9e7700a21268622a5bfb3b8e29acde95360 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Fri, 21 Apr 2023 18:12:57 +0200 Subject: [PATCH 119/231] Bump version to 2.44.0-rc0 Including CHANGELOG. Signed-off-by: Bryan Boreham --- CHANGELOG.md | 15 +++++++++++++++ VERSION | 2 +- web/ui/module/codemirror-promql/package.json | 4 ++-- web/ui/module/lezer-promql/package.json | 2 +- web/ui/package-lock.json | 14 +++++++------- web/ui/react-app/package.json | 4 ++-- 6 files changed, 28 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 13bdde0316..526ec89c6f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,20 @@ # Changelog +## 2.44.0-rc.0 / 2023-04-22 + +* [CHANGE] remote-write: raise default samples per send to 2,000. #12203 +* [FEATURE] remote-read: handle native histograms. #12085, #12192 +* [FEATURE] promtool: health and readiness check of prometheus server in CLI. #12096 +* [FEATURE] promql engine: add query_samples_total metric, the total number of samples loaded by all queries. #12251 +* [ENHANCEMENT] scraping: reduce memory allocations on Target labels. #12084 +* [ENHANCEMENT] promql: use faster heap method for topk/bottomk. #12190 +* [ENHANCEMENT] rules API: Allow filtering by rule name. #12270 +* [ENHANCEMENT] native histograms: various fixes and improvements. #11687, #12264, #12272 +* [ENHANCEMENT] ui: search of "Scrape Pools" is now case-insensitive. #12207 +* [ENHANCEMENT] tsdb: add an affirmative log message for successful WAL repair. #12135 +* [BUGFIX] tsdb: block compaction failed when shutting down. #12179 +* [BUGFIX] tsdb: out-of-order chunks could be ignored if the write-behind log was deleted. #12127 + ## 2.43.0 / 2023-03-21 We are working on some performance improvements in Prometheus, which are only diff --git a/VERSION b/VERSION index 5b9cd9afd5..a4494d974d 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.43.0 +2.44.0-rc.0 diff --git a/web/ui/module/codemirror-promql/package.json b/web/ui/module/codemirror-promql/package.json index f20a634683..6521c87298 100644 --- a/web/ui/module/codemirror-promql/package.json +++ b/web/ui/module/codemirror-promql/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/codemirror-promql", - "version": "0.43.0", + "version": "0.44.0-rc.0", "description": "a CodeMirror mode for the PromQL language", "types": "dist/esm/index.d.ts", "module": "dist/esm/index.js", @@ -29,7 +29,7 @@ }, "homepage": "https://github.com/prometheus/prometheus/blob/main/web/ui/module/codemirror-promql/README.md", "dependencies": { - "@prometheus-io/lezer-promql": "0.43.0", + "@prometheus-io/lezer-promql": "0.44.0-rc.0", "lru-cache": "^6.0.0" }, "devDependencies": { diff --git a/web/ui/module/lezer-promql/package.json b/web/ui/module/lezer-promql/package.json index 411251e56c..be92ac46f6 100644 --- a/web/ui/module/lezer-promql/package.json +++ b/web/ui/module/lezer-promql/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/lezer-promql", - "version": "0.43.0", + "version": "0.44.0-rc.0", "description": "lezer-based PromQL grammar", "main": "dist/index.cjs", "type": "module", diff --git a/web/ui/package-lock.json b/web/ui/package-lock.json index 2e334cf9e4..09a4fd3fef 100644 --- a/web/ui/package-lock.json +++ b/web/ui/package-lock.json @@ -28,10 +28,10 @@ }, "module/codemirror-promql": { "name": "@prometheus-io/codemirror-promql", - "version": "0.43.0", + "version": "0.44.0-rc.0", "license": "Apache-2.0", "dependencies": { - "@prometheus-io/lezer-promql": "0.43.0", + "@prometheus-io/lezer-promql": "0.44.0-rc.0", "lru-cache": "^6.0.0" }, "devDependencies": { @@ -61,7 +61,7 @@ }, "module/lezer-promql": { "name": "@prometheus-io/lezer-promql", - "version": "0.43.0", + "version": "0.44.0-rc.0", "license": "Apache-2.0", "devDependencies": { "@lezer/generator": "^1.2.2", @@ -20763,7 +20763,7 @@ }, "react-app": { "name": "@prometheus-io/app", - "version": "0.43.0", + "version": "0.44.0-rc.0", "dependencies": { "@codemirror/autocomplete": "^6.4.0", "@codemirror/commands": "^6.2.0", @@ -20781,7 +20781,7 @@ "@lezer/lr": "^1.3.1", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.43.0", + "@prometheus-io/codemirror-promql": "0.44.0-rc.0", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", "downshift": "^7.2.0", @@ -23417,7 +23417,7 @@ "@lezer/lr": "^1.3.1", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.43.0", + "@prometheus-io/codemirror-promql": "0.44.0-rc.0", "@testing-library/react-hooks": "^7.0.2", "@types/enzyme": "^3.10.12", "@types/flot": "0.0.32", @@ -23468,7 +23468,7 @@ "@lezer/common": "^1.0.2", "@lezer/highlight": "^1.1.3", "@lezer/lr": "^1.3.1", - "@prometheus-io/lezer-promql": "0.43.0", + "@prometheus-io/lezer-promql": "0.44.0-rc.0", "@types/lru-cache": "^5.1.1", "isomorphic-fetch": "^3.0.0", "lru-cache": "^6.0.0", diff --git a/web/ui/react-app/package.json b/web/ui/react-app/package.json index c8b115582a..b4929d664a 100644 --- a/web/ui/react-app/package.json +++ b/web/ui/react-app/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/app", - "version": "0.43.0", + "version": "0.44.0-rc.0", "private": true, "dependencies": { "@codemirror/autocomplete": "^6.4.0", @@ -19,7 +19,7 @@ "@lezer/common": "^1.0.2", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.43.0", + "@prometheus-io/codemirror-promql": "0.44.0-rc.0", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", "downshift": "^7.2.0", From dfabc69303a92804883f597570569b0e95fd9836 Mon Sep 17 00:00:00 2001 From: Jeanette Tan Date: Tue, 25 Apr 2023 01:41:04 +0800 Subject: [PATCH 120/231] Add tests according to code review Signed-off-by: Jeanette Tan --- scrape/scrape_test.go | 38 +++++++++++++++------------ scrape/target_test.go | 61 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 83 insertions(+), 16 deletions(-) diff --git a/scrape/scrape_test.go b/scrape/scrape_test.go index 8822ed96de..745765ee8f 100644 --- a/scrape/scrape_test.go +++ b/scrape/scrape_test.go @@ -1742,18 +1742,24 @@ func TestScrapeLoop_HistogramBucketLimit(t *testing.T) { require.NoError(t, err) beforeMetricValue := metric.GetCounter().GetValue() - nativeHistogram := prometheus.NewHistogram(prometheus.HistogramOpts{ - Namespace: "testing", - Name: "example_native_histogram", - Help: "This is used for testing", - ConstLabels: map[string]string{"some": "value"}, - NativeHistogramBucketFactor: 1.1, // 10% increase from bucket to bucket - NativeHistogramMaxBucketNumber: 100, // intentionally higher than the limit we'll use in the scraper - }) + nativeHistogram := prometheus.NewHistogramVec( + prometheus.HistogramOpts{ + Namespace: "testing", + Name: "example_native_histogram", + Help: "This is used for testing", + ConstLabels: map[string]string{"some": "value"}, + NativeHistogramBucketFactor: 1.1, // 10% increase from bucket to bucket + NativeHistogramMaxBucketNumber: 100, // intentionally higher than the limit we'll use in the scraper + }, + []string{"size"}, + ) registry := prometheus.NewRegistry() registry.Register(nativeHistogram) - nativeHistogram.Observe(1.0) - nativeHistogram.Observe(10.0) // in different bucket since > 1*1.1 + nativeHistogram.WithLabelValues("S").Observe(1.0) + nativeHistogram.WithLabelValues("M").Observe(1.0) + nativeHistogram.WithLabelValues("L").Observe(1.0) + nativeHistogram.WithLabelValues("M").Observe(10.0) + nativeHistogram.WithLabelValues("L").Observe(10.0) // in different bucket since > 1*1.1 gathered, err := registry.Gather() require.NoError(t, err) @@ -1766,9 +1772,9 @@ func TestScrapeLoop_HistogramBucketLimit(t *testing.T) { now := time.Now() total, added, seriesAdded, err := sl.append(app, msg, "application/vnd.google.protobuf", now) require.NoError(t, err) - require.Equal(t, 1, total) - require.Equal(t, 1, added) - require.Equal(t, 1, seriesAdded) + require.Equal(t, 3, total) + require.Equal(t, 3, added) + require.Equal(t, 3, seriesAdded) err = targetScrapeNativeHistogramBucketLimit.Write(&metric) require.NoError(t, err) @@ -1776,7 +1782,7 @@ func TestScrapeLoop_HistogramBucketLimit(t *testing.T) { require.Equal(t, beforeMetricValue, metricValue) beforeMetricValue = metricValue - nativeHistogram.Observe(100.0) // in different bucket since > 10*1.1 + nativeHistogram.WithLabelValues("L").Observe(100.0) // in different bucket since > 10*1.1 gathered, err = registry.Gather() require.NoError(t, err) @@ -1792,8 +1798,8 @@ func TestScrapeLoop_HistogramBucketLimit(t *testing.T) { t.Fatalf("Did not see expected histogram bucket limit error: %s", err) } require.NoError(t, app.Rollback()) - require.Equal(t, 1, total) - require.Equal(t, 1, added) + require.Equal(t, 3, total) + require.Equal(t, 3, added) require.Equal(t, 0, seriesAdded) err = targetScrapeNativeHistogramBucketLimit.Write(&metric) diff --git a/scrape/target_test.go b/scrape/target_test.go index 9d25df4149..12d3b5a4d7 100644 --- a/scrape/target_test.go +++ b/scrape/target_test.go @@ -31,6 +31,7 @@ import ( "github.com/prometheus/prometheus/config" "github.com/prometheus/prometheus/discovery/targetgroup" + "github.com/prometheus/prometheus/model/histogram" "github.com/prometheus/prometheus/model/labels" ) @@ -488,3 +489,63 @@ scrape_configs: }) } } + +func TestBucketLimitAppender(t *testing.T) { + example := histogram.Histogram{ + Schema: 0, + Count: 21, + Sum: 33, + ZeroThreshold: 0.001, + ZeroCount: 3, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 3}, + }, + PositiveBuckets: []int64{3, 0, 0}, + NegativeSpans: []histogram.Span{ + {Offset: 0, Length: 3}, + }, + NegativeBuckets: []int64{3, 0, 0}, + } + + cases := []struct { + h histogram.Histogram + limit int + expectError bool + }{ + { + h: example, + limit: 3, + expectError: true, + }, + { + h: example, + limit: 10, + expectError: false, + }, + } + + resApp := &collectResultAppender{} + + for _, c := range cases { + for _, floatHisto := range []bool{true, false} { + t.Run(fmt.Sprintf("floatHistogram=%t", floatHisto), func(t *testing.T) { + app := &bucketLimitAppender{Appender: resApp, limit: c.limit} + ts := int64(10 * time.Minute / time.Millisecond) + h := c.h + lbls := labels.FromStrings("__name__", "sparse_histogram_series") + var err error + if floatHisto { + _, err = app.AppendHistogram(0, lbls, ts, nil, h.Copy().ToFloat()) + } else { + _, err = app.AppendHistogram(0, lbls, ts, h.Copy(), nil) + } + if c.expectError { + require.Error(t, err) + } else { + require.NoError(t, err) + } + require.NoError(t, app.Commit()) + }) + } + } +} From 276ca6a883bed19c2c78c356f93e0b939841f584 Mon Sep 17 00:00:00 2001 From: cui fliter Date: Tue, 25 Apr 2023 14:19:16 +0800 Subject: [PATCH 121/231] fix some comments Signed-off-by: cui fliter --- CONTRIBUTING.md | 2 +- discovery/kubernetes/node.go | 2 +- model/labels/labels.go | 2 +- model/labels/labels_string.go | 2 +- tsdb/db.go | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2fbed38809..57055ef38c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -85,7 +85,7 @@ The PromQL parser grammar is located in `promql/parser/generated_parser.y` and i The parser is built using [goyacc](https://pkg.go.dev/golang.org/x/tools/cmd/goyacc) If doing some sort of debugging, then it is possible to add some verbose output. After generating the parser, then you -can modify the the `./promql/parser/generated_parser.y.go` manually. +can modify the `./promql/parser/generated_parser.y.go` manually. ```golang // As of writing this was somewhere around line 600. diff --git a/discovery/kubernetes/node.go b/discovery/kubernetes/node.go index 16a06e7a0c..d0a6d2780d 100644 --- a/discovery/kubernetes/node.go +++ b/discovery/kubernetes/node.go @@ -209,7 +209,7 @@ func (n *Node) buildNode(node *apiv1.Node) *targetgroup.Group { return tg } -// nodeAddresses returns the provided node's address, based on the priority: +// nodeAddress returns the provided node's address, based on the priority: // 1. NodeInternalIP // 2. NodeInternalDNS // 3. NodeExternalIP diff --git a/model/labels/labels.go b/model/labels/labels.go index 93524ddcfc..4f8aedd49d 100644 --- a/model/labels/labels.go +++ b/model/labels/labels.go @@ -622,7 +622,7 @@ func (b *ScratchBuilder) Sort() { slices.SortFunc(b.add, func(a, b Label) bool { return a.Name < b.Name }) } -// Asssign is for when you already have a Labels which you want this ScratchBuilder to return. +// Assign is for when you already have a Labels which you want this ScratchBuilder to return. func (b *ScratchBuilder) Assign(ls Labels) { b.add = append(b.add[:0], ls...) // Copy on top of our slice, so we don't retain the input slice. } diff --git a/model/labels/labels_string.go b/model/labels/labels_string.go index ff46103ebc..da8229796b 100644 --- a/model/labels/labels_string.go +++ b/model/labels/labels_string.go @@ -799,7 +799,7 @@ func (b *ScratchBuilder) Sort() { slices.SortFunc(b.add, func(a, b Label) bool { return a.Name < b.Name }) } -// Asssign is for when you already have a Labels which you want this ScratchBuilder to return. +// Assign is for when you already have a Labels which you want this ScratchBuilder to return. func (b *ScratchBuilder) Assign(l Labels) { b.output = l } diff --git a/tsdb/db.go b/tsdb/db.go index de37bdb4f9..61333c9720 100644 --- a/tsdb/db.go +++ b/tsdb/db.go @@ -1841,7 +1841,7 @@ func (db *DB) Querier(_ context.Context, mint, maxt int64) (storage.Querier, err return storage.NewMergeQuerier(blockQueriers, nil, storage.ChainedSeriesMerge), nil } -// blockQueriersForRange returns individual block chunk queriers from the persistent blocks, in-order head block, and the +// blockChunkQuerierForRange returns individual block chunk queriers from the persistent blocks, in-order head block, and the // out-of-order head block, overlapping with the given time range. func (db *DB) blockChunkQuerierForRange(mint, maxt int64) ([]storage.ChunkQuerier, error) { var blocks []BlockReader From 8bfd1621989a0cfb4c18743588e1944fd99fc363 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Tue, 25 Apr 2023 10:03:41 +0000 Subject: [PATCH 122/231] Review feedback - mostly capitalization Signed-off-by: Bryan Boreham --- CHANGELOG.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 526ec89c6f..9daeaae040 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,18 +2,18 @@ ## 2.44.0-rc.0 / 2023-04-22 -* [CHANGE] remote-write: raise default samples per send to 2,000. #12203 -* [FEATURE] remote-read: handle native histograms. #12085, #12192 -* [FEATURE] promtool: health and readiness check of prometheus server in CLI. #12096 -* [FEATURE] promql engine: add query_samples_total metric, the total number of samples loaded by all queries. #12251 -* [ENHANCEMENT] scraping: reduce memory allocations on Target labels. #12084 -* [ENHANCEMENT] promql: use faster heap method for topk/bottomk. #12190 -* [ENHANCEMENT] rules API: Allow filtering by rule name. #12270 -* [ENHANCEMENT] native histograms: various fixes and improvements. #11687, #12264, #12272 -* [ENHANCEMENT] ui: search of "Scrape Pools" is now case-insensitive. #12207 -* [ENHANCEMENT] tsdb: add an affirmative log message for successful WAL repair. #12135 -* [BUGFIX] tsdb: block compaction failed when shutting down. #12179 -* [BUGFIX] tsdb: out-of-order chunks could be ignored if the write-behind log was deleted. #12127 +* [CHANGE] Remote-write: Raise default samples per send to 2,000. #12203 +* [FEATURE] Remote-read: Handle native histograms. #12085, #12192 +* [FEATURE] Promtool: Health and readiness check of prometheus server in CLI. #12096 +* [FEATURE] PromQL: Add `query_samples_total` metric, the total number of samples loaded by all queries. #12251 +* [ENHANCEMENT] Scrape: Reduce memory allocations on target labels. #12084 +* [ENHANCEMENT] PromQL: Use faster heap method for `topk()` / `bottomk()`. #12190 +* [ENHANCEMENT] Rules API: Allow filtering by rule name. #12270 +* [ENHANCEMENT] Native Histograms: Various fixes and improvements. #11687, #12264, #12272 +* [ENHANCEMENT] UI: Search of scraping pools is now case-insensitive. #12207 +* [ENHANCEMENT] TSDB: Add an affirmative log message for successful WAL repair. #12135 +* [BUGFIX] TSDB: Block compaction failed when shutting down. #12179 +* [BUGFIX] TSDB: Out-of-order chunks could be ignored if the write-behind log was deleted. #12127 ## 2.43.0 / 2023-03-21 From 6b25e9a923ec5f5a7274a012af4c94f8fb669b9f Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Tue, 25 Apr 2023 10:07:26 +0000 Subject: [PATCH 123/231] build: turn on stringlabels by default This setting uses less memory, and was optional in previous release 2.43. Signed-off-by: Bryan Boreham --- .promu.yml | 2 ++ CHANGELOG.md | 4 ++++ 2 files changed, 6 insertions(+) diff --git a/.promu.yml b/.promu.yml index ef69c35c8e..f724dc34f3 100644 --- a/.promu.yml +++ b/.promu.yml @@ -14,8 +14,10 @@ build: all: - netgo - builtinassets + - stringlabels windows: - builtinassets + - stringlabels flags: -a ldflags: | -X github.com/prometheus/common/version.Version={{.Version}} diff --git a/CHANGELOG.md b/CHANGELOG.md index 9daeaae040..8fc24345cc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## 2.44.0-rc.0 / 2023-04-22 +This version is built with Go tag `stringlabels`, to use the smaller data +structure for Labels that was optional in the previous release. For more +details about this code change see #10991. + * [CHANGE] Remote-write: Raise default samples per send to 2,000. #12203 * [FEATURE] Remote-read: Handle native histograms. #12085, #12192 * [FEATURE] Promtool: Health and readiness check of prometheus server in CLI. #12096 From c06c02b3b10b3d98bd7d59400d0a1b3762e90af3 Mon Sep 17 00:00:00 2001 From: Paschalis Tsilias Date: Tue, 25 Apr 2023 16:27:39 +0300 Subject: [PATCH 124/231] Fix recommended protoc version Signed-off-by: Paschalis Tsilias --- prompb/README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/prompb/README.md b/prompb/README.md index 8c19b17e9d..1b85bc2f3c 100644 --- a/prompb/README.md +++ b/prompb/README.md @@ -4,6 +4,8 @@ re-compile them when building Prometheus. If however you have modified the defs and do need to re-compile, run `make proto` from the parent dir. -In order for the script to run, you'll need `protoc` (version 3.12.3) in your -PATH. +In order for the [script][] to run, you'll need `protoc` (version 3.15.8) in +your PATH. + +[script]: ../scripts/genproto.sh From 417e847662e248e2a0213f9ee34cc99326f5f4e0 Mon Sep 17 00:00:00 2001 From: Paschalis Tsilias Date: Tue, 25 Apr 2023 17:17:22 +0300 Subject: [PATCH 125/231] Update prompb/README.md Co-authored-by: Julien Pivotto Signed-off-by: Paschalis Tsilias --- prompb/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prompb/README.md b/prompb/README.md index 1b85bc2f3c..c7c31399c1 100644 --- a/prompb/README.md +++ b/prompb/README.md @@ -4,7 +4,7 @@ re-compile them when building Prometheus. If however you have modified the defs and do need to re-compile, run `make proto` from the parent dir. -In order for the [script][] to run, you'll need `protoc` (version 3.15.8) in +In order for the [script][../scripts/genproto.sh] to run, you'll need `protoc` (version 3.15.8) in your PATH. [script]: ../scripts/genproto.sh From 38de61b59b38b35fc33b1ef8f9a35f394e733f13 Mon Sep 17 00:00:00 2001 From: Jesus Vazquez Date: Tue, 25 Apr 2023 17:58:42 +0200 Subject: [PATCH 126/231] Propose Jesus Vazquez as 2.45 release shepherd Signed-off-by: Jesus Vazquez --- RELEASE.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/RELEASE.md b/RELEASE.md index 26f72ba628..d7f24dabd5 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -49,7 +49,8 @@ Release cadence of first pre-releases being cut is 6 weeks. | v2.42 | 2023-01-25 | Kemal Akkoyun (GitHub: @kakkoyun) | | v2.43 | 2023-03-08 | Julien Pivotto (GitHub: @roidelapluie) | | v2.44 | 2023-04-19 | Bryan Boreham (GitHub: @bboreham) | -| v2.45 | 2023-05-31 | **searching for volunteer** | +| v2.45 | 2023-05-31 | Jesus Vazquez (Github: @jesusvazquez) | +| v2.46 | 2023-07-12 | **searching for volunteer** | If you are interested in volunteering please create a pull request against the [prometheus/prometheus](https://github.com/prometheus/prometheus) repository and propose yourself for the release series of your choice. From 93f64754c049f05274c4fdbfd20e5054da7c4682 Mon Sep 17 00:00:00 2001 From: Paschalis Tsilias Date: Wed, 26 Apr 2023 12:43:19 +0300 Subject: [PATCH 127/231] Remove extra line Signed-off-by: Paschalis Tsilias --- prompb/README.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/prompb/README.md b/prompb/README.md index c7c31399c1..d71438afb9 100644 --- a/prompb/README.md +++ b/prompb/README.md @@ -6,6 +6,3 @@ If however you have modified the defs and do need to re-compile, run In order for the [script][../scripts/genproto.sh] to run, you'll need `protoc` (version 3.15.8) in your PATH. - -[script]: ../scripts/genproto.sh - From 55626c6911e0ceb87920d94e5a15c8cf6c49d962 Mon Sep 17 00:00:00 2001 From: Paschalis Tsilias Date: Wed, 26 Apr 2023 12:44:15 +0300 Subject: [PATCH 128/231] Fix final newline Signed-off-by: Paschalis Tsilias --- prompb/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/prompb/README.md b/prompb/README.md index d71438afb9..c1acb508aa 100644 --- a/prompb/README.md +++ b/prompb/README.md @@ -6,3 +6,4 @@ If however you have modified the defs and do need to re-compile, run In order for the [script][../scripts/genproto.sh] to run, you'll need `protoc` (version 3.15.8) in your PATH. + From 8a34c43515fc0b9452ecf4f238c44a9f249d5da3 Mon Sep 17 00:00:00 2001 From: Paschalis Tsilias Date: Wed, 26 Apr 2023 12:57:20 +0300 Subject: [PATCH 129/231] Update prompb/README.md Co-authored-by: Julien Pivotto Signed-off-by: Paschalis Tsilias --- prompb/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prompb/README.md b/prompb/README.md index c1acb508aa..a33d7bfb88 100644 --- a/prompb/README.md +++ b/prompb/README.md @@ -4,6 +4,6 @@ re-compile them when building Prometheus. If however you have modified the defs and do need to re-compile, run `make proto` from the parent dir. -In order for the [script][../scripts/genproto.sh] to run, you'll need `protoc` (version 3.15.8) in +In order for the [script](../scripts/genproto.sh) to run, you'll need `protoc` (version 3.15.8) in your PATH. From 28f5502828e88ac81a4cc5f1fd00bd4b2091189f Mon Sep 17 00:00:00 2001 From: Russ Cox Date: Wed, 26 Apr 2023 10:26:58 -0400 Subject: [PATCH 130/231] scrape: fix two loop variable scoping bugs in test Consider code like: for i := 0; i < numTargets; i++ { stopFuncs = append(stopFuncs, func() { time.Sleep(i*20*time.Millisecond) }) } Because the loop variable i is shared by all closures, all the stopFuncs sleep for numTargets*20 ms. If the i were made per-iteration, as we are considering for a future Go release, the stopFuncs would have sleep durations ranging from 0 to (numTargets-1)*20 ms. Two tests had code like this and were checking that the aggregate sleep was at least numTargets*20 ms ("at least as long as the last target slept"). This is only true today because i == numTarget during all the sleeps. To keep the code working even if the semantics of this loop change, this PR computes d := time.Duration((i+1)*20) * time.Millisecond outside the closure (but inside the loop body), and then each closure has its own d. Now the sleeps range from 20 ms to numTargets*20 ms, keeping the test passing (and probably behaving closer to the intent of the test author). The failure being fixed can be reproduced by using the current Go development branch with GOEXPERIMENT=loopvar go test Signed-off-by: Russ Cox --- scrape/scrape_test.go | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/scrape/scrape_test.go b/scrape/scrape_test.go index 2a2ca09485..60e335b5b0 100644 --- a/scrape/scrape_test.go +++ b/scrape/scrape_test.go @@ -190,8 +190,9 @@ func TestScrapePoolStop(t *testing.T) { labels: labels.FromStrings(model.AddressLabel, fmt.Sprintf("example.com:%d", i)), } l := &testLoop{} + d := time.Duration((i+1)*20) * time.Millisecond l.stopFunc = func() { - time.Sleep(time.Duration(i*20) * time.Millisecond) + time.Sleep(d) mtx.Lock() stopped[t.hash()] = true @@ -273,8 +274,9 @@ func TestScrapePoolReload(t *testing.T) { discoveredLabels: labels, } l := &testLoop{} + d := time.Duration((i+1)*20) * time.Millisecond l.stopFunc = func() { - time.Sleep(time.Duration(i*20) * time.Millisecond) + time.Sleep(d) mtx.Lock() stopped[t.hash()] = true From b1bab7bc54ed06f9a8619e7dd09b7b2ed55d3ee4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fran=C3=A7ois=20Gouteroux?= Date: Thu, 27 Apr 2023 13:23:52 +0200 Subject: [PATCH 131/231] feat(promtool): add push metrics command MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: François Gouteroux --- cmd/promtool/main.go | 15 +++ cmd/promtool/metrics.go | 234 ++++++++++++++++++++++++++++++++++ docs/command-line/promtool.md | 43 +++++++ 3 files changed, 292 insertions(+) create mode 100644 cmd/promtool/metrics.go diff --git a/cmd/promtool/main.go b/cmd/promtool/main.go index de002a0b28..c4077954ea 100644 --- a/cmd/promtool/main.go +++ b/cmd/promtool/main.go @@ -178,6 +178,18 @@ func main() { queryLabelsEnd := queryLabelsCmd.Flag("end", "End time (RFC3339 or Unix timestamp).").String() queryLabelsMatch := queryLabelsCmd.Flag("match", "Series selector. Can be specified multiple times.").Strings() + pushCmd := app.Command("push", "Push to a Prometheus server.") + pushCmd.Flag("http.config.file", "HTTP client configuration file for promtool to connect to Prometheus.").PlaceHolder("").ExistingFileVar(&httpConfigFilePath) + pushMetricsCmd := pushCmd.Command("metrics", "Push metrics to a prometheus remote write.") + pushMetricsCmd.Arg("remote-write-url", "Prometheus remote write url to push metrics.").Required().URLVar(&serverURL) + metricFiles := pushMetricsCmd.Arg( + "metric-files", + "The metric files to push.", + ).Required().ExistingFiles() + metricJobLabel := pushMetricsCmd.Flag("job-label", "Job label to attach to metrics.").Default("promtool").String() + pushMetricsTimeout := pushMetricsCmd.Flag("timeout", "The time to wait for pushing metrics.").Default("30s").Duration() + pushMetricsHeaders := pushMetricsCmd.Flag("header", "Prometheus remote write header.").StringMap() + testCmd := app.Command("test", "Unit testing.") testRulesCmd := testCmd.Command("rules", "Unit tests for rules.") testRulesFiles := testRulesCmd.Arg( @@ -301,6 +313,9 @@ func main() { case checkMetricsCmd.FullCommand(): os.Exit(CheckMetrics(*checkMetricsExtended)) + case pushMetricsCmd.FullCommand(): + os.Exit(PushMetrics(serverURL, httpRoundTripper, *pushMetricsHeaders, *pushMetricsTimeout, *metricJobLabel, *metricFiles...)) + case queryInstantCmd.FullCommand(): os.Exit(QueryInstant(serverURL, httpRoundTripper, *queryInstantExpr, *queryInstantTime, p)) diff --git a/cmd/promtool/metrics.go b/cmd/promtool/metrics.go new file mode 100644 index 0000000000..21fcd3e662 --- /dev/null +++ b/cmd/promtool/metrics.go @@ -0,0 +1,234 @@ +// Copyright 2023 The Prometheus Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "bytes" + "context" + "fmt" + "io" + "net/http" + "net/url" + "os" + "sort" + "time" + + "github.com/golang/snappy" + dto "github.com/prometheus/client_model/go" + config_util "github.com/prometheus/common/config" + "github.com/prometheus/common/expfmt" + "github.com/prometheus/common/model" + + "github.com/prometheus/prometheus/prompb" + "github.com/prometheus/prometheus/storage/remote" +) + +// Push metrics to a prometheus remote write. +func PushMetrics(url *url.URL, roundTripper http.RoundTripper, headers map[string]string, timeout time.Duration, jobLabel string, files ...string) int { + // remote write should respect specification: https://prometheus.io/docs/concepts/remote_write_spec/ + failed := false + + addressURL, err := url.Parse(url.String()) + if err != nil { + fmt.Fprintln(os.Stderr, err) + return failureExitCode + } + + // build remote write client + writeClient, err := remote.NewWriteClient("remote-write", &remote.ClientConfig{ + URL: &config_util.URL{URL: addressURL}, + Timeout: model.Duration(timeout), + }) + if err != nil { + fmt.Fprintln(os.Stderr, err) + return failureExitCode + } + + // set custom tls config from httpConfigFilePath + // set custom headers to every request + client, ok := writeClient.(*remote.Client) + if !ok { + fmt.Fprintln(os.Stderr, fmt.Errorf("unexpected type %T", writeClient)) + return failureExitCode + } + client.Client.Transport = &setHeadersTransport{ + RoundTripper: roundTripper, + headers: headers, + } + + for _, f := range files { + var data []byte + var err error + data, err = os.ReadFile(f) + if err != nil { + fmt.Fprintln(os.Stderr, err) + failed = true + continue + } + + fmt.Printf("Parsing metric file %s\n", f) + metricsData, err := parseMetricsTextAndFormat(bytes.NewReader(data), jobLabel) + if err != nil { + fmt.Fprintln(os.Stderr, err) + failed = true + continue + } + + raw, err := metricsData.Marshal() + if err != nil { + fmt.Fprintln(os.Stderr, err) + failed = true + continue + } + + // Encode the request body into snappy encoding. + compressed := snappy.Encode(nil, raw) + err = client.Store(context.Background(), compressed) + if err != nil { + fmt.Fprintln(os.Stderr, err) + failed = true + continue + } + fmt.Printf("Successfully pushed metric file %s\n", f) + } + + if failed { + return failureExitCode + } + + return successExitCode +} + +type setHeadersTransport struct { + http.RoundTripper + headers map[string]string +} + +func (s *setHeadersTransport) RoundTrip(req *http.Request) (*http.Response, error) { + for key, value := range s.headers { + req.Header.Set(key, value) + } + return s.RoundTripper.RoundTrip(req) +} + +var MetricMetadataTypeValue = map[string]int32{ + "UNKNOWN": 0, + "COUNTER": 1, + "GAUGE": 2, + "HISTOGRAM": 3, + "GAUGEHISTOGRAM": 4, + "SUMMARY": 5, + "INFO": 6, + "STATESET": 7, +} + +// formatMetrics convert metric family to a writerequest +func formatMetrics(mf map[string]*dto.MetricFamily, jobLabel string) (*prompb.WriteRequest, error) { + wr := &prompb.WriteRequest{} + + // build metric list + sortedMetricNames := make([]string, 0, len(mf)) + for metric := range mf { + sortedMetricNames = append(sortedMetricNames, metric) + } + // sort metrics name in lexicographical order + sort.Strings(sortedMetricNames) + + for _, metricName := range sortedMetricNames { + // Set metadata writerequest + mtype := MetricMetadataTypeValue[mf[metricName].Type.String()] + metadata := prompb.MetricMetadata{ + MetricFamilyName: mf[metricName].GetName(), + Type: prompb.MetricMetadata_MetricType(mtype), + Help: mf[metricName].GetHelp(), + } + wr.Metadata = append(wr.Metadata, metadata) + + for _, metric := range mf[metricName].Metric { + var timeserie prompb.TimeSeries + + // build labels map + labels := make(map[string]string, len(metric.Label)+2) + labels[model.MetricNameLabel] = metricName + labels[model.JobLabel] = jobLabel + + for _, label := range metric.Label { + labelname := label.GetName() + if labelname == model.JobLabel { + labelname = fmt.Sprintf("%s%s", model.ExportedLabelPrefix, labelname) + } + labels[labelname] = label.GetValue() + } + + // build labels name list + sortedLabelNames := make([]string, 0, len(labels)) + for label := range labels { + sortedLabelNames = append(sortedLabelNames, label) + } + // sort labels name in lexicographical order + sort.Strings(sortedLabelNames) + + for _, label := range sortedLabelNames { + timeserie.Labels = append(timeserie.Labels, prompb.Label{ + Name: label, + Value: labels[label], + }) + } + + timeserie.Samples = []prompb.Sample{ + { + Timestamp: time.Now().UnixNano() / int64(time.Millisecond), + Value: getMetricsValue(metric), + }, + } + + wr.Timeseries = append(wr.Timeseries, timeserie) + } + } + return wr, nil +} + +// parseMetricsTextReader consumes an io.Reader and returns the MetricFamily +func parseMetricsTextReader(input io.Reader) (map[string]*dto.MetricFamily, error) { + var parser expfmt.TextParser + mf, err := parser.TextToMetricFamilies(input) + if err != nil { + return nil, err + } + return mf, nil +} + +// getMetricsValue return the value of a timeserie without the need to give value type +func getMetricsValue(m *dto.Metric) float64 { + switch { + case m.Gauge != nil: + return m.GetGauge().GetValue() + case m.Counter != nil: + return m.GetCounter().GetValue() + case m.Untyped != nil: + return m.GetUntyped().GetValue() + default: + return 0. + } +} + +// parseMetricsTextAndFormat return the data in the expected prometheus metrics write request format +func parseMetricsTextAndFormat(input io.Reader, jobLabel string) (*prompb.WriteRequest, error) { + mf, err := parseMetricsTextReader(input) + if err != nil { + return nil, err + } + + return formatMetrics(mf, jobLabel) +} diff --git a/docs/command-line/promtool.md b/docs/command-line/promtool.md index e149d374a0..ac159a9214 100644 --- a/docs/command-line/promtool.md +++ b/docs/command-line/promtool.md @@ -27,6 +27,7 @@ Tooling for the Prometheus monitoring system. | check | Check the resources for validity. | | query | Run query against a Prometheus server. | | debug | Fetch debug information. | +| push | Push to a Prometheus server. | | test | Unit testing. | | tsdb | Run tsdb commands. | @@ -372,6 +373,48 @@ Fetch all debug information. +### `promtool push` + +Push to a Prometheus server. + + + +#### Flags + +| Flag | Description | +| --- | --- | +| --http.config.file | HTTP client configuration file for promtool to connect to Prometheus. | + + + + +##### `promtool push metrics` + +Push metrics to a prometheus remote write. + + + +###### Flags + +| Flag | Description | Default | +| --- | --- | --- | +| --job-label | Job label to attach to metrics. | `promtool` | +| --timeout | The time to wait for pushing metrics. | `30s` | +| --header | Prometheus remote write header. | | + + + + +###### Arguments + +| Argument | Description | Required | +| --- | --- | --- | +| remote-write-url | Prometheus remote write url to push metrics. | Yes | +| metric-files | The metric files to push. | Yes | + + + + ### `promtool test` Unit testing. From 0d049feac760444cbbafadbe8794843a6f47dba4 Mon Sep 17 00:00:00 2001 From: Filip Petkovski Date: Fri, 28 Apr 2023 22:52:21 +0200 Subject: [PATCH 132/231] Fix encoding samples in ChunkSeries (#12185) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The storage.ChunkSeries iterator assumes that a histogram sample can always be appended to the currently open chunk. This is not the case when there is a counter reset, or when appending a stale sample to a chunk with non-stale samples. In addition, the open chunk sometimes needs to be recoded before a sample can be appended. This commit addresses the issue by implementing a RecodingAppender which can recode incoming samples in a transparent way. It also detects cases when a sample cannot be appended at all and returns `false` so that the caller can open a new chunk. Signed-off-by: Filip Petkovski Signed-off-by: György Krajcsovits Signed-off-by: Ganesh Vernekar Co-authored-by: György Krajcsovits Co-authored-by: Ganesh Vernekar --- storage/series.go | 192 ++++++++++++++++++++++++--- storage/series_test.go | 286 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 461 insertions(+), 17 deletions(-) diff --git a/storage/series.go b/storage/series.go index f609df3f02..5daa6255da 100644 --- a/storage/series.go +++ b/storage/series.go @@ -281,7 +281,7 @@ func NewSeriesToChunkEncoder(series Series) ChunkSeries { func (s *seriesToChunkEncoder) Iterator(it chunks.Iterator) chunks.Iterator { var ( chk chunkenc.Chunk - app chunkenc.Appender + app *RecodingAppender err error ) mint := int64(math.MaxInt64) @@ -299,21 +299,16 @@ func (s *seriesToChunkEncoder) Iterator(it chunks.Iterator) chunks.Iterator { for typ := seriesIter.Next(); typ != chunkenc.ValNone; typ = seriesIter.Next() { if typ != lastType || i >= seriesToChunkEncoderSplit { // Create a new chunk if the sample type changed or too many samples in the current one. - if chk != nil { - chks = append(chks, chunks.Meta{ - MinTime: mint, - MaxTime: maxt, - Chunk: chk, - }) - } + chks = appendChunk(chks, mint, maxt, chk) chk, err = chunkenc.NewEmptyChunk(typ.ChunkEncoding()) if err != nil { return errChunksIterator{err: err} } - app, err = chk.Appender() + chkAppender, err := chk.Appender() if err != nil { return errChunksIterator{err: err} } + app = NewRecodingAppender(&chk, chkAppender) mint = int64(math.MaxInt64) // maxt is immediately overwritten below which is why setting it here won't make a difference. i = 0 @@ -332,10 +327,45 @@ func (s *seriesToChunkEncoder) Iterator(it chunks.Iterator) chunks.Iterator { app.Append(t, v) case chunkenc.ValHistogram: t, h = seriesIter.AtHistogram() - app.AppendHistogram(t, h) + if ok, counterReset := app.AppendHistogram(t, h); !ok { + chks = appendChunk(chks, mint, maxt, chk) + histChunk := chunkenc.NewHistogramChunk() + if counterReset { + histChunk.SetCounterResetHeader(chunkenc.CounterReset) + } + chk = histChunk + + chkAppender, err := chk.Appender() + if err != nil { + return errChunksIterator{err: err} + } + mint = int64(math.MaxInt64) + i = 0 + app = NewRecodingAppender(&chk, chkAppender) + if ok, _ := app.AppendHistogram(t, h); !ok { + panic("unexpected error while appending histogram") + } + } case chunkenc.ValFloatHistogram: t, fh = seriesIter.AtFloatHistogram() - app.AppendFloatHistogram(t, fh) + if ok, counterReset := app.AppendFloatHistogram(t, fh); !ok { + chks = appendChunk(chks, mint, maxt, chk) + floatHistChunk := chunkenc.NewFloatHistogramChunk() + if counterReset { + floatHistChunk.SetCounterResetHeader(chunkenc.CounterReset) + } + chk = floatHistChunk + chkAppender, err := chk.Appender() + if err != nil { + return errChunksIterator{err: err} + } + mint = int64(math.MaxInt64) + i = 0 + app = NewRecodingAppender(&chk, chkAppender) + if ok, _ := app.AppendFloatHistogram(t, fh); !ok { + panic("unexpected error while float appending histogram") + } + } default: return errChunksIterator{err: fmt.Errorf("unknown sample type %s", typ.String())} } @@ -350,6 +380,16 @@ func (s *seriesToChunkEncoder) Iterator(it chunks.Iterator) chunks.Iterator { return errChunksIterator{err: err} } + chks = appendChunk(chks, mint, maxt, chk) + + if existing { + lcsi.Reset(chks...) + return lcsi + } + return NewListChunkSeriesIterator(chks...) +} + +func appendChunk(chks []chunks.Meta, mint, maxt int64, chk chunkenc.Chunk) []chunks.Meta { if chk != nil { chks = append(chks, chunks.Meta{ MinTime: mint, @@ -357,12 +397,7 @@ func (s *seriesToChunkEncoder) Iterator(it chunks.Iterator) chunks.Iterator { Chunk: chk, }) } - - if existing { - lcsi.Reset(chks...) - return lcsi - } - return NewListChunkSeriesIterator(chks...) + return chks } type errChunksIterator struct { @@ -420,3 +455,126 @@ func ExpandChunks(iter chunks.Iterator) ([]chunks.Meta, error) { } return result, iter.Err() } + +// RecodingAppender is a tsdb.Appender that recodes histogram samples if needed during appends. +// It takes an existing appender and a chunk to which samples are appended. +type RecodingAppender struct { + chk *chunkenc.Chunk + app chunkenc.Appender +} + +func NewRecodingAppender(chk *chunkenc.Chunk, app chunkenc.Appender) *RecodingAppender { + return &RecodingAppender{ + chk: chk, + app: app, + } +} + +// Append appends a float sample to the appender. +func (a *RecodingAppender) Append(t int64, v float64) { + a.app.Append(t, v) +} + +// AppendHistogram appends a histogram sample to the underlying chunk. +// The method returns false if the sample cannot be appended and a boolean value set to true +// when it is not appendable because of a counter reset. +// If counterReset is true, okToAppend is always false. +func (a *RecodingAppender) AppendHistogram(t int64, h *histogram.Histogram) (okToAppend, counterReset bool) { + app, ok := a.app.(*chunkenc.HistogramAppender) + if !ok { + return false, false + } + + if app.NumSamples() == 0 { + a.app.AppendHistogram(t, h) + return true, false + } + + var ( + pForwardInserts, nForwardInserts []chunkenc.Insert + pBackwardInserts, nBackwardInserts []chunkenc.Insert + pMergedSpans, nMergedSpans []histogram.Span + ) + switch h.CounterResetHint { + case histogram.GaugeType: + pForwardInserts, nForwardInserts, + pBackwardInserts, nBackwardInserts, + pMergedSpans, nMergedSpans, + okToAppend = app.AppendableGauge(h) + default: + pForwardInserts, nForwardInserts, okToAppend, counterReset = app.Appendable(h) + } + if !okToAppend || counterReset { + return false, counterReset + } + + if len(pBackwardInserts)+len(nBackwardInserts) > 0 { + h.PositiveSpans = pMergedSpans + h.NegativeSpans = nMergedSpans + app.RecodeHistogram(h, pBackwardInserts, nBackwardInserts) + } + if len(pForwardInserts) > 0 || len(nForwardInserts) > 0 { + chk, app := app.Recode( + pForwardInserts, nForwardInserts, + h.PositiveSpans, h.NegativeSpans, + ) + *a.chk = chk + a.app = app + } + + a.app.AppendHistogram(t, h) + return true, counterReset +} + +// AppendFloatHistogram appends a float histogram sample to the underlying chunk. +// The method returns false if the sample cannot be appended and a boolean value set to true +// when it is not appendable because of a counter reset. +// If counterReset is true, okToAppend is always false. +func (a *RecodingAppender) AppendFloatHistogram(t int64, fh *histogram.FloatHistogram) (okToAppend, counterReset bool) { + app, ok := a.app.(*chunkenc.FloatHistogramAppender) + if !ok { + return false, false + } + + if app.NumSamples() == 0 { + a.app.AppendFloatHistogram(t, fh) + return true, false + } + + var ( + pForwardInserts, nForwardInserts []chunkenc.Insert + pBackwardInserts, nBackwardInserts []chunkenc.Insert + pMergedSpans, nMergedSpans []histogram.Span + ) + switch fh.CounterResetHint { + case histogram.GaugeType: + pForwardInserts, nForwardInserts, + pBackwardInserts, nBackwardInserts, + pMergedSpans, nMergedSpans, + okToAppend = app.AppendableGauge(fh) + default: + pForwardInserts, nForwardInserts, okToAppend, counterReset = app.Appendable(fh) + } + + if !okToAppend || counterReset { + return false, counterReset + } + + if len(pBackwardInserts)+len(nBackwardInserts) > 0 { + fh.PositiveSpans = pMergedSpans + fh.NegativeSpans = nMergedSpans + app.RecodeHistogramm(fh, pBackwardInserts, nBackwardInserts) + } + + if len(pForwardInserts) > 0 || len(nForwardInserts) > 0 { + chunk, app := app.Recode( + pForwardInserts, nForwardInserts, + fh.PositiveSpans, fh.NegativeSpans, + ) + *a.chk = chunk + a.app = app + } + + a.app.AppendFloatHistogram(t, fh) + return true, counterReset +} diff --git a/storage/series_test.go b/storage/series_test.go index 210a68e283..4c318f1a0e 100644 --- a/storage/series_test.go +++ b/storage/series_test.go @@ -14,12 +14,17 @@ package storage import ( + "fmt" + "math" "testing" "github.com/stretchr/testify/require" + "github.com/prometheus/prometheus/model/histogram" "github.com/prometheus/prometheus/model/labels" + "github.com/prometheus/prometheus/model/value" "github.com/prometheus/prometheus/tsdb/chunkenc" + "github.com/prometheus/prometheus/tsdb/chunks" "github.com/prometheus/prometheus/tsdb/tsdbutil" ) @@ -119,3 +124,284 @@ func TestChunkSeriesSetToSeriesSet(t *testing.T) { } } } + +type histogramTest struct { + samples []tsdbutil.Sample + expectedChunks int + expectedCounterReset bool +} + +func TestHistogramSeriesToChunks(t *testing.T) { + h1 := &histogram.Histogram{ + Count: 3, + ZeroCount: 2, + ZeroThreshold: 0.001, + Sum: 100, + Schema: 0, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 2}, + }, + PositiveBuckets: []int64{2, 1}, // Abs: 2, 3 + } + // Appendable to h1. + h2 := &histogram.Histogram{ + Count: 12, + ZeroCount: 2, + ZeroThreshold: 0.001, + Sum: 100, + Schema: 0, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 2}, + {Offset: 1, Length: 2}, + }, + PositiveBuckets: []int64{2, 1, -2, 3}, // Abs: 2, 3, 1, 4 + } + // Implicit counter reset by reduction in buckets, not appendable. + h2down := &histogram.Histogram{ + Count: 8, + ZeroCount: 2, + ZeroThreshold: 0.001, + Sum: 100, + Schema: 0, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 2}, + {Offset: 1, Length: 2}, + }, + PositiveBuckets: []int64{1, 1, -1, 3}, // Abs: 1, 2, 1, 4 + } + + fh1 := &histogram.FloatHistogram{ + Count: 4, + ZeroCount: 2, + ZeroThreshold: 0.001, + Sum: 100, + Schema: 0, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 2}, + }, + PositiveBuckets: []float64{3, 1}, + } + // Appendable to fh1. + fh2 := &histogram.FloatHistogram{ + Count: 15, + ZeroCount: 2, + ZeroThreshold: 0.001, + Sum: 100, + Schema: 0, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 2}, + {Offset: 1, Length: 2}, + }, + PositiveBuckets: []float64{4, 2, 7, 2}, + } + // Implicit counter reset by reduction in buckets, not appendable. + fh2down := &histogram.FloatHistogram{ + Count: 13, + ZeroCount: 2, + ZeroThreshold: 0.001, + Sum: 100, + Schema: 0, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 2}, + {Offset: 1, Length: 2}, + }, + PositiveBuckets: []float64{2, 2, 7, 2}, + } + + staleHistogram := &histogram.Histogram{ + Sum: math.Float64frombits(value.StaleNaN), + } + staleFloatHistogram := &histogram.FloatHistogram{ + Sum: math.Float64frombits(value.StaleNaN), + } + + tests := map[string]histogramTest{ + "single histogram to single chunk": { + samples: []tsdbutil.Sample{ + hSample{t: 1, h: h1}, + }, + expectedChunks: 1, + }, + "two histograms encoded to a single chunk": { + samples: []tsdbutil.Sample{ + hSample{t: 1, h: h1}, + hSample{t: 2, h: h2}, + }, + expectedChunks: 1, + }, + "two histograms encoded to two chunks": { + samples: []tsdbutil.Sample{ + hSample{t: 1, h: h2}, + hSample{t: 2, h: h1}, + }, + expectedChunks: 2, + expectedCounterReset: true, + }, + "histogram and stale sample encoded to two chunks": { + samples: []tsdbutil.Sample{ + hSample{t: 1, h: staleHistogram}, + hSample{t: 2, h: h1}, + }, + expectedChunks: 2, + }, + "histogram and reduction in bucket encoded to two chunks": { + samples: []tsdbutil.Sample{ + hSample{t: 1, h: h1}, + hSample{t: 2, h: h2down}, + }, + expectedChunks: 2, + expectedCounterReset: true, + }, + // Float histograms. + "single float histogram to single chunk": { + samples: []tsdbutil.Sample{ + fhSample{t: 1, fh: fh1}, + }, + expectedChunks: 1, + }, + "two float histograms encoded to a single chunk": { + samples: []tsdbutil.Sample{ + fhSample{t: 1, fh: fh1}, + fhSample{t: 2, fh: fh2}, + }, + expectedChunks: 1, + }, + "two float histograms encoded to two chunks": { + samples: []tsdbutil.Sample{ + fhSample{t: 1, fh: fh2}, + fhSample{t: 2, fh: fh1}, + }, + expectedChunks: 2, + expectedCounterReset: true, + }, + "float histogram and stale sample encoded to two chunks": { + samples: []tsdbutil.Sample{ + fhSample{t: 1, fh: staleFloatHistogram}, + fhSample{t: 2, fh: fh1}, + }, + expectedChunks: 2, + }, + "float histogram and reduction in bucket encoded to two chunks": { + samples: []tsdbutil.Sample{ + fhSample{t: 1, fh: fh1}, + fhSample{t: 2, fh: fh2down}, + }, + expectedChunks: 2, + expectedCounterReset: true, + }, + // Mixed. + "histogram and float histogram encoded to two chunks": { + samples: []tsdbutil.Sample{ + hSample{t: 1, h: h1}, + fhSample{t: 2, fh: fh2}, + }, + expectedChunks: 2, + }, + "float histogram and histogram encoded to two chunks": { + samples: []tsdbutil.Sample{ + fhSample{t: 1, fh: fh1}, + hSample{t: 2, h: h2}, + }, + expectedChunks: 2, + }, + "histogram and stale float histogram encoded to two chunks": { + samples: []tsdbutil.Sample{ + hSample{t: 1, h: h1}, + fhSample{t: 2, fh: staleFloatHistogram}, + }, + expectedChunks: 2, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + testHistogramsSeriesToChunks(t, test) + }) + } +} + +func testHistogramsSeriesToChunks(t *testing.T, test histogramTest) { + lbs := labels.FromStrings("__name__", "up", "instance", "localhost:8080") + series := NewListSeries(lbs, test.samples) + encoder := NewSeriesToChunkEncoder(series) + require.EqualValues(t, lbs, encoder.Labels()) + + chks, err := ExpandChunks(encoder.Iterator(nil)) + require.NoError(t, err) + require.Equal(t, test.expectedChunks, len(chks)) + + // Decode all encoded samples and assert they are equal to the original ones. + encodedSamples := expandHistogramSamples(chks) + require.Equal(t, len(test.samples), len(encodedSamples)) + + for i, s := range test.samples { + switch expectedSample := s.(type) { + case hSample: + encodedSample, ok := encodedSamples[i].(hSample) + require.True(t, ok, "expect histogram", fmt.Sprintf("at idx %d", i)) + // Ignore counter reset here, will check on chunk level. + encodedSample.h.CounterResetHint = histogram.UnknownCounterReset + if value.IsStaleNaN(expectedSample.h.Sum) { + require.True(t, value.IsStaleNaN(encodedSample.h.Sum), fmt.Sprintf("at idx %d", i)) + continue + } + require.Equal(t, *expectedSample.h, *encodedSample.h.Compact(0), fmt.Sprintf("at idx %d", i)) + case fhSample: + encodedSample, ok := encodedSamples[i].(fhSample) + require.True(t, ok, "expect float histogram", fmt.Sprintf("at idx %d", i)) + // Ignore counter reset here, will check on chunk level. + encodedSample.fh.CounterResetHint = histogram.UnknownCounterReset + if value.IsStaleNaN(expectedSample.fh.Sum) { + require.True(t, value.IsStaleNaN(encodedSample.fh.Sum), fmt.Sprintf("at idx %d", i)) + continue + } + require.Equal(t, *expectedSample.fh, *encodedSample.fh.Compact(0), fmt.Sprintf("at idx %d", i)) + default: + t.Error("internal error, unexpected type") + } + } + + // If a counter reset hint is expected, it can only be found in the second chunk. + // Otherwise, we assert an unknown counter reset hint in all chunks. + if test.expectedCounterReset { + require.Equal(t, chunkenc.UnknownCounterReset, getCounterResetHint(chks[0])) + require.Equal(t, chunkenc.CounterReset, getCounterResetHint(chks[1])) + } else { + for _, chk := range chks { + require.Equal(t, chunkenc.UnknownCounterReset, getCounterResetHint(chk)) + } + } +} + +func expandHistogramSamples(chunks []chunks.Meta) (result []tsdbutil.Sample) { + if len(chunks) == 0 { + return + } + + for _, chunk := range chunks { + it := chunk.Chunk.Iterator(nil) + for vt := it.Next(); vt != chunkenc.ValNone; vt = it.Next() { + switch vt { + case chunkenc.ValHistogram: + t, h := it.AtHistogram() + result = append(result, hSample{t: t, h: h}) + case chunkenc.ValFloatHistogram: + t, fh := it.AtFloatHistogram() + result = append(result, fhSample{t: t, fh: fh}) + default: + panic("unexpected value type") + } + } + } + return +} + +func getCounterResetHint(chunk chunks.Meta) chunkenc.CounterResetHeader { + switch chk := chunk.Chunk.(type) { + case *chunkenc.HistogramChunk: + return chk.GetCounterResetHeader() + case *chunkenc.FloatHistogramChunk: + return chk.GetCounterResetHeader() + } + return chunkenc.UnknownCounterReset +} From 0ab95536115adfe50af249d36d73674be694ca3f Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Mon, 1 May 2023 16:43:15 +0100 Subject: [PATCH 133/231] tsdb: drop deleted series from the WAL sooner (#12297) `head.deleted` holds the WAL segment in use at the time each series was removed from the head. At the end of `truncateWAL()` we will delete all segments up to `last`, so we can drop any series that were last seen in a segment at or before that point. (same change in Prometheus Agent too) Signed-off-by: Bryan Boreham --- tsdb/agent/db.go | 4 ++-- tsdb/head.go | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/tsdb/agent/db.go b/tsdb/agent/db.go index 3343ee18ef..13cad6bfca 100644 --- a/tsdb/agent/db.go +++ b/tsdb/agent/db.go @@ -665,7 +665,7 @@ func (db *DB) truncate(mint int64) error { } seg, ok := db.deleted[id] - return ok && seg >= first + return ok && seg > last } db.metrics.checkpointCreationTotal.Inc() @@ -687,7 +687,7 @@ func (db *DB) truncate(mint int64) error { // The checkpoint is written and segments before it are truncated, so we // no longer need to track deleted series that were being kept around. for ref, segment := range db.deleted { - if segment < first { + if segment <= last { delete(db.deleted, ref) } } diff --git a/tsdb/head.go b/tsdb/head.go index 5bd5bbccc9..f839adb728 100644 --- a/tsdb/head.go +++ b/tsdb/head.go @@ -1212,9 +1212,9 @@ func (h *Head) truncateWAL(mint int64) error { return true } h.deletedMtx.Lock() - _, ok := h.deleted[id] + keepUntil, ok := h.deleted[id] h.deletedMtx.Unlock() - return ok + return ok && keepUntil > last } h.metrics.checkpointCreationTotal.Inc() if _, err = wlog.Checkpoint(h.logger, h.wal, first, last, keep, mint); err != nil { @@ -1235,7 +1235,7 @@ func (h *Head) truncateWAL(mint int64) error { // longer need to track deleted series that are before it. h.deletedMtx.Lock() for ref, segment := range h.deleted { - if segment < first { + if segment <= last { delete(h.deleted, ref) } } From 1068be199149c8c1ef3bcef66cfdf6c3bdcf1fdc Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 1 May 2023 23:57:09 +0000 Subject: [PATCH 134/231] build(deps): bump bufbuild/buf-setup-action from 1.16.0 to 1.17.0 Bumps [bufbuild/buf-setup-action](https://github.com/bufbuild/buf-setup-action) from 1.16.0 to 1.17.0. - [Release notes](https://github.com/bufbuild/buf-setup-action/releases) - [Commits](https://github.com/bufbuild/buf-setup-action/compare/v1.16.0...v1.17.0) --- updated-dependencies: - dependency-name: bufbuild/buf-setup-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- .github/workflows/buf-lint.yml | 2 +- .github/workflows/buf.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/buf-lint.yml b/.github/workflows/buf-lint.yml index 3275d08fc3..79430ee56a 100644 --- a/.github/workflows/buf-lint.yml +++ b/.github/workflows/buf-lint.yml @@ -10,7 +10,7 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - - uses: bufbuild/buf-setup-action@v1.16.0 + - uses: bufbuild/buf-setup-action@v1.17.0 with: github_token: ${{ secrets.GITHUB_TOKEN }} - uses: bufbuild/buf-lint-action@v1 diff --git a/.github/workflows/buf.yml b/.github/workflows/buf.yml index c80183b18a..06e53172e8 100644 --- a/.github/workflows/buf.yml +++ b/.github/workflows/buf.yml @@ -9,7 +9,7 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - - uses: bufbuild/buf-setup-action@v1.16.0 + - uses: bufbuild/buf-setup-action@v1.17.0 with: github_token: ${{ secrets.GITHUB_TOKEN }} - uses: bufbuild/buf-lint-action@v1 From 1cbe1d85666285ffd84cf776d52e00f2ac73c59c Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 1 May 2023 23:58:14 +0000 Subject: [PATCH 135/231] build(deps): bump github.com/digitalocean/godo from 1.98.0 to 1.99.0 Bumps [github.com/digitalocean/godo](https://github.com/digitalocean/godo) from 1.98.0 to 1.99.0. - [Release notes](https://github.com/digitalocean/godo/releases) - [Changelog](https://github.com/digitalocean/godo/blob/main/CHANGELOG.md) - [Commits](https://github.com/digitalocean/godo/compare/v1.98.0...v1.99.0) --- updated-dependencies: - dependency-name: github.com/digitalocean/godo dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 7b309b03f5..f4f22239d4 100644 --- a/go.mod +++ b/go.mod @@ -11,7 +11,7 @@ require ( github.com/aws/aws-sdk-go v1.44.245 github.com/cespare/xxhash/v2 v2.2.0 github.com/dennwc/varint v1.0.0 - github.com/digitalocean/godo v1.98.0 + github.com/digitalocean/godo v1.99.0 github.com/docker/docker v23.0.4+incompatible github.com/edsrzf/mmap-go v1.1.0 github.com/envoyproxy/go-control-plane v0.11.0 diff --git a/go.sum b/go.sum index c699aa9dff..753cc2ea14 100644 --- a/go.sum +++ b/go.sum @@ -149,8 +149,8 @@ github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs github.com/dennwc/varint v1.0.0 h1:kGNFFSSw8ToIy3obO/kKr8U9GZYUAxQEVuix4zfDWzE= github.com/dennwc/varint v1.0.0/go.mod h1:hnItb35rvZvJrbTALZtY/iQfDs48JKRG1RPpgziApxA= github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ= -github.com/digitalocean/godo v1.98.0 h1:potyC1eD0N9n5/P4/WmJuKgg+OGYZOBWEW+/aKTX6QQ= -github.com/digitalocean/godo v1.98.0/go.mod h1:NRpFznZFvhHjBoqZAaOD3khVzsJ3EibzKqFL4R60dmA= +github.com/digitalocean/godo v1.99.0 h1:gUHO7n9bDaZFWvbzOum4bXE0/09ZuYA9yA8idQHX57E= +github.com/digitalocean/godo v1.99.0/go.mod h1:SsS2oXo2rznfM/nORlZ/6JaUJZFhmKTib1YhopUc8NA= github.com/dnaeon/go-vcr v1.2.0 h1:zHCHvJYTMh1N7xnV7zf1m1GPBF9Ad0Jk/whtQ1663qI= github.com/docker/distribution v2.8.1+incompatible h1:Q50tZOPR6T/hjNsyc9g8/syEs6bk8XXApsHjKukMl68= github.com/docker/distribution v2.8.1+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w= From 6dfcd002ae3765be1099e1eecc1c98b15aa69acb Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 1 May 2023 23:58:22 +0000 Subject: [PATCH 136/231] build(deps): bump github.com/hetznercloud/hcloud-go Bumps [github.com/hetznercloud/hcloud-go](https://github.com/hetznercloud/hcloud-go) from 1.42.0 to 1.43.0. - [Release notes](https://github.com/hetznercloud/hcloud-go/releases) - [Changelog](https://github.com/hetznercloud/hcloud-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/hetznercloud/hcloud-go/compare/v1.42.0...v1.43.0) --- updated-dependencies: - dependency-name: github.com/hetznercloud/hcloud-go dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 7b309b03f5..240e81afd2 100644 --- a/go.mod +++ b/go.mod @@ -29,7 +29,7 @@ require ( github.com/grpc-ecosystem/grpc-gateway v1.16.0 github.com/hashicorp/consul/api v1.20.0 github.com/hashicorp/nomad/api v0.0.0-20230418003350-3067191c5197 - github.com/hetznercloud/hcloud-go v1.42.0 + github.com/hetznercloud/hcloud-go v1.43.0 github.com/ionos-cloud/sdk-go/v6 v6.1.6 github.com/json-iterator/go v1.1.12 github.com/kolo/xmlrpc v0.0.0-20220921171641-a4b6fa1dd06b diff --git a/go.sum b/go.sum index c699aa9dff..b44b5f4cba 100644 --- a/go.sum +++ b/go.sum @@ -456,8 +456,8 @@ github.com/hashicorp/nomad/api v0.0.0-20230418003350-3067191c5197/go.mod h1:2TCr github.com/hashicorp/serf v0.8.2/go.mod h1:6hOLApaqBFA1NXqRQAsxw9QxuDEvNxSQRwA/JwenrHc= github.com/hashicorp/serf v0.10.1 h1:Z1H2J60yRKvfDYAOZLd2MU0ND4AH/WDz7xYHDWQsIPY= github.com/hashicorp/serf v0.10.1/go.mod h1:yL2t6BqATOLGc5HF7qbFkTfXoPIY0WZdWHfEvMqbG+4= -github.com/hetznercloud/hcloud-go v1.42.0 h1:Es/CDOForQN3nOOP5Vxh1N/YHjpCg386iYEX5zCgi+A= -github.com/hetznercloud/hcloud-go v1.42.0/go.mod h1:YADL8AbmQYH0Eo+1lkuyoc8LutT0UeMvaKP47nNUb+Y= +github.com/hetznercloud/hcloud-go v1.43.0 h1:m4p5+mz32Tt+bHkNQEg9RQdtMIu+SUdMjs29LsOJjUk= +github.com/hetznercloud/hcloud-go v1.43.0/go.mod h1:DPs7Dvae8LrTVOWyq2abwQQOwfpfICAzKHm2HQMU5/E= github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= github.com/hudl/fargo v1.3.0/go.mod h1:y3CKSmjA+wD2gak7sUSXTAoopbhU08POFhmITJgmKTg= github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc= From 8deface2da9c234e69de99e4d76446b4ee09acea Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 2 May 2023 00:00:22 +0000 Subject: [PATCH 137/231] build(deps): bump github.com/prometheus/client_golang Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.14.0 to 1.15.0. - [Release notes](https://github.com/prometheus/client_golang/releases) - [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md) - [Commits](https://github.com/prometheus/client_golang/compare/v1.14.0...v1.15.0) --- updated-dependencies: - dependency-name: github.com/prometheus/client_golang dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- documentation/examples/remote_storage/go.mod | 6 +++--- documentation/examples/remote_storage/go.sum | 14 +++++--------- 2 files changed, 8 insertions(+), 12 deletions(-) diff --git a/documentation/examples/remote_storage/go.mod b/documentation/examples/remote_storage/go.mod index 5e0e6ddb13..1db3e9a9e4 100644 --- a/documentation/examples/remote_storage/go.mod +++ b/documentation/examples/remote_storage/go.mod @@ -8,7 +8,7 @@ require ( github.com/gogo/protobuf v1.3.2 github.com/golang/snappy v0.0.4 github.com/influxdata/influxdb v1.11.0 - github.com/prometheus/client_golang v1.14.0 + github.com/prometheus/client_golang v1.15.0 github.com/prometheus/common v0.42.0 github.com/prometheus/prometheus v0.43.0 github.com/stretchr/testify v1.8.2 @@ -29,7 +29,7 @@ require ( github.com/grafana/regexp v0.0.0-20221122212121-6b5c0a4cb7fd // indirect github.com/jmespath/go-jmespath v0.4.0 // indirect github.com/jpillora/backoff v1.0.0 // indirect - github.com/kr/pretty v0.3.0 // indirect + github.com/kr/text v0.2.0 // indirect github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f // indirect github.com/pkg/errors v0.9.1 // indirect @@ -51,7 +51,7 @@ require ( golang.org/x/text v0.8.0 // indirect golang.org/x/time v0.3.0 // indirect google.golang.org/appengine v1.6.7 // indirect - google.golang.org/protobuf v1.29.0 // indirect + google.golang.org/protobuf v1.30.0 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect ) diff --git a/documentation/examples/remote_storage/go.sum b/documentation/examples/remote_storage/go.sum index 11c2945151..c4350e78b0 100644 --- a/documentation/examples/remote_storage/go.sum +++ b/documentation/examples/remote_storage/go.sum @@ -141,8 +141,7 @@ github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxv github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= -github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0= -github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= @@ -181,8 +180,8 @@ github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXP github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo= github.com/prometheus/client_golang v1.7.1/go.mod h1:PY5Wy2awLA44sXw4AOSfFBetzPP4j5+D6mVACh+pe2M= github.com/prometheus/client_golang v1.11.0/go.mod h1:Z6t4BnS23TR94PD6BsDNk8yVqroYurpAkEiz0P2BEV0= -github.com/prometheus/client_golang v1.14.0 h1:nJdhIvne2eSX/XRAFV9PcvFFRbrjbcTUj0VP62TMhnw= -github.com/prometheus/client_golang v1.14.0/go.mod h1:8vpkKitgIVNcqrRBWh1C4TIUQgYNtG/XQE4E/Zae36Y= +github.com/prometheus/client_golang v1.15.0 h1:5fCgGYogn0hFdhyhLbw7hEsWxufKtY9klyvdNfFlFhM= +github.com/prometheus/client_golang v1.15.0/go.mod h1:e9yaBhRPU2pPNsZwE+JdQl0KEt1N9XgF6zxWmaC0xOk= github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo= github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= github.com/prometheus/client_model v0.2.0/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= @@ -204,7 +203,6 @@ github.com/prometheus/procfs v0.9.0 h1:wzCHvIvM5SxWqYvwgVL7yJY8Lz3PKn49KQtpgMYJf github.com/prometheus/procfs v0.9.0/go.mod h1:+pB4zwohETzFnmlpe6yd2lSc+0/46IYZRB/chUwxUZY= github.com/prometheus/prometheus v0.43.0 h1:18iCSfrbAHbXvYFvR38U1Pt4uZmU9SmDcCpCrBKUiGg= github.com/prometheus/prometheus v0.43.0/go.mod h1:2BA14LgBeqlPuzObSEbh+Y+JwLH2GcqDlJKbF2sA6FM= -github.com/rogpeppe/go-internal v1.6.1/go.mod h1:xXDCJY+GAPziupqXw64V24skbSoqbTEfhy4qGm1nDQc= github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8= github.com/scaleway/scaleway-sdk-go v1.0.0-beta.14 h1:yFl3jyaSVLNYXlnNYM5z2pagEk1dYQhfr1p20T1NyKY= github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo= @@ -341,14 +339,12 @@ google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzi google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= -google.golang.org/protobuf v1.29.0 h1:44S3JjaKmLEE4YIkjzexaP+NzZsudE3Zin5Njn/pYX0= -google.golang.org/protobuf v1.29.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= +google.golang.org/protobuf v1.30.0 h1:kPPoIgf3TsEvrm0PFe15JQ+570QVxYzEvvHqChK+cng= +google.golang.org/protobuf v1.30.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= -gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc= gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA= gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= From 6985dcbe73561081e2e47541db99dfa1f8bd8831 Mon Sep 17 00:00:00 2001 From: Justin Lei Date: Mon, 10 Apr 2023 13:07:43 -0700 Subject: [PATCH 138/231] Optimize and test MemoizedSeriesIterator Signed-off-by: Justin Lei --- promql/engine.go | 2 +- storage/memoized_iterator.go | 26 +++++----- storage/memoized_iterator_test.go | 82 +++++++++++++++++++++++-------- 3 files changed, 75 insertions(+), 35 deletions(-) diff --git a/promql/engine.go b/promql/engine.go index cbeeb82a1a..688048e7a6 100644 --- a/promql/engine.go +++ b/promql/engine.go @@ -1857,7 +1857,7 @@ func (ev *evaluator) vectorSelectorSingle(it *storage.MemoizedSeriesIterator, no } if valueType == chunkenc.ValNone || t > refTime { var ok bool - t, v, _, h, ok = it.PeekPrev() + t, v, h, ok = it.PeekPrev() if !ok || t < refTime-durationMilliseconds(ev.lookbackDelta) { return 0, 0, nil, false } diff --git a/storage/memoized_iterator.go b/storage/memoized_iterator.go index 88eee0756b..465afa91cd 100644 --- a/storage/memoized_iterator.go +++ b/storage/memoized_iterator.go @@ -31,12 +31,7 @@ type MemoizedSeriesIterator struct { // Keep track of the previously returned value. prevTime int64 prevValue float64 - prevHistogram *histogram.Histogram prevFloatHistogram *histogram.FloatHistogram - // TODO(beorn7): MemoizedSeriesIterator is currently only used by the - // PromQL engine, which only works with FloatHistograms. For better - // performance, we could change MemoizedSeriesIterator to also only - // handle FloatHistograms. } // NewMemoizedEmptyIterator is like NewMemoizedIterator but it's initialised with an empty iterator. @@ -66,11 +61,11 @@ func (b *MemoizedSeriesIterator) Reset(it chunkenc.Iterator) { // PeekPrev returns the previous element of the iterator. If there is none buffered, // ok is false. -func (b *MemoizedSeriesIterator) PeekPrev() (t int64, v float64, h *histogram.Histogram, fh *histogram.FloatHistogram, ok bool) { +func (b *MemoizedSeriesIterator) PeekPrev() (t int64, v float64, fh *histogram.FloatHistogram, ok bool) { if b.prevTime == math.MinInt64 { - return 0, 0, nil, nil, false + return 0, 0, nil, false } - return b.prevTime, b.prevValue, b.prevHistogram, b.prevFloatHistogram, true + return b.prevTime, b.prevValue, b.prevFloatHistogram, true } // Seek advances the iterator to the element at time t or greater. @@ -108,15 +103,14 @@ func (b *MemoizedSeriesIterator) Next() chunkenc.ValueType { return chunkenc.ValNone case chunkenc.ValFloat: b.prevTime, b.prevValue = b.it.At() - b.prevHistogram = nil b.prevFloatHistogram = nil case chunkenc.ValHistogram: b.prevValue = 0 - b.prevTime, b.prevHistogram = b.it.AtHistogram() - _, b.prevFloatHistogram = b.it.AtFloatHistogram() + ts, h := b.it.AtHistogram() + b.prevTime = ts + b.prevFloatHistogram = h.ToFloat() case chunkenc.ValFloatHistogram: b.prevValue = 0 - b.prevHistogram = nil b.prevTime, b.prevFloatHistogram = b.it.AtFloatHistogram() } @@ -132,13 +126,17 @@ func (b *MemoizedSeriesIterator) At() (int64, float64) { return b.it.At() } -// AtHistogram returns the current histogram element of the iterator. +// AtHistogram is not supported by this iterator. func (b *MemoizedSeriesIterator) AtHistogram() (int64, *histogram.Histogram) { - return b.it.AtHistogram() + panic("AtHistogram is not supported by this iterator.") } // AtFloatHistogram returns the current float-histogram element of the iterator. func (b *MemoizedSeriesIterator) AtFloatHistogram() (int64, *histogram.FloatHistogram) { + if b.valueType == chunkenc.ValHistogram { + ts, h := b.it.AtHistogram() + return ts, h.ToFloat() + } return b.it.AtFloatHistogram() } diff --git a/storage/memoized_iterator_test.go b/storage/memoized_iterator_test.go index d996436e00..f922aaeece 100644 --- a/storage/memoized_iterator_test.go +++ b/storage/memoized_iterator_test.go @@ -16,25 +16,36 @@ package storage import ( "testing" - "github.com/stretchr/testify/require" - + "github.com/prometheus/prometheus/model/histogram" "github.com/prometheus/prometheus/tsdb/chunkenc" + "github.com/prometheus/prometheus/tsdb/tsdbutil" + + "github.com/stretchr/testify/require" ) func TestMemoizedSeriesIterator(t *testing.T) { - // TODO(beorn7): Include histograms in testing. var it *MemoizedSeriesIterator - sampleEq := func(ets int64, ev float64) { - ts, v := it.At() - require.Equal(t, ets, ts, "timestamp mismatch") - require.Equal(t, ev, v, "value mismatch") + sampleEq := func(ets int64, ev float64, efh *histogram.FloatHistogram) { + if efh == nil { + ts, v := it.At() + require.Equal(t, ets, ts, "timestamp mismatch") + require.Equal(t, ev, v, "value mismatch") + } else { + ts, fh := it.AtFloatHistogram() + require.Equal(t, ets, ts, "timestamp mismatch") + require.Equal(t, efh, fh, "histogram mismatch") + } } - prevSampleEq := func(ets int64, ev float64, eok bool) { - ts, v, _, _, ok := it.PeekPrev() + prevSampleEq := func(ets int64, ev float64, efh *histogram.FloatHistogram, eok bool) { + ts, v, fh, ok := it.PeekPrev() require.Equal(t, eok, ok, "exist mismatch") require.Equal(t, ets, ts, "timestamp mismatch") - require.Equal(t, ev, v, "value mismatch") + if efh == nil { + require.Equal(t, ev, v, "value mismatch") + } else { + require.Equal(t, efh, fh, "histogram mismatch") + } } it = NewMemoizedIterator(NewListSeriesIterator(samples{ @@ -46,29 +57,60 @@ func TestMemoizedSeriesIterator(t *testing.T) { fSample{t: 99, f: 8}, fSample{t: 100, f: 9}, fSample{t: 101, f: 10}, + hSample{t: 102, h: tsdbutil.GenerateTestHistogram(0)}, + hSample{t: 103, h: tsdbutil.GenerateTestHistogram(1)}, + fhSample{t: 104, fh: tsdbutil.GenerateTestFloatHistogram(2)}, + fhSample{t: 199, fh: tsdbutil.GenerateTestFloatHistogram(3)}, + hSample{t: 200, h: tsdbutil.GenerateTestHistogram(4)}, + fhSample{t: 299, fh: tsdbutil.GenerateTestFloatHistogram(5)}, + fSample{t: 300, f: 11}, }), 2) require.Equal(t, it.Seek(-123), chunkenc.ValFloat, "seek failed") - sampleEq(1, 2) - prevSampleEq(0, 0, false) + sampleEq(1, 2, nil) + prevSampleEq(0, 0, nil, false) require.Equal(t, it.Next(), chunkenc.ValFloat, "next failed") - sampleEq(2, 3) - prevSampleEq(1, 2, true) + sampleEq(2, 3, nil) + prevSampleEq(1, 2, nil, true) require.Equal(t, it.Next(), chunkenc.ValFloat, "next failed") require.Equal(t, it.Next(), chunkenc.ValFloat, "next failed") require.Equal(t, it.Next(), chunkenc.ValFloat, "next failed") - sampleEq(5, 6) - prevSampleEq(4, 5, true) + sampleEq(5, 6, nil) + prevSampleEq(4, 5, nil, true) require.Equal(t, it.Seek(5), chunkenc.ValFloat, "seek failed") - sampleEq(5, 6) - prevSampleEq(4, 5, true) + sampleEq(5, 6, nil) + prevSampleEq(4, 5, nil, true) require.Equal(t, it.Seek(101), chunkenc.ValFloat, "seek failed") - sampleEq(101, 10) - prevSampleEq(100, 9, true) + sampleEq(101, 10, nil) + prevSampleEq(100, 9, nil, true) + + require.Equal(t, chunkenc.ValHistogram, it.Next(), "next failed") + sampleEq(102, 0, tsdbutil.GenerateTestFloatHistogram(0)) + prevSampleEq(101, 10, nil, true) + + require.Equal(t, chunkenc.ValHistogram, it.Next(), "next failed") + sampleEq(103, 0, tsdbutil.GenerateTestFloatHistogram(1)) + prevSampleEq(102, 0, tsdbutil.GenerateTestFloatHistogram(0), true) + + require.Equal(t, chunkenc.ValFloatHistogram, it.Seek(104), "seek failed") + sampleEq(104, 0, tsdbutil.GenerateTestFloatHistogram(2)) + prevSampleEq(103, 0, tsdbutil.GenerateTestFloatHistogram(1), true) + + require.Equal(t, chunkenc.ValFloatHistogram, it.Next(), "next failed") + sampleEq(199, 0, tsdbutil.GenerateTestFloatHistogram(3)) + prevSampleEq(104, 0, tsdbutil.GenerateTestFloatHistogram(2), true) + + require.Equal(t, chunkenc.ValFloatHistogram, it.Seek(280), "seek failed") + sampleEq(299, 0, tsdbutil.GenerateTestFloatHistogram(5)) + prevSampleEq(0, 0, nil, false) + + require.Equal(t, chunkenc.ValFloat, it.Next(), "next failed") + sampleEq(300, 11, nil) + prevSampleEq(299, 0, tsdbutil.GenerateTestFloatHistogram(5), true) require.Equal(t, it.Next(), chunkenc.ValNone, "next succeeded unexpectedly") require.Equal(t, it.Seek(1024), chunkenc.ValNone, "seek succeeded unexpectedly") From 7a48a266b675d7531234c66b54f31e7c71bcd341 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Wed, 3 May 2023 11:59:27 +0100 Subject: [PATCH 139/231] labels: respect Set after Del in Builder (#12322) * labels: respect Set after Del in Builder The implementations are not symmetric between `Set()` and `Del()`, so we must be careful. Add tests for this, both in labels and in relabel where the issue was reported. Also make the slice implementation consistent re `slices.Contains`. Signed-off-by: Bryan Boreham --- model/labels/labels.go | 9 ++++----- model/labels/labels_string.go | 7 ++++--- model/labels/labels_test.go | 7 +++++++ model/relabel/relabel_test.go | 28 ++++++++++++++++++++++++++++ 4 files changed, 43 insertions(+), 8 deletions(-) diff --git a/model/labels/labels.go b/model/labels/labels.go index 93524ddcfc..9ac0e5b53f 100644 --- a/model/labels/labels.go +++ b/model/labels/labels.go @@ -533,16 +533,15 @@ func (b *Builder) Set(n, v string) *Builder { } func (b *Builder) Get(n string) string { - for _, d := range b.del { - if d == n { - return "" - } - } + // Del() removes entries from .add but Set() does not remove from .del, so check .add first. for _, a := range b.add { if a.Name == n { return a.Value } } + if slices.Contains(b.del, n) { + return "" + } return b.base.Get(n) } diff --git a/model/labels/labels_string.go b/model/labels/labels_string.go index ff46103ebc..6d54e98ab9 100644 --- a/model/labels/labels_string.go +++ b/model/labels/labels_string.go @@ -593,14 +593,15 @@ func (b *Builder) Set(n, v string) *Builder { } func (b *Builder) Get(n string) string { - if slices.Contains(b.del, n) { - return "" - } + // Del() removes entries from .add but Set() does not remove from .del, so check .add first. for _, a := range b.add { if a.Name == n { return a.Value } } + if slices.Contains(b.del, n) { + return "" + } return b.base.Get(n) } diff --git a/model/labels/labels_test.go b/model/labels/labels_test.go index 9e60c22516..108d8b0de0 100644 --- a/model/labels/labels_test.go +++ b/model/labels/labels_test.go @@ -607,6 +607,13 @@ func TestBuilder(t *testing.T) { require.Equal(t, tcase.want.BytesWithoutLabels(nil, "aaa", "bbb"), b.Labels().Bytes(nil)) }) } + t.Run("set_after_del", func(t *testing.T) { + b := NewBuilder(FromStrings("aaa", "111")) + b.Del("bbb") + b.Set("bbb", "222") + require.Equal(t, FromStrings("aaa", "111", "bbb", "222"), b.Labels()) + require.Equal(t, "222", b.Get("bbb")) + }) } func TestScratchBuilder(t *testing.T) { diff --git a/model/relabel/relabel_test.go b/model/relabel/relabel_test.go index d277d778d1..b50ff4010a 100644 --- a/model/relabel/relabel_test.go +++ b/model/relabel/relabel_test.go @@ -397,6 +397,34 @@ func TestRelabel(t *testing.T) { "foo": "bar", }), }, + { // From https://github.com/prometheus/prometheus/issues/12283 + input: labels.FromMap(map[string]string{ + "__meta_kubernetes_pod_container_port_name": "foo", + "__meta_kubernetes_pod_annotation_XXX_metrics_port": "9091", + }), + relabel: []*Config{ + { + Regex: MustNewRegexp("^__meta_kubernetes_pod_container_port_name$"), + Action: LabelDrop, + }, + { + SourceLabels: model.LabelNames{"__meta_kubernetes_pod_annotation_XXX_metrics_port"}, + Regex: MustNewRegexp("(.+)"), + Action: Replace, + Replacement: "metrics", + TargetLabel: "__meta_kubernetes_pod_container_port_name", + }, + { + SourceLabels: model.LabelNames{"__meta_kubernetes_pod_container_port_name"}, + Regex: MustNewRegexp("^metrics$"), + Action: Keep, + }, + }, + output: labels.FromMap(map[string]string{ + "__meta_kubernetes_pod_annotation_XXX_metrics_port": "9091", + "__meta_kubernetes_pod_container_port_name": "metrics", + }), + }, { input: labels.FromMap(map[string]string{ "a": "foo", From 3d26faade4069b70b5a5cc3a53b56c7654c38a49 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Wed, 3 May 2023 16:18:28 +0100 Subject: [PATCH 140/231] Create 2.44.0-rc.1 (#12323) Signed-off-by: Bryan Boreham --- CHANGELOG.md | 4 ++++ VERSION | 2 +- web/ui/module/codemirror-promql/package.json | 4 ++-- web/ui/module/lezer-promql/package.json | 2 +- web/ui/package-lock.json | 14 +++++++------- web/ui/react-app/package.json | 4 ++-- 6 files changed, 17 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8fc24345cc..06fb01ad07 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,9 @@ # Changelog +## 2.44.0-rc.1 / 2023-05-03 + +* [BUGFIX] Labels: Set after Del would be ignored, which broke some relabeling rules. #12322 + ## 2.44.0-rc.0 / 2023-04-22 This version is built with Go tag `stringlabels`, to use the smaller data diff --git a/VERSION b/VERSION index a4494d974d..662635da4e 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.44.0-rc.0 +2.44.0-rc.1 diff --git a/web/ui/module/codemirror-promql/package.json b/web/ui/module/codemirror-promql/package.json index 6521c87298..11fdeee455 100644 --- a/web/ui/module/codemirror-promql/package.json +++ b/web/ui/module/codemirror-promql/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/codemirror-promql", - "version": "0.44.0-rc.0", + "version": "0.44.0-rc.1", "description": "a CodeMirror mode for the PromQL language", "types": "dist/esm/index.d.ts", "module": "dist/esm/index.js", @@ -29,7 +29,7 @@ }, "homepage": "https://github.com/prometheus/prometheus/blob/main/web/ui/module/codemirror-promql/README.md", "dependencies": { - "@prometheus-io/lezer-promql": "0.44.0-rc.0", + "@prometheus-io/lezer-promql": "0.44.0-rc.1", "lru-cache": "^6.0.0" }, "devDependencies": { diff --git a/web/ui/module/lezer-promql/package.json b/web/ui/module/lezer-promql/package.json index be92ac46f6..4b3f29b521 100644 --- a/web/ui/module/lezer-promql/package.json +++ b/web/ui/module/lezer-promql/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/lezer-promql", - "version": "0.44.0-rc.0", + "version": "0.44.0-rc.1", "description": "lezer-based PromQL grammar", "main": "dist/index.cjs", "type": "module", diff --git a/web/ui/package-lock.json b/web/ui/package-lock.json index 09a4fd3fef..37804eebf2 100644 --- a/web/ui/package-lock.json +++ b/web/ui/package-lock.json @@ -28,10 +28,10 @@ }, "module/codemirror-promql": { "name": "@prometheus-io/codemirror-promql", - "version": "0.44.0-rc.0", + "version": "0.44.0-rc.1", "license": "Apache-2.0", "dependencies": { - "@prometheus-io/lezer-promql": "0.44.0-rc.0", + "@prometheus-io/lezer-promql": "0.44.0-rc.1", "lru-cache": "^6.0.0" }, "devDependencies": { @@ -61,7 +61,7 @@ }, "module/lezer-promql": { "name": "@prometheus-io/lezer-promql", - "version": "0.44.0-rc.0", + "version": "0.44.0-rc.1", "license": "Apache-2.0", "devDependencies": { "@lezer/generator": "^1.2.2", @@ -20763,7 +20763,7 @@ }, "react-app": { "name": "@prometheus-io/app", - "version": "0.44.0-rc.0", + "version": "0.44.0-rc.1", "dependencies": { "@codemirror/autocomplete": "^6.4.0", "@codemirror/commands": "^6.2.0", @@ -20781,7 +20781,7 @@ "@lezer/lr": "^1.3.1", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.44.0-rc.0", + "@prometheus-io/codemirror-promql": "0.44.0-rc.1", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", "downshift": "^7.2.0", @@ -23417,7 +23417,7 @@ "@lezer/lr": "^1.3.1", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.44.0-rc.0", + "@prometheus-io/codemirror-promql": "0.44.0-rc.1", "@testing-library/react-hooks": "^7.0.2", "@types/enzyme": "^3.10.12", "@types/flot": "0.0.32", @@ -23468,7 +23468,7 @@ "@lezer/common": "^1.0.2", "@lezer/highlight": "^1.1.3", "@lezer/lr": "^1.3.1", - "@prometheus-io/lezer-promql": "0.44.0-rc.0", + "@prometheus-io/lezer-promql": "0.44.0-rc.1", "@types/lru-cache": "^5.1.1", "isomorphic-fetch": "^3.0.0", "lru-cache": "^6.0.0", diff --git a/web/ui/react-app/package.json b/web/ui/react-app/package.json index b4929d664a..57e4d8120d 100644 --- a/web/ui/react-app/package.json +++ b/web/ui/react-app/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/app", - "version": "0.44.0-rc.0", + "version": "0.44.0-rc.1", "private": true, "dependencies": { "@codemirror/autocomplete": "^6.4.0", @@ -19,7 +19,7 @@ "@lezer/common": "^1.0.2", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.44.0-rc.0", + "@prometheus-io/codemirror-promql": "0.44.0-rc.1", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", "downshift": "^7.2.0", From b0272255b735ec5bbd04f0bc26c157febd7678a2 Mon Sep 17 00:00:00 2001 From: beorn7 Date: Wed, 3 May 2023 20:06:12 +0200 Subject: [PATCH 141/231] storage: optimise sampleRing Replace many checks for the lengths of slices with a single tracking variable. Signed-off-by: beorn7 --- storage/buffer.go | 164 ++++++++++++++++++++++++++++------------------ 1 file changed, 99 insertions(+), 65 deletions(-) diff --git a/storage/buffer.go b/storage/buffer.go index 2229e52591..38f5591039 100644 --- a/storage/buffer.go +++ b/storage/buffer.go @@ -242,15 +242,16 @@ func (s fhSample) Type() chunkenc.ValueType { type sampleRing struct { delta int64 - // Lookback buffers. We use buf for mixed samples, but one of the three + // Lookback buffers. We use iBuf for mixed samples, but one of the three // concrete ones for homogenous samples. (Only one of the four bufs is // allowed to be populated!) This avoids the overhead of the interface // wrapper for the happy (and by far most common) case of homogenous // samples. - buf []tsdbutil.Sample - fBuf []fSample - hBuf []hSample - fhBuf []fhSample + iBuf []tsdbutil.Sample + fBuf []fSample + hBuf []hSample + fhBuf []fhSample + bufInUse bufType i int // Position of most recent element in ring buffer. f int // Position of first element in ring buffer. @@ -259,6 +260,16 @@ type sampleRing struct { it sampleRingIterator } +type bufType int + +const ( + noBuf bufType = iota // Nothing yet stored in sampleRing. + iBuf + fBuf + hBuf + fhBuf +) + // newSampleRing creates a new sampleRing. If you do not know the prefereed // value type yet, use a size of 0 (in which case the provided typ doesn't // matter). On the first add, a buffer of size 16 will be allocated with the @@ -278,7 +289,7 @@ func newSampleRing(delta int64, size int, typ chunkenc.ValueType) *sampleRing { case chunkenc.ValFloatHistogram: r.fhBuf = make([]fhSample, size) default: - r.buf = make([]tsdbutil.Sample, size) + r.iBuf = make([]tsdbutil.Sample, size) } return r } @@ -287,6 +298,7 @@ func (r *sampleRing) reset() { r.l = 0 r.i = -1 r.f = 0 + r.bufInUse = noBuf } // Returns the current iterator. Invalidates previously returned iterators. @@ -310,18 +322,18 @@ func (it *sampleRingIterator) Next() chunkenc.ValueType { if it.i >= it.r.l { return chunkenc.ValNone } - switch { - case len(it.r.fBuf) > 0: + switch it.r.bufInUse { + case fBuf: s := it.r.atF(it.i) it.t = s.t it.f = s.f return chunkenc.ValFloat - case len(it.r.hBuf) > 0: + case hBuf: s := it.r.atH(it.i) it.t = s.t it.h = s.h return chunkenc.ValHistogram - case len(it.r.fhBuf) > 0: + case fhBuf: s := it.r.atFH(it.i) it.t = s.t it.fh = s.fh @@ -372,8 +384,8 @@ func (it *sampleRingIterator) AtT() int64 { } func (r *sampleRing) at(i int) tsdbutil.Sample { - j := (r.f + i) % len(r.buf) - return r.buf[j] + j := (r.f + i) % len(r.iBuf) + return r.iBuf[j] } func (r *sampleRing) atF(i int) fSample { @@ -397,91 +409,113 @@ func (r *sampleRing) atFH(i int) fhSample { // from this package (fSample, hSample, fhSample), call one of the specialized // methods addF, addH, or addFH for better performance. func (r *sampleRing) add(s tsdbutil.Sample) { - if len(r.buf) == 0 { + if r.bufInUse == noBuf { + // First sample. + switch s := s.(type) { + case fSample: + r.bufInUse = fBuf + r.fBuf = addF(s, r.fBuf, r) + case hSample: + r.bufInUse = hBuf + r.hBuf = addH(s, r.hBuf, r) + case fhSample: + r.bufInUse = fhBuf + r.fhBuf = addFH(s, r.fhBuf, r) + } + return + } + if r.bufInUse != iBuf { // Nothing added to the interface buf yet. Let's check if we can // stay specialized. switch s := s.(type) { case fSample: - if len(r.hBuf)+len(r.fhBuf) == 0 { + if r.bufInUse == fBuf { r.fBuf = addF(s, r.fBuf, r) return } case hSample: - if len(r.fBuf)+len(r.fhBuf) == 0 { + if r.bufInUse == hBuf { r.hBuf = addH(s, r.hBuf, r) return } case fhSample: - if len(r.fBuf)+len(r.hBuf) == 0 { + if r.bufInUse == fhBuf { r.fhBuf = addFH(s, r.fhBuf, r) return } } // The new sample isn't a fit for the already existing // ones. Copy the latter into the interface buffer where needed. - switch { - case len(r.fBuf) > 0: + switch r.bufInUse { + case fBuf: for _, s := range r.fBuf { - r.buf = append(r.buf, s) + r.iBuf = append(r.iBuf, s) } r.fBuf = nil - case len(r.hBuf) > 0: + case hBuf: for _, s := range r.hBuf { - r.buf = append(r.buf, s) + r.iBuf = append(r.iBuf, s) } r.hBuf = nil - case len(r.fhBuf) > 0: + case fhBuf: for _, s := range r.fhBuf { - r.buf = append(r.buf, s) + r.iBuf = append(r.iBuf, s) } r.fhBuf = nil } + r.bufInUse = iBuf } - r.buf = addSample(s, r.buf, r) + r.iBuf = addSample(s, r.iBuf, r) } // addF is a version of the add method specialized for fSample. func (r *sampleRing) addF(s fSample) { - switch { - case len(r.buf) > 0: - // Already have interface samples. Add to the interface buf. - r.buf = addSample(s, r.buf, r) - case len(r.hBuf)+len(r.fhBuf) > 0: + switch r.bufInUse { + case fBuf: // Add to existing fSamples. + r.fBuf = addF(s, r.fBuf, r) + case noBuf: // Add first sample. + r.fBuf = addF(s, r.fBuf, r) + r.bufInUse = fBuf + case iBuf: // Already have interface samples. Add to the interface buf. + r.iBuf = addSample(s, r.iBuf, r) + default: // Already have specialized samples that are not fSamples. // Need to call the checked add method for conversion. r.add(s) - default: - r.fBuf = addF(s, r.fBuf, r) } } // addH is a version of the add method specialized for hSample. func (r *sampleRing) addH(s hSample) { - switch { - case len(r.buf) > 0: - // Already have interface samples. Add to the interface buf. - r.buf = addSample(s, r.buf, r) - case len(r.fBuf)+len(r.fhBuf) > 0: - // Already have samples that are not hSamples. + switch r.bufInUse { + case hBuf: // Add to existing hSamples. + r.hBuf = addH(s, r.hBuf, r) + case noBuf: // Add first sample. + r.hBuf = addH(s, r.hBuf, r) + r.bufInUse = hBuf + case iBuf: // Already have interface samples. Add to the interface buf. + r.iBuf = addSample(s, r.iBuf, r) + default: + // Already have specialized samples that are not hSamples. // Need to call the checked add method for conversion. r.add(s) - default: - r.hBuf = addH(s, r.hBuf, r) } } // addFH is a version of the add method specialized for fhSample. func (r *sampleRing) addFH(s fhSample) { - switch { - case len(r.buf) > 0: - // Already have interface samples. Add to the interface buf. - r.buf = addSample(s, r.buf, r) - case len(r.fBuf)+len(r.hBuf) > 0: - // Already have samples that are not fhSamples. + switch r.bufInUse { + case fhBuf: // Add to existing fhSamples. + r.fhBuf = addFH(s, r.fhBuf, r) + case noBuf: // Add first sample. + r.fhBuf = addFH(s, r.fhBuf, r) + r.bufInUse = fhBuf + case iBuf: // Already have interface samples. Add to the interface buf. + r.iBuf = addSample(s, r.iBuf, r) + default: + // Already have specialized samples that are not fhSamples. // Need to call the checked add method for conversion. r.add(s) - default: - r.fhBuf = addFH(s, r.fhBuf, r) } } @@ -701,15 +735,15 @@ func (r *sampleRing) reduceDelta(delta int64) bool { return true } - switch { - case len(r.fBuf) > 0: + switch r.bufInUse { + case fBuf: genericReduceDelta(r.fBuf, r) - case len(r.hBuf) > 0: + case hBuf: genericReduceDelta(r.hBuf, r) - case len(r.fhBuf) > 0: + case fhBuf: genericReduceDelta(r.fhBuf, r) default: - genericReduceDelta(r.buf, r) + genericReduceDelta(r.iBuf, r) } return true } @@ -733,12 +767,12 @@ func (r *sampleRing) nthLast(n int) (tsdbutil.Sample, bool) { return fSample{}, false } i := r.l - n - switch { - case len(r.fBuf) > 0: + switch r.bufInUse { + case fBuf: return r.atF(i), true - case len(r.hBuf) > 0: + case hBuf: return r.atH(i), true - case len(r.fhBuf) > 0: + case fhBuf: return r.atFH(i), true default: return r.at(i), true @@ -751,15 +785,15 @@ func (r *sampleRing) samples() []tsdbutil.Sample { k := r.f + r.l var j int - switch { - case len(r.buf) > 0: - if k > len(r.buf) { - k = len(r.buf) + switch r.bufInUse { + case iBuf: + if k > len(r.iBuf) { + k = len(r.iBuf) j = r.l - k + r.f } - n := copy(res, r.buf[r.f:k]) - copy(res[n:], r.buf[:j]) - case len(r.fBuf) > 0: + n := copy(res, r.iBuf[r.f:k]) + copy(res[n:], r.iBuf[:j]) + case fBuf: if k > len(r.fBuf) { k = len(r.fBuf) j = r.l - k + r.f @@ -770,7 +804,7 @@ func (r *sampleRing) samples() []tsdbutil.Sample { for i, s := range resF { res[i] = s } - case len(r.hBuf) > 0: + case hBuf: if k > len(r.hBuf) { k = len(r.hBuf) j = r.l - k + r.f @@ -781,7 +815,7 @@ func (r *sampleRing) samples() []tsdbutil.Sample { for i, s := range resH { res[i] = s } - case len(r.fhBuf) > 0: + case fhBuf: if k > len(r.fhBuf) { k = len(r.fhBuf) j = r.l - k + r.f From 7bbf24b707966f6577e37beb95634bc58bae4f42 Mon Sep 17 00:00:00 2001 From: Justin Lei Date: Sun, 30 Apr 2023 13:13:25 -0700 Subject: [PATCH 142/231] Make MemoizedSeriesIterator not implement chunkenc.Iterator Signed-off-by: Justin Lei --- promql/engine.go | 2 +- storage/memoized_iterator.go | 35 ++++++++------------- storage/memoized_iterator_test.go | 52 ++++++++++++------------------- 3 files changed, 34 insertions(+), 55 deletions(-) diff --git a/promql/engine.go b/promql/engine.go index 688048e7a6..ae46f60054 100644 --- a/promql/engine.go +++ b/promql/engine.go @@ -1850,7 +1850,7 @@ func (ev *evaluator) vectorSelectorSingle(it *storage.MemoizedSeriesIterator, no } case chunkenc.ValFloat: t, v = it.At() - case chunkenc.ValHistogram, chunkenc.ValFloatHistogram: + case chunkenc.ValFloatHistogram: t, h = it.AtFloatHistogram() default: panic(fmt.Errorf("unknown value type %v", valueType)) diff --git a/storage/memoized_iterator.go b/storage/memoized_iterator.go index 465afa91cd..cb9fdeef46 100644 --- a/storage/memoized_iterator.go +++ b/storage/memoized_iterator.go @@ -21,6 +21,9 @@ import ( ) // MemoizedSeriesIterator wraps an iterator with a buffer to look back the previous element. +// +// This iterator regards integer histograms as float histograms; calls to Seek() will never return chunkenc.Histogram. +// This iterator deliberately does not implement chunkenc.Iterator. type MemoizedSeriesIterator struct { it chunkenc.Iterator delta int64 @@ -78,8 +81,11 @@ func (b *MemoizedSeriesIterator) Seek(t int64) chunkenc.ValueType { b.prevTime = math.MinInt64 b.valueType = b.it.Seek(t0) - if b.valueType == chunkenc.ValNone { + switch b.valueType { + case chunkenc.ValNone: return chunkenc.ValNone + case chunkenc.ValHistogram: + b.valueType = chunkenc.ValFloatHistogram } b.lastTime = b.it.AtT() } @@ -95,7 +101,8 @@ func (b *MemoizedSeriesIterator) Seek(t int64) chunkenc.ValueType { return chunkenc.ValNone } -// Next advances the iterator to the next element. +// Next advances the iterator to the next element. Note that this does not check whether the element being buffered is +// within the time range of the current element and the duration of delta before. func (b *MemoizedSeriesIterator) Next() chunkenc.ValueType { // Keep track of the previous element. switch b.valueType { @@ -104,12 +111,7 @@ func (b *MemoizedSeriesIterator) Next() chunkenc.ValueType { case chunkenc.ValFloat: b.prevTime, b.prevValue = b.it.At() b.prevFloatHistogram = nil - case chunkenc.ValHistogram: - b.prevValue = 0 - ts, h := b.it.AtHistogram() - b.prevTime = ts - b.prevFloatHistogram = h.ToFloat() - case chunkenc.ValFloatHistogram: + case chunkenc.ValHistogram, chunkenc.ValFloatHistogram: b.prevValue = 0 b.prevTime, b.prevFloatHistogram = b.it.AtFloatHistogram() } @@ -118,6 +120,9 @@ func (b *MemoizedSeriesIterator) Next() chunkenc.ValueType { if b.valueType != chunkenc.ValNone { b.lastTime = b.it.AtT() } + if b.valueType == chunkenc.ValHistogram { + b.valueType = chunkenc.ValFloatHistogram + } return b.valueType } @@ -126,25 +131,11 @@ func (b *MemoizedSeriesIterator) At() (int64, float64) { return b.it.At() } -// AtHistogram is not supported by this iterator. -func (b *MemoizedSeriesIterator) AtHistogram() (int64, *histogram.Histogram) { - panic("AtHistogram is not supported by this iterator.") -} - // AtFloatHistogram returns the current float-histogram element of the iterator. func (b *MemoizedSeriesIterator) AtFloatHistogram() (int64, *histogram.FloatHistogram) { - if b.valueType == chunkenc.ValHistogram { - ts, h := b.it.AtHistogram() - return ts, h.ToFloat() - } return b.it.AtFloatHistogram() } -// AtT returns the current timestamp of the iterator. -func (b *MemoizedSeriesIterator) AtT() int64 { - return b.it.AtT() -} - // Err returns the last encountered error. func (b *MemoizedSeriesIterator) Err() error { return b.it.Err() diff --git a/storage/memoized_iterator_test.go b/storage/memoized_iterator_test.go index f922aaeece..1c87119283 100644 --- a/storage/memoized_iterator_test.go +++ b/storage/memoized_iterator_test.go @@ -16,11 +16,11 @@ package storage import ( "testing" + "github.com/stretchr/testify/require" + "github.com/prometheus/prometheus/model/histogram" "github.com/prometheus/prometheus/tsdb/chunkenc" "github.com/prometheus/prometheus/tsdb/tsdbutil" - - "github.com/stretchr/testify/require" ) func TestMemoizedSeriesIterator(t *testing.T) { @@ -64,55 +64,43 @@ func TestMemoizedSeriesIterator(t *testing.T) { hSample{t: 200, h: tsdbutil.GenerateTestHistogram(4)}, fhSample{t: 299, fh: tsdbutil.GenerateTestFloatHistogram(5)}, fSample{t: 300, f: 11}, + hSample{t: 399, h: tsdbutil.GenerateTestHistogram(6)}, + fSample{t: 400, f: 12}, }), 2) require.Equal(t, it.Seek(-123), chunkenc.ValFloat, "seek failed") sampleEq(1, 2, nil) prevSampleEq(0, 0, nil, false) - require.Equal(t, it.Next(), chunkenc.ValFloat, "next failed") - sampleEq(2, 3, nil) - prevSampleEq(1, 2, nil, true) - - require.Equal(t, it.Next(), chunkenc.ValFloat, "next failed") - require.Equal(t, it.Next(), chunkenc.ValFloat, "next failed") - require.Equal(t, it.Next(), chunkenc.ValFloat, "next failed") - sampleEq(5, 6, nil) - prevSampleEq(4, 5, nil, true) - require.Equal(t, it.Seek(5), chunkenc.ValFloat, "seek failed") sampleEq(5, 6, nil) prevSampleEq(4, 5, nil, true) - require.Equal(t, it.Seek(101), chunkenc.ValFloat, "seek failed") - sampleEq(101, 10, nil) - prevSampleEq(100, 9, nil, true) - - require.Equal(t, chunkenc.ValHistogram, it.Next(), "next failed") - sampleEq(102, 0, tsdbutil.GenerateTestFloatHistogram(0)) + // Seek to a histogram sample with a previous float sample. + require.Equal(t, it.Seek(102), chunkenc.ValFloatHistogram, "seek failed") + sampleEq(102, 10, tsdbutil.GenerateTestFloatHistogram(0)) prevSampleEq(101, 10, nil, true) - require.Equal(t, chunkenc.ValHistogram, it.Next(), "next failed") - sampleEq(103, 0, tsdbutil.GenerateTestFloatHistogram(1)) - prevSampleEq(102, 0, tsdbutil.GenerateTestFloatHistogram(0), true) + // Attempt to seek backwards (no-op). + require.Equal(t, it.Seek(50), chunkenc.ValFloatHistogram, "seek failed") + sampleEq(102, 10, tsdbutil.GenerateTestFloatHistogram(0)) + prevSampleEq(101, 10, nil, true) - require.Equal(t, chunkenc.ValFloatHistogram, it.Seek(104), "seek failed") + // Seek to a float histogram sample with a previous histogram sample. + require.Equal(t, it.Seek(104), chunkenc.ValFloatHistogram, "seek failed") sampleEq(104, 0, tsdbutil.GenerateTestFloatHistogram(2)) prevSampleEq(103, 0, tsdbutil.GenerateTestFloatHistogram(1), true) - require.Equal(t, chunkenc.ValFloatHistogram, it.Next(), "next failed") - sampleEq(199, 0, tsdbutil.GenerateTestFloatHistogram(3)) - prevSampleEq(104, 0, tsdbutil.GenerateTestFloatHistogram(2), true) - - require.Equal(t, chunkenc.ValFloatHistogram, it.Seek(280), "seek failed") - sampleEq(299, 0, tsdbutil.GenerateTestFloatHistogram(5)) - prevSampleEq(0, 0, nil, false) - - require.Equal(t, chunkenc.ValFloat, it.Next(), "next failed") + // Seek to a float sample with a previous float histogram sample. + require.Equal(t, chunkenc.ValFloat, it.Seek(300), "seek failed") sampleEq(300, 11, nil) prevSampleEq(299, 0, tsdbutil.GenerateTestFloatHistogram(5), true) - require.Equal(t, it.Next(), chunkenc.ValNone, "next succeeded unexpectedly") + // Seek to a float sample with a previous histogram sample. + require.Equal(t, chunkenc.ValFloat, it.Seek(400), "seek failed") + sampleEq(400, 12, nil) + prevSampleEq(399, 0, tsdbutil.GenerateTestFloatHistogram(6), true) + require.Equal(t, it.Seek(1024), chunkenc.ValNone, "seek succeeded unexpectedly") } From e9b2d874431ca464f55ea11685e3278e4c19045a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gy=C3=B6rgy=20Krajcsovits?= Date: Thu, 4 May 2023 08:16:37 +0200 Subject: [PATCH 143/231] Revert change to model/textparse/protobufparse_test.go MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: György Krajcsovits --- model/textparse/protobufparse_test.go | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/model/textparse/protobufparse_test.go b/model/textparse/protobufparse_test.go index fb8669197e..90c6a90f32 100644 --- a/model/textparse/protobufparse_test.go +++ b/model/textparse/protobufparse_test.go @@ -15,6 +15,7 @@ package textparse import ( "bytes" + "encoding/binary" "errors" "io" "testing" @@ -25,9 +26,8 @@ import ( "github.com/prometheus/prometheus/model/exemplar" "github.com/prometheus/prometheus/model/histogram" "github.com/prometheus/prometheus/model/labels" - "github.com/prometheus/prometheus/util/testutil" - dto "github.com/prometheus/client_model/go" + dto "github.com/prometheus/prometheus/prompb/io/prometheus/client" ) func TestProtobufParse(t *testing.T) { @@ -449,6 +449,7 @@ metric: < `, } + varintBuf := make([]byte, binary.MaxVarintLen32) inputBuf := &bytes.Buffer{} for _, tmf := range textMetricFamilies { @@ -456,8 +457,13 @@ metric: < // From text to proto message. require.NoError(t, proto.UnmarshalText(tmf, pb)) // From proto message to binary protobuf. - err := testutil.AddMetricFamilyToProtobuf(inputBuf, pb) + protoBuf, err := proto.Marshal(pb) require.NoError(t, err) + + // Write first length, then binary protobuf. + varintLength := binary.PutUvarint(varintBuf, uint64(len(protoBuf))) + inputBuf.Write(varintBuf[:varintLength]) + inputBuf.Write(protoBuf) } exp := []struct { From 19a4f314f53a80e653de6fcbf6b90d70a35fc93d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gy=C3=B6rgy=20Krajcsovits?= Date: Thu, 4 May 2023 08:36:44 +0200 Subject: [PATCH 144/231] Refactor testutil/protobuf.go into scrape package MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Renamed to clientprotobuf.go and added comments to indicate the intended usage. Signed-off-by: György Krajcsovits --- util/testutil/protobuf.go => scrape/clientprotobuf.go | 9 ++++++--- scrape/scrape_test.go | 4 ++-- 2 files changed, 8 insertions(+), 5 deletions(-) rename util/testutil/protobuf.go => scrape/clientprotobuf.go (78%) diff --git a/util/testutil/protobuf.go b/scrape/clientprotobuf.go similarity index 78% rename from util/testutil/protobuf.go rename to scrape/clientprotobuf.go index 8650f13730..bf165e0343 100644 --- a/util/testutil/protobuf.go +++ b/scrape/clientprotobuf.go @@ -11,17 +11,19 @@ // See the License for the specific language governing permissions and // limitations under the License. -package testutil +package scrape import ( "bytes" "encoding/binary" "github.com/gogo/protobuf/proto" + // Intentionally using client model to simulate client in tests. dto "github.com/prometheus/client_model/go" ) -// Write a MetricFamily into a protobuf +// Write a MetricFamily into a protobuf. +// This function is intended for testing scraping by providing protobuf serialized input. func MetricFamilyToProtobuf(metricFamily *dto.MetricFamily) ([]byte, error) { buffer := &bytes.Buffer{} err := AddMetricFamilyToProtobuf(buffer, metricFamily) @@ -31,7 +33,8 @@ func MetricFamilyToProtobuf(metricFamily *dto.MetricFamily) ([]byte, error) { return buffer.Bytes(), nil } -// Append a MetricFamily protobuf representation to a buffer +// Append a MetricFamily protobuf representation to a buffer. +// This function is intended for testing scraping by providing protobuf serialized input. func AddMetricFamilyToProtobuf(buffer *bytes.Buffer, metricFamily *dto.MetricFamily) error { protoBuf, err := proto.Marshal(metricFamily) if err != nil { diff --git a/scrape/scrape_test.go b/scrape/scrape_test.go index 745765ee8f..fc3ad926eb 100644 --- a/scrape/scrape_test.go +++ b/scrape/scrape_test.go @@ -1766,7 +1766,7 @@ func TestScrapeLoop_HistogramBucketLimit(t *testing.T) { require.NotEmpty(t, gathered) histogramMetricFamily := gathered[0] - msg, err := testutil.MetricFamilyToProtobuf(histogramMetricFamily) + msg, err := MetricFamilyToProtobuf(histogramMetricFamily) require.NoError(t, err) now := time.Now() @@ -1789,7 +1789,7 @@ func TestScrapeLoop_HistogramBucketLimit(t *testing.T) { require.NotEmpty(t, gathered) histogramMetricFamily = gathered[0] - msg, err = testutil.MetricFamilyToProtobuf(histogramMetricFamily) + msg, err = MetricFamilyToProtobuf(histogramMetricFamily) require.NoError(t, err) now = time.Now() From 40240c9c1cb290fe95f1e61886b23fab860aeacd Mon Sep 17 00:00:00 2001 From: Jeanette Tan Date: Fri, 5 May 2023 02:29:50 +0800 Subject: [PATCH 145/231] Update according to code review Signed-off-by: Jeanette Tan --- config/config.go | 2 +- docs/configuration/configuration.md | 8 ++++---- scrape/clientprotobuf.go | 1 + scrape/scrape.go | 14 +++++++------- 4 files changed, 13 insertions(+), 12 deletions(-) diff --git a/config/config.go b/config/config.go index 31ebb288e3..1b6a6cf6b6 100644 --- a/config/config.go +++ b/config/config.go @@ -491,7 +491,7 @@ type ScrapeConfig struct { LabelValueLengthLimit uint `yaml:"label_value_length_limit,omitempty"` // More than this many buckets in a native histogram will cause the scrape to // fail. - NativeHistogramBucketLimit uint `yaml:"bucket_limit,omitempty"` + NativeHistogramBucketLimit uint `yaml:"native_histogram_bucket_limit,omitempty"` // We cannot do proper Go type embedding below as the parser will then parse // values arbitrarily into the overflow maps of further-down types. diff --git a/docs/configuration/configuration.md b/docs/configuration/configuration.md index cfb11e21e4..0a8c4a5cdf 100644 --- a/docs/configuration/configuration.md +++ b/docs/configuration/configuration.md @@ -377,10 +377,10 @@ metric_relabel_configs: # change in the future. [ target_limit: | default = 0 ] -# Limit on total number of positive and negative buckets allowed in a native -# histogram. If this is exceeded, the entire scrape will be treated as failed. -# 0 means no limit. -[ sample_limit: | default = 0 ] +# Limit on total number of positive and negative buckets allowed in a single +# native histogram. If this is exceeded, the entire scrape will be treated as +# failed. 0 means no limit. +[ native_histogram_bucket_limit: | default = 0 ] ``` Where `` must be unique across all scrape configurations. diff --git a/scrape/clientprotobuf.go b/scrape/clientprotobuf.go index bf165e0343..2213268d59 100644 --- a/scrape/clientprotobuf.go +++ b/scrape/clientprotobuf.go @@ -18,6 +18,7 @@ import ( "encoding/binary" "github.com/gogo/protobuf/proto" + // Intentionally using client model to simulate client in tests. dto "github.com/prometheus/client_model/go" ) diff --git a/scrape/scrape.go b/scrape/scrape.go index 179adbb2a2..f094ee8257 100644 --- a/scrape/scrape.go +++ b/scrape/scrape.go @@ -193,8 +193,8 @@ var ( ) targetScrapeNativeHistogramBucketLimit = prometheus.NewCounter( prometheus.CounterOpts{ - Name: "prometheus_target_scrapes_histogram_exceeded_bucket_limit_total", - Help: "Total number of scrapes that hit the native histograms bucket limit and were rejected.", + Name: "prometheus_target_scrapes_exceeded_native_histogram_bucket_limit_total", + Help: "Total number of scrapes that hit the native histogram bucket limit and were rejected.", }, ) ) @@ -744,17 +744,17 @@ func mutateReportSampleLabels(lset labels.Labels, target *Target) labels.Labels } // appender returns an appender for ingested samples from the target. -func appender(app storage.Appender, limit, bucketLimit int) storage.Appender { +func appender(app storage.Appender, sampleLimit, bucketLimit int) storage.Appender { app = &timeLimitAppender{ Appender: app, maxTime: timestamp.FromTime(time.Now().Add(maxAheadTime)), } - // The limit is applied after metrics are potentially dropped via relabeling. - if limit > 0 { + // The sampleLimit is applied after metrics are potentially dropped via relabeling. + if sampleLimit > 0 { app = &limitAppender{ Appender: app, - limit: limit, + limit: sampleLimit, } } @@ -1707,7 +1707,7 @@ loop: } if bucketLimitErr != nil { if err == nil { - err = bucketLimitErr // if sample limit is hit, that error takes precedence + err = bucketLimitErr // If sample limit is hit, that error takes precedence. } // We only want to increment this once per scrape, so this is Inc'd outside the loop. targetScrapeNativeHistogramBucketLimit.Inc() From e278195e3983c966c2a0f42211f62fa8f40c5561 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Thu, 4 May 2023 20:09:31 +0100 Subject: [PATCH 146/231] Cherry-pick bugfix #12322 and create v2.43.1 (#12324) * labels: respect Set after Del in Builder (#12322) The implementations are not symmetric between `Set()` and `Del()`, so we must be careful. Add tests for this, both in labels and in relabel where the issue was reported. Also make the slice implementation consistent re `slices.Contains`. * Create v2.43.1 with bugfix Signed-off-by: Bryan Boreham Co-authored-by: Julius Volz --- CHANGELOG.md | 4 +++ VERSION | 2 +- model/labels/labels.go | 9 +++---- model/labels/labels_string.go | 7 ++--- model/labels/labels_test.go | 7 +++++ model/relabel/relabel_test.go | 28 ++++++++++++++++++++ web/ui/module/codemirror-promql/package.json | 4 +-- web/ui/module/lezer-promql/package.json | 2 +- web/ui/package-lock.json | 14 +++++----- web/ui/react-app/package.json | 4 +-- 10 files changed, 60 insertions(+), 21 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6e6b30f47a..af90af1e14 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,9 @@ # Changelog +## 2.43.1 / 2023-05-03 + +* [BUGFIX] Labels: `Set()` after `Del()` would be ignored, which broke some relabeling rules. #12322 + ## 2.43.0 / 2023-03-21 We are working on some performance improvements in Prometheus, which are only diff --git a/VERSION b/VERSION index 5b9cd9afd5..b1d9893408 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.43.0 +2.43.1 diff --git a/model/labels/labels.go b/model/labels/labels.go index 6de001c3ce..2622f79410 100644 --- a/model/labels/labels.go +++ b/model/labels/labels.go @@ -531,16 +531,15 @@ func (b *Builder) Set(n, v string) *Builder { } func (b *Builder) Get(n string) string { - for _, d := range b.del { - if d == n { - return "" - } - } + // Del() removes entries from .add but Set() does not remove from .del, so check .add first. for _, a := range b.add { if a.Name == n { return a.Value } } + if slices.Contains(b.del, n) { + return "" + } return b.base.Get(n) } diff --git a/model/labels/labels_string.go b/model/labels/labels_string.go index 98db29d254..db8c981e03 100644 --- a/model/labels/labels_string.go +++ b/model/labels/labels_string.go @@ -587,14 +587,15 @@ func (b *Builder) Set(n, v string) *Builder { } func (b *Builder) Get(n string) string { - if slices.Contains(b.del, n) { - return "" - } + // Del() removes entries from .add but Set() does not remove from .del, so check .add first. for _, a := range b.add { if a.Name == n { return a.Value } } + if slices.Contains(b.del, n) { + return "" + } return b.base.Get(n) } diff --git a/model/labels/labels_test.go b/model/labels/labels_test.go index 588a84b984..98cb12f362 100644 --- a/model/labels/labels_test.go +++ b/model/labels/labels_test.go @@ -607,6 +607,13 @@ func TestBuilder(t *testing.T) { require.Equal(t, tcase.want.BytesWithoutLabels(nil, "aaa", "bbb"), b.Labels(tcase.base).Bytes(nil)) }) } + t.Run("set_after_del", func(t *testing.T) { + b := NewBuilder(FromStrings("aaa", "111")) + b.Del("bbb") + b.Set("bbb", "222") + require.Equal(t, FromStrings("aaa", "111", "bbb", "222"), b.Labels(EmptyLabels())) + require.Equal(t, "222", b.Get("bbb")) + }) } func TestScratchBuilder(t *testing.T) { diff --git a/model/relabel/relabel_test.go b/model/relabel/relabel_test.go index d277d778d1..b50ff4010a 100644 --- a/model/relabel/relabel_test.go +++ b/model/relabel/relabel_test.go @@ -397,6 +397,34 @@ func TestRelabel(t *testing.T) { "foo": "bar", }), }, + { // From https://github.com/prometheus/prometheus/issues/12283 + input: labels.FromMap(map[string]string{ + "__meta_kubernetes_pod_container_port_name": "foo", + "__meta_kubernetes_pod_annotation_XXX_metrics_port": "9091", + }), + relabel: []*Config{ + { + Regex: MustNewRegexp("^__meta_kubernetes_pod_container_port_name$"), + Action: LabelDrop, + }, + { + SourceLabels: model.LabelNames{"__meta_kubernetes_pod_annotation_XXX_metrics_port"}, + Regex: MustNewRegexp("(.+)"), + Action: Replace, + Replacement: "metrics", + TargetLabel: "__meta_kubernetes_pod_container_port_name", + }, + { + SourceLabels: model.LabelNames{"__meta_kubernetes_pod_container_port_name"}, + Regex: MustNewRegexp("^metrics$"), + Action: Keep, + }, + }, + output: labels.FromMap(map[string]string{ + "__meta_kubernetes_pod_annotation_XXX_metrics_port": "9091", + "__meta_kubernetes_pod_container_port_name": "metrics", + }), + }, { input: labels.FromMap(map[string]string{ "a": "foo", diff --git a/web/ui/module/codemirror-promql/package.json b/web/ui/module/codemirror-promql/package.json index f20a634683..1ea56ae459 100644 --- a/web/ui/module/codemirror-promql/package.json +++ b/web/ui/module/codemirror-promql/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/codemirror-promql", - "version": "0.43.0", + "version": "0.43.1", "description": "a CodeMirror mode for the PromQL language", "types": "dist/esm/index.d.ts", "module": "dist/esm/index.js", @@ -29,7 +29,7 @@ }, "homepage": "https://github.com/prometheus/prometheus/blob/main/web/ui/module/codemirror-promql/README.md", "dependencies": { - "@prometheus-io/lezer-promql": "0.43.0", + "@prometheus-io/lezer-promql": "0.43.1", "lru-cache": "^6.0.0" }, "devDependencies": { diff --git a/web/ui/module/lezer-promql/package.json b/web/ui/module/lezer-promql/package.json index 411251e56c..99b98bdeaf 100644 --- a/web/ui/module/lezer-promql/package.json +++ b/web/ui/module/lezer-promql/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/lezer-promql", - "version": "0.43.0", + "version": "0.43.1", "description": "lezer-based PromQL grammar", "main": "dist/index.cjs", "type": "module", diff --git a/web/ui/package-lock.json b/web/ui/package-lock.json index 2e334cf9e4..797a177c1a 100644 --- a/web/ui/package-lock.json +++ b/web/ui/package-lock.json @@ -28,10 +28,10 @@ }, "module/codemirror-promql": { "name": "@prometheus-io/codemirror-promql", - "version": "0.43.0", + "version": "0.43.1", "license": "Apache-2.0", "dependencies": { - "@prometheus-io/lezer-promql": "0.43.0", + "@prometheus-io/lezer-promql": "0.43.1", "lru-cache": "^6.0.0" }, "devDependencies": { @@ -61,7 +61,7 @@ }, "module/lezer-promql": { "name": "@prometheus-io/lezer-promql", - "version": "0.43.0", + "version": "0.43.1", "license": "Apache-2.0", "devDependencies": { "@lezer/generator": "^1.2.2", @@ -20763,7 +20763,7 @@ }, "react-app": { "name": "@prometheus-io/app", - "version": "0.43.0", + "version": "0.43.1", "dependencies": { "@codemirror/autocomplete": "^6.4.0", "@codemirror/commands": "^6.2.0", @@ -20781,7 +20781,7 @@ "@lezer/lr": "^1.3.1", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.43.0", + "@prometheus-io/codemirror-promql": "0.43.1", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", "downshift": "^7.2.0", @@ -23417,7 +23417,7 @@ "@lezer/lr": "^1.3.1", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.43.0", + "@prometheus-io/codemirror-promql": "0.43.1", "@testing-library/react-hooks": "^7.0.2", "@types/enzyme": "^3.10.12", "@types/flot": "0.0.32", @@ -23468,7 +23468,7 @@ "@lezer/common": "^1.0.2", "@lezer/highlight": "^1.1.3", "@lezer/lr": "^1.3.1", - "@prometheus-io/lezer-promql": "0.43.0", + "@prometheus-io/lezer-promql": "0.43.1", "@types/lru-cache": "^5.1.1", "isomorphic-fetch": "^3.0.0", "lru-cache": "^6.0.0", diff --git a/web/ui/react-app/package.json b/web/ui/react-app/package.json index c8b115582a..37ad5e1959 100644 --- a/web/ui/react-app/package.json +++ b/web/ui/react-app/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/app", - "version": "0.43.0", + "version": "0.43.1", "private": true, "dependencies": { "@codemirror/autocomplete": "^6.4.0", @@ -19,7 +19,7 @@ "@lezer/common": "^1.0.2", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.43.0", + "@prometheus-io/codemirror-promql": "0.43.1", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", "downshift": "^7.2.0", From f5fcaa3872ce03808567fabc56afc9cf61c732cb Mon Sep 17 00:00:00 2001 From: George Krajcsovits Date: Fri, 5 May 2023 14:34:30 +0200 Subject: [PATCH 147/231] Fix setting reset header to gauge histogram in seriesToChunkEncoder (#12329) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: György Krajcsovits --- storage/series.go | 10 +++ storage/series_test.go | 177 +++++++++++++++++++++++++++++++---------- 2 files changed, 147 insertions(+), 40 deletions(-) diff --git a/storage/series.go b/storage/series.go index 5daa6255da..b73f1e35ce 100644 --- a/storage/series.go +++ b/storage/series.go @@ -297,9 +297,11 @@ func (s *seriesToChunkEncoder) Iterator(it chunks.Iterator) chunks.Iterator { seriesIter := s.Series.Iterator(nil) lastType := chunkenc.ValNone for typ := seriesIter.Next(); typ != chunkenc.ValNone; typ = seriesIter.Next() { + chunkCreated := false if typ != lastType || i >= seriesToChunkEncoderSplit { // Create a new chunk if the sample type changed or too many samples in the current one. chks = appendChunk(chks, mint, maxt, chk) + chunkCreated = true chk, err = chunkenc.NewEmptyChunk(typ.ChunkEncoding()) if err != nil { return errChunksIterator{err: err} @@ -330,6 +332,7 @@ func (s *seriesToChunkEncoder) Iterator(it chunks.Iterator) chunks.Iterator { if ok, counterReset := app.AppendHistogram(t, h); !ok { chks = appendChunk(chks, mint, maxt, chk) histChunk := chunkenc.NewHistogramChunk() + chunkCreated = true if counterReset { histChunk.SetCounterResetHeader(chunkenc.CounterReset) } @@ -346,11 +349,15 @@ func (s *seriesToChunkEncoder) Iterator(it chunks.Iterator) chunks.Iterator { panic("unexpected error while appending histogram") } } + if chunkCreated && h.CounterResetHint == histogram.GaugeType { + chk.(*chunkenc.HistogramChunk).SetCounterResetHeader(chunkenc.GaugeType) + } case chunkenc.ValFloatHistogram: t, fh = seriesIter.AtFloatHistogram() if ok, counterReset := app.AppendFloatHistogram(t, fh); !ok { chks = appendChunk(chks, mint, maxt, chk) floatHistChunk := chunkenc.NewFloatHistogramChunk() + chunkCreated = true if counterReset { floatHistChunk.SetCounterResetHeader(chunkenc.CounterReset) } @@ -366,6 +373,9 @@ func (s *seriesToChunkEncoder) Iterator(it chunks.Iterator) chunks.Iterator { panic("unexpected error while float appending histogram") } } + if chunkCreated && fh.CounterResetHint == histogram.GaugeType { + chk.(*chunkenc.FloatHistogramChunk).SetCounterResetHeader(chunkenc.GaugeType) + } default: return errChunksIterator{err: fmt.Errorf("unknown sample type %s", typ.String())} } diff --git a/storage/series_test.go b/storage/series_test.go index 4c318f1a0e..5c74fae096 100644 --- a/storage/series_test.go +++ b/storage/series_test.go @@ -126,14 +126,13 @@ func TestChunkSeriesSetToSeriesSet(t *testing.T) { } type histogramTest struct { - samples []tsdbutil.Sample - expectedChunks int - expectedCounterReset bool + samples []tsdbutil.Sample + expectedCounterResetHeaders []chunkenc.CounterResetHeader } func TestHistogramSeriesToChunks(t *testing.T) { h1 := &histogram.Histogram{ - Count: 3, + Count: 7, ZeroCount: 2, ZeroThreshold: 0.001, Sum: 100, @@ -158,7 +157,7 @@ func TestHistogramSeriesToChunks(t *testing.T) { } // Implicit counter reset by reduction in buckets, not appendable. h2down := &histogram.Histogram{ - Count: 8, + Count: 10, ZeroCount: 2, ZeroThreshold: 0.001, Sum: 100, @@ -171,7 +170,7 @@ func TestHistogramSeriesToChunks(t *testing.T) { } fh1 := &histogram.FloatHistogram{ - Count: 4, + Count: 6, ZeroCount: 2, ZeroThreshold: 0.001, Sum: 100, @@ -183,7 +182,7 @@ func TestHistogramSeriesToChunks(t *testing.T) { } // Appendable to fh1. fh2 := &histogram.FloatHistogram{ - Count: 15, + Count: 17, ZeroCount: 2, ZeroThreshold: 0.001, Sum: 100, @@ -196,7 +195,7 @@ func TestHistogramSeriesToChunks(t *testing.T) { } // Implicit counter reset by reduction in buckets, not appendable. fh2down := &histogram.FloatHistogram{ - Count: 13, + Count: 15, ZeroCount: 2, ZeroThreshold: 0.001, Sum: 100, @@ -208,6 +207,60 @@ func TestHistogramSeriesToChunks(t *testing.T) { PositiveBuckets: []float64{2, 2, 7, 2}, } + // Gauge histogram. + gh1 := &histogram.Histogram{ + CounterResetHint: histogram.GaugeType, + Count: 7, + ZeroCount: 2, + ZeroThreshold: 0.001, + Sum: 100, + Schema: 0, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 2}, + }, + PositiveBuckets: []int64{2, 1}, // Abs: 2, 3 + } + gh2 := &histogram.Histogram{ + CounterResetHint: histogram.GaugeType, + Count: 12, + ZeroCount: 2, + ZeroThreshold: 0.001, + Sum: 100, + Schema: 0, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 2}, + {Offset: 1, Length: 2}, + }, + PositiveBuckets: []int64{2, 1, -2, 3}, // Abs: 2, 3, 1, 4 + } + + // Float gauge histogram. + gfh1 := &histogram.FloatHistogram{ + CounterResetHint: histogram.GaugeType, + Count: 6, + ZeroCount: 2, + ZeroThreshold: 0.001, + Sum: 100, + Schema: 0, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 2}, + }, + PositiveBuckets: []float64{3, 1}, + } + gfh2 := &histogram.FloatHistogram{ + CounterResetHint: histogram.GaugeType, + Count: 17, + ZeroCount: 2, + ZeroThreshold: 0.001, + Sum: 100, + Schema: 0, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 2}, + {Offset: 1, Length: 2}, + }, + PositiveBuckets: []float64{4, 2, 7, 2}, + } + staleHistogram := &histogram.Histogram{ Sum: math.Float64frombits(value.StaleNaN), } @@ -220,74 +273,70 @@ func TestHistogramSeriesToChunks(t *testing.T) { samples: []tsdbutil.Sample{ hSample{t: 1, h: h1}, }, - expectedChunks: 1, + expectedCounterResetHeaders: []chunkenc.CounterResetHeader{chunkenc.UnknownCounterReset}, }, "two histograms encoded to a single chunk": { samples: []tsdbutil.Sample{ hSample{t: 1, h: h1}, hSample{t: 2, h: h2}, }, - expectedChunks: 1, + expectedCounterResetHeaders: []chunkenc.CounterResetHeader{chunkenc.UnknownCounterReset}, }, "two histograms encoded to two chunks": { samples: []tsdbutil.Sample{ hSample{t: 1, h: h2}, hSample{t: 2, h: h1}, }, - expectedChunks: 2, - expectedCounterReset: true, + expectedCounterResetHeaders: []chunkenc.CounterResetHeader{chunkenc.UnknownCounterReset, chunkenc.CounterReset}, }, "histogram and stale sample encoded to two chunks": { samples: []tsdbutil.Sample{ hSample{t: 1, h: staleHistogram}, hSample{t: 2, h: h1}, }, - expectedChunks: 2, + expectedCounterResetHeaders: []chunkenc.CounterResetHeader{chunkenc.UnknownCounterReset, chunkenc.UnknownCounterReset}, }, "histogram and reduction in bucket encoded to two chunks": { samples: []tsdbutil.Sample{ hSample{t: 1, h: h1}, hSample{t: 2, h: h2down}, }, - expectedChunks: 2, - expectedCounterReset: true, + expectedCounterResetHeaders: []chunkenc.CounterResetHeader{chunkenc.UnknownCounterReset, chunkenc.CounterReset}, }, // Float histograms. "single float histogram to single chunk": { samples: []tsdbutil.Sample{ fhSample{t: 1, fh: fh1}, }, - expectedChunks: 1, + expectedCounterResetHeaders: []chunkenc.CounterResetHeader{chunkenc.UnknownCounterReset}, }, "two float histograms encoded to a single chunk": { samples: []tsdbutil.Sample{ fhSample{t: 1, fh: fh1}, fhSample{t: 2, fh: fh2}, }, - expectedChunks: 1, + expectedCounterResetHeaders: []chunkenc.CounterResetHeader{chunkenc.UnknownCounterReset}, }, "two float histograms encoded to two chunks": { samples: []tsdbutil.Sample{ fhSample{t: 1, fh: fh2}, fhSample{t: 2, fh: fh1}, }, - expectedChunks: 2, - expectedCounterReset: true, + expectedCounterResetHeaders: []chunkenc.CounterResetHeader{chunkenc.UnknownCounterReset, chunkenc.CounterReset}, }, "float histogram and stale sample encoded to two chunks": { samples: []tsdbutil.Sample{ fhSample{t: 1, fh: staleFloatHistogram}, fhSample{t: 2, fh: fh1}, }, - expectedChunks: 2, + expectedCounterResetHeaders: []chunkenc.CounterResetHeader{chunkenc.UnknownCounterReset, chunkenc.UnknownCounterReset}, }, "float histogram and reduction in bucket encoded to two chunks": { samples: []tsdbutil.Sample{ fhSample{t: 1, fh: fh1}, fhSample{t: 2, fh: fh2down}, }, - expectedChunks: 2, - expectedCounterReset: true, + expectedCounterResetHeaders: []chunkenc.CounterResetHeader{chunkenc.UnknownCounterReset, chunkenc.CounterReset}, }, // Mixed. "histogram and float histogram encoded to two chunks": { @@ -295,21 +344,61 @@ func TestHistogramSeriesToChunks(t *testing.T) { hSample{t: 1, h: h1}, fhSample{t: 2, fh: fh2}, }, - expectedChunks: 2, + expectedCounterResetHeaders: []chunkenc.CounterResetHeader{chunkenc.UnknownCounterReset, chunkenc.UnknownCounterReset}, }, "float histogram and histogram encoded to two chunks": { samples: []tsdbutil.Sample{ fhSample{t: 1, fh: fh1}, hSample{t: 2, h: h2}, }, - expectedChunks: 2, + expectedCounterResetHeaders: []chunkenc.CounterResetHeader{chunkenc.UnknownCounterReset, chunkenc.UnknownCounterReset}, }, "histogram and stale float histogram encoded to two chunks": { samples: []tsdbutil.Sample{ hSample{t: 1, h: h1}, fhSample{t: 2, fh: staleFloatHistogram}, }, - expectedChunks: 2, + expectedCounterResetHeaders: []chunkenc.CounterResetHeader{chunkenc.UnknownCounterReset, chunkenc.UnknownCounterReset}, + }, + "single gauge histogram encoded to one chunk": { + samples: []tsdbutil.Sample{ + hSample{t: 1, h: gh1}, + }, + expectedCounterResetHeaders: []chunkenc.CounterResetHeader{chunkenc.GaugeType}, + }, + "two gauge histograms encoded to one chunk when counter increases": { + samples: []tsdbutil.Sample{ + hSample{t: 1, h: gh1}, + hSample{t: 2, h: gh2}, + }, + expectedCounterResetHeaders: []chunkenc.CounterResetHeader{chunkenc.GaugeType}, + }, + "two gauge histograms encoded to one chunk when counter decreases": { + samples: []tsdbutil.Sample{ + hSample{t: 1, h: gh2}, + hSample{t: 2, h: gh1}, + }, + expectedCounterResetHeaders: []chunkenc.CounterResetHeader{chunkenc.GaugeType}, + }, + "single gauge float histogram encoded to one chunk": { + samples: []tsdbutil.Sample{ + fhSample{t: 1, fh: gfh1}, + }, + expectedCounterResetHeaders: []chunkenc.CounterResetHeader{chunkenc.GaugeType}, + }, + "two float gauge histograms encoded to one chunk when counter increases": { + samples: []tsdbutil.Sample{ + fhSample{t: 1, fh: gfh1}, + fhSample{t: 2, fh: gfh2}, + }, + expectedCounterResetHeaders: []chunkenc.CounterResetHeader{chunkenc.GaugeType}, + }, + "two float gauge histograms encoded to one chunk when counter decreases": { + samples: []tsdbutil.Sample{ + fhSample{t: 1, fh: gfh2}, + fhSample{t: 2, fh: gfh1}, + }, + expectedCounterResetHeaders: []chunkenc.CounterResetHeader{chunkenc.GaugeType}, }, } @@ -322,13 +411,24 @@ func TestHistogramSeriesToChunks(t *testing.T) { func testHistogramsSeriesToChunks(t *testing.T, test histogramTest) { lbs := labels.FromStrings("__name__", "up", "instance", "localhost:8080") - series := NewListSeries(lbs, test.samples) + copiedSamples := []tsdbutil.Sample{} + for _, s := range test.samples { + switch cs := s.(type) { + case hSample: + copiedSamples = append(copiedSamples, hSample{t: cs.t, h: cs.h.Copy()}) + case fhSample: + copiedSamples = append(copiedSamples, fhSample{t: cs.t, fh: cs.fh.Copy()}) + default: + t.Error("internal error, unexpected type") + } + } + series := NewListSeries(lbs, copiedSamples) encoder := NewSeriesToChunkEncoder(series) require.EqualValues(t, lbs, encoder.Labels()) chks, err := ExpandChunks(encoder.Iterator(nil)) require.NoError(t, err) - require.Equal(t, test.expectedChunks, len(chks)) + require.Equal(t, len(test.expectedCounterResetHeaders), len(chks)) // Decode all encoded samples and assert they are equal to the original ones. encodedSamples := expandHistogramSamples(chks) @@ -339,8 +439,10 @@ func testHistogramsSeriesToChunks(t *testing.T, test histogramTest) { case hSample: encodedSample, ok := encodedSamples[i].(hSample) require.True(t, ok, "expect histogram", fmt.Sprintf("at idx %d", i)) - // Ignore counter reset here, will check on chunk level. - encodedSample.h.CounterResetHint = histogram.UnknownCounterReset + // Ignore counter reset if not gauge here, will check on chunk level. + if expectedSample.h.CounterResetHint != histogram.GaugeType { + encodedSample.h.CounterResetHint = histogram.UnknownCounterReset + } if value.IsStaleNaN(expectedSample.h.Sum) { require.True(t, value.IsStaleNaN(encodedSample.h.Sum), fmt.Sprintf("at idx %d", i)) continue @@ -349,8 +451,10 @@ func testHistogramsSeriesToChunks(t *testing.T, test histogramTest) { case fhSample: encodedSample, ok := encodedSamples[i].(fhSample) require.True(t, ok, "expect float histogram", fmt.Sprintf("at idx %d", i)) - // Ignore counter reset here, will check on chunk level. - encodedSample.fh.CounterResetHint = histogram.UnknownCounterReset + // Ignore counter reset if not gauge here, will check on chunk level. + if expectedSample.fh.CounterResetHint != histogram.GaugeType { + encodedSample.fh.CounterResetHint = histogram.UnknownCounterReset + } if value.IsStaleNaN(expectedSample.fh.Sum) { require.True(t, value.IsStaleNaN(encodedSample.fh.Sum), fmt.Sprintf("at idx %d", i)) continue @@ -361,15 +465,8 @@ func testHistogramsSeriesToChunks(t *testing.T, test histogramTest) { } } - // If a counter reset hint is expected, it can only be found in the second chunk. - // Otherwise, we assert an unknown counter reset hint in all chunks. - if test.expectedCounterReset { - require.Equal(t, chunkenc.UnknownCounterReset, getCounterResetHint(chks[0])) - require.Equal(t, chunkenc.CounterReset, getCounterResetHint(chks[1])) - } else { - for _, chk := range chks { - require.Equal(t, chunkenc.UnknownCounterReset, getCounterResetHint(chk)) - } + for i, expectedCounterResetHint := range test.expectedCounterResetHeaders { + require.Equal(t, expectedCounterResetHint, getCounterResetHint(chks[i]), fmt.Sprintf("chunk at index %d", i)) } } From 94d9367bbf13a5325c0d55b22babf56d164f91e1 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Mon, 8 May 2023 11:30:29 +0100 Subject: [PATCH 148/231] Create 2.44.0-rc.2 (#12341) Signed-off-by: Bryan Boreham --- CHANGELOG.md | 4 ++++ VERSION | 2 +- web/ui/module/codemirror-promql/package.json | 4 ++-- web/ui/module/lezer-promql/package.json | 2 +- web/ui/package-lock.json | 14 +++++++------- web/ui/react-app/package.json | 4 ++-- 6 files changed, 17 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 06fb01ad07..ebc602800d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,9 @@ # Changelog +## 2.44.0-rc.2 / 2023-05-07 + +* [ENHANCEMENT] Storage: Optimise buffer used to iterate through samples. #12326 + ## 2.44.0-rc.1 / 2023-05-03 * [BUGFIX] Labels: Set after Del would be ignored, which broke some relabeling rules. #12322 diff --git a/VERSION b/VERSION index 662635da4e..3f6719d550 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.44.0-rc.1 +2.44.0-rc.2 diff --git a/web/ui/module/codemirror-promql/package.json b/web/ui/module/codemirror-promql/package.json index 11fdeee455..048b1aab46 100644 --- a/web/ui/module/codemirror-promql/package.json +++ b/web/ui/module/codemirror-promql/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/codemirror-promql", - "version": "0.44.0-rc.1", + "version": "0.44.0-rc.2", "description": "a CodeMirror mode for the PromQL language", "types": "dist/esm/index.d.ts", "module": "dist/esm/index.js", @@ -29,7 +29,7 @@ }, "homepage": "https://github.com/prometheus/prometheus/blob/main/web/ui/module/codemirror-promql/README.md", "dependencies": { - "@prometheus-io/lezer-promql": "0.44.0-rc.1", + "@prometheus-io/lezer-promql": "0.44.0-rc.2", "lru-cache": "^6.0.0" }, "devDependencies": { diff --git a/web/ui/module/lezer-promql/package.json b/web/ui/module/lezer-promql/package.json index 4b3f29b521..622e9bea4a 100644 --- a/web/ui/module/lezer-promql/package.json +++ b/web/ui/module/lezer-promql/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/lezer-promql", - "version": "0.44.0-rc.1", + "version": "0.44.0-rc.2", "description": "lezer-based PromQL grammar", "main": "dist/index.cjs", "type": "module", diff --git a/web/ui/package-lock.json b/web/ui/package-lock.json index 37804eebf2..0cd9eb7291 100644 --- a/web/ui/package-lock.json +++ b/web/ui/package-lock.json @@ -28,10 +28,10 @@ }, "module/codemirror-promql": { "name": "@prometheus-io/codemirror-promql", - "version": "0.44.0-rc.1", + "version": "0.44.0-rc.2", "license": "Apache-2.0", "dependencies": { - "@prometheus-io/lezer-promql": "0.44.0-rc.1", + "@prometheus-io/lezer-promql": "0.44.0-rc.2", "lru-cache": "^6.0.0" }, "devDependencies": { @@ -61,7 +61,7 @@ }, "module/lezer-promql": { "name": "@prometheus-io/lezer-promql", - "version": "0.44.0-rc.1", + "version": "0.44.0-rc.2", "license": "Apache-2.0", "devDependencies": { "@lezer/generator": "^1.2.2", @@ -20763,7 +20763,7 @@ }, "react-app": { "name": "@prometheus-io/app", - "version": "0.44.0-rc.1", + "version": "0.44.0-rc.2", "dependencies": { "@codemirror/autocomplete": "^6.4.0", "@codemirror/commands": "^6.2.0", @@ -20781,7 +20781,7 @@ "@lezer/lr": "^1.3.1", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.44.0-rc.1", + "@prometheus-io/codemirror-promql": "0.44.0-rc.2", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", "downshift": "^7.2.0", @@ -23417,7 +23417,7 @@ "@lezer/lr": "^1.3.1", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.44.0-rc.1", + "@prometheus-io/codemirror-promql": "0.44.0-rc.2", "@testing-library/react-hooks": "^7.0.2", "@types/enzyme": "^3.10.12", "@types/flot": "0.0.32", @@ -23468,7 +23468,7 @@ "@lezer/common": "^1.0.2", "@lezer/highlight": "^1.1.3", "@lezer/lr": "^1.3.1", - "@prometheus-io/lezer-promql": "0.44.0-rc.1", + "@prometheus-io/lezer-promql": "0.44.0-rc.2", "@types/lru-cache": "^5.1.1", "isomorphic-fetch": "^3.0.0", "lru-cache": "^6.0.0", diff --git a/web/ui/react-app/package.json b/web/ui/react-app/package.json index 57e4d8120d..1e0c4ed154 100644 --- a/web/ui/react-app/package.json +++ b/web/ui/react-app/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/app", - "version": "0.44.0-rc.1", + "version": "0.44.0-rc.2", "private": true, "dependencies": { "@codemirror/autocomplete": "^6.4.0", @@ -19,7 +19,7 @@ "@lezer/common": "^1.0.2", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.44.0-rc.1", + "@prometheus-io/codemirror-promql": "0.44.0-rc.2", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", "downshift": "^7.2.0", From b1b8fd77c419795d23b97a91cfe4e1f4c8d4d30c Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Tue, 9 May 2023 10:24:55 +0000 Subject: [PATCH 149/231] docs: state that remote write is stable Since https://github.com/prometheus/docs/pull/2313 has been merged declaring remote write to be stable at version 1.0. Signed-off-by: Bryan Boreham --- docs/stability.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/stability.md b/docs/stability.md index e4cc3203b2..1fd2e51e0c 100644 --- a/docs/stability.md +++ b/docs/stability.md @@ -18,12 +18,13 @@ Things considered stable for 2.x: * Configuration file format (minus the service discovery remote read/write, see below) * Rule/alert file format * Console template syntax and semantics +* Remote write sending, per the [1.0 specification](https://prometheus.io/docs/concepts/remote_write_spec/). Things considered unstable for 2.x: * Any feature listed as experimental or subject to change, including: * The [`holt_winters` PromQL function](https://github.com/prometheus/prometheus/issues/2458) - * Remote read, remote write and the remote read endpoint + * Remote write receiving, remote read and the remote read endpoint * Server-side HTTPS and basic authentication * Service discovery integrations, with the exception of `static_configs` and `file_sd_configs` * Go APIs of packages that are part of the server From a96350f15f02607d59a7e54373963a11926815b3 Mon Sep 17 00:00:00 2001 From: Ganesh Vernekar Date: Tue, 9 May 2023 19:30:18 +0530 Subject: [PATCH 150/231] Remove codesome and add jesusvazquez in CODEOWNERS for tsdb Signed-off-by: Ganesh Vernekar --- .github/CODEOWNERS | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 75d24cdef7..b75a7896ee 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -2,6 +2,6 @@ /web/ui/module @juliusv @nexucis /storage/remote @csmarchbanks @cstyan @bwplotka @tomwilkie /discovery/kubernetes @brancz -/tsdb @codesome +/tsdb @jesusvazquez /promql @roidelapluie /cmd/promtool @dgl From 9e4e2a4a51d69fe621b1628d0618b25e3a6baa1c Mon Sep 17 00:00:00 2001 From: Robert Fratto Date: Wed, 10 May 2023 12:38:02 -0400 Subject: [PATCH 151/231] wlog: use filepath for getting checkpoint number This changes usage of path to be replaced with path/filepath, allowing for filepath.Base to properly return the base directory on systems where `/` is not the standard path separator. This resolves an issue on Windows where intermediate folders containing a `.` were incorrectly considered to be a part of the checkpoint name. Related to grafana/agent#3826. Signed-off-by: Robert Fratto --- tsdb/wlog/watcher.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tsdb/wlog/watcher.go b/tsdb/wlog/watcher.go index b0c17dcbac..31f60feab5 100644 --- a/tsdb/wlog/watcher.go +++ b/tsdb/wlog/watcher.go @@ -18,7 +18,7 @@ import ( "io" "math" "os" - "path" + "path/filepath" "strconv" "strings" "time" @@ -156,7 +156,7 @@ func NewWatcher(metrics *WatcherMetrics, readerMetrics *LiveReaderMetrics, logge writer: writer, metrics: metrics, readerMetrics: readerMetrics, - walDir: path.Join(dir, "wal"), + walDir: filepath.Join(dir, "wal"), name: name, sendExemplars: sendExemplars, sendHistograms: sendHistograms, @@ -691,7 +691,7 @@ func (w *Watcher) readCheckpoint(checkpointDir string, readFn segmentReadFn) err func checkpointNum(dir string) (int, error) { // Checkpoint dir names are in the format checkpoint.000001 // dir may contain a hidden directory, so only check the base directory - chunks := strings.Split(path.Base(dir), ".") + chunks := strings.Split(filepath.Base(dir), ".") if len(chunks) != 2 { return 0, errors.Errorf("invalid checkpoint dir string: %s", dir) } From c0f1abb574ae928662146907e0c0fbe6eca979e0 Mon Sep 17 00:00:00 2001 From: Alan Protasio Date: Wed, 10 May 2023 15:46:14 -0700 Subject: [PATCH 152/231] MatchNotRegexp optimization Signed-off-by: Alan Protasio --- tsdb/querier.go | 8 +++++ tsdb/querier_bench_test.go | 3 ++ tsdb/querier_test.go | 60 ++++++++++++++++++++++++++------------ 3 files changed, 52 insertions(+), 19 deletions(-) diff --git a/tsdb/querier.go b/tsdb/querier.go index 70f8f6de8d..8b17bb745a 100644 --- a/tsdb/querier.go +++ b/tsdb/querier.go @@ -371,6 +371,14 @@ func inversePostingsForMatcher(ix IndexReader, m *labels.Matcher) (index.Posting if m.Type == labels.MatchEqual && m.Value == "" { res = vals } else { + // Inverse of a MatchNotRegexp is MatchRegexp (double negation). + // Fast-path for set matching. + if m.Type == labels.MatchNotRegexp { + setMatches := findSetMatches(m.GetRegexString()) + if len(setMatches) > 0 { + return ix.Postings(m.Name, setMatches...) + } + } for _, val := range vals { if !m.Matches(val) { res = append(res, val) diff --git a/tsdb/querier_bench_test.go b/tsdb/querier_bench_test.go index 89758a1d31..6e5243ef11 100644 --- a/tsdb/querier_bench_test.go +++ b/tsdb/querier_bench_test.go @@ -114,6 +114,7 @@ func benchmarkPostingsForMatchers(b *testing.B, ir IndexReader) { iCharSet := labels.MustNewMatcher(labels.MatchRegexp, "i", "1[0-9]") iAlternate := labels.MustNewMatcher(labels.MatchRegexp, "i", "(1|2|3|4|5|6|20|55)") iXYZ := labels.MustNewMatcher(labels.MatchRegexp, "i", "X|Y|Z") + iNotXYZ := labels.MustNewMatcher(labels.MatchNotRegexp, "i", "X|Y|Z") cases := []struct { name string matchers []*labels.Matcher @@ -131,6 +132,7 @@ func benchmarkPostingsForMatchers(b *testing.B, ir IndexReader) { {`j=~"X.+"`, []*labels.Matcher{jXplus}}, {`i=~"(1|2|3|4|5|6|20|55)"`, []*labels.Matcher{iAlternate}}, {`i=~"X|Y|Z"`, []*labels.Matcher{iXYZ}}, + {`i!~"X|Y|Z"`, []*labels.Matcher{iNotXYZ}}, {`i=~".*"`, []*labels.Matcher{iStar}}, {`i=~"1.*"`, []*labels.Matcher{i1Star}}, {`i=~".*1"`, []*labels.Matcher{iStar1}}, @@ -146,6 +148,7 @@ func benchmarkPostingsForMatchers(b *testing.B, ir IndexReader) { {`n="1",i!="",j=~"X.+"`, []*labels.Matcher{n1, iNotEmpty, jXplus}}, {`n="1",i!="",j=~"XXX|YYY"`, []*labels.Matcher{n1, iNotEmpty, jXXXYYY}}, {`n="1",i=~"X|Y|Z",j="foo"`, []*labels.Matcher{n1, iXYZ, jFoo}}, + {`n="1",i!~"X|Y|Z",j="foo"`, []*labels.Matcher{n1, iNotXYZ, jFoo}}, {`n="1",i=~".+",j="foo"`, []*labels.Matcher{n1, iPlus, jFoo}}, {`n="1",i=~"1.+",j="foo"`, []*labels.Matcher{n1, i1Plus, jFoo}}, {`n="1",i=~".*1.*",j="foo"`, []*labels.Matcher{n1, iStar1Star, jFoo}}, diff --git a/tsdb/querier_test.go b/tsdb/querier_test.go index 5bb99a4539..9e17f5b8d8 100644 --- a/tsdb/querier_test.go +++ b/tsdb/querier_test.go @@ -1801,6 +1801,19 @@ func TestPostingsForMatchers(t *testing.T) { labels.FromStrings("n", "2.5"), }, }, + { + matchers: []*labels.Matcher{labels.MustNewMatcher(labels.MatchNotRegexp, "n", "1")}, + exp: []labels.Labels{ + labels.FromStrings("n", "2"), + labels.FromStrings("n", "2.5"), + }, + }, + { + matchers: []*labels.Matcher{labels.MustNewMatcher(labels.MatchNotRegexp, "n", "1|2.5")}, + exp: []labels.Labels{ + labels.FromStrings("n", "2"), + }, + }, { matchers: []*labels.Matcher{labels.MustNewMatcher(labels.MatchEqual, "n", "1"), labels.MustNewMatcher(labels.MatchNotRegexp, "i", "^a$")}, exp: []labels.Labels{ @@ -1890,27 +1903,36 @@ func TestPostingsForMatchers(t *testing.T) { require.NoError(t, err) for _, c := range cases { - exp := map[string]struct{}{} - for _, l := range c.exp { - exp[l.String()] = struct{}{} - } - p, err := PostingsForMatchers(ir, c.matchers...) - require.NoError(t, err) - - var builder labels.ScratchBuilder - for p.Next() { - require.NoError(t, ir.Series(p.At(), &builder, &[]chunks.Meta{})) - lbls := builder.Labels() - if _, ok := exp[lbls.String()]; !ok { - t.Errorf("Evaluating %v, unexpected result %s", c.matchers, lbls.String()) - } else { - delete(exp, lbls.String()) + name := "" + for i, matcher := range c.matchers { + if i > 0 { + name += "," } + name += matcher.String() } - require.NoError(t, p.Err()) - if len(exp) != 0 { - t.Errorf("Evaluating %v, missing results %+v", c.matchers, exp) - } + t.Run(name, func(t *testing.T) { + exp := map[string]struct{}{} + for _, l := range c.exp { + exp[l.String()] = struct{}{} + } + p, err := PostingsForMatchers(ir, c.matchers...) + require.NoError(t, err) + + var builder labels.ScratchBuilder + for p.Next() { + require.NoError(t, ir.Series(p.At(), &builder, &[]chunks.Meta{})) + lbls := builder.Labels() + if _, ok := exp[lbls.String()]; !ok { + t.Errorf("Evaluating %v, unexpected result %s", c.matchers, lbls.String()) + } else { + delete(exp, lbls.String()) + } + } + require.NoError(t, p.Err()) + if len(exp) != 0 { + t.Errorf("Evaluating %v, missing results %+v", c.matchers, exp) + } + }) } } From 2f3561971078259fedb2425f1cc218b6cecd84ce Mon Sep 17 00:00:00 2001 From: Mickael Carl Date: Wed, 3 May 2023 21:46:13 +0100 Subject: [PATCH 153/231] discovery/kubernetes: attach node labels when the endpoints TargetRef's kind are Node Signed-off-by: Mickael Carl --- discovery/kubernetes/endpoints.go | 7 +- discovery/kubernetes/endpoints_test.go | 102 ++++++++++-- discovery/kubernetes/endpointslice.go | 6 +- discovery/kubernetes/endpointslice_test.go | 178 ++++++++++++++++++++- discovery/kubernetes/kubernetes.go | 38 +++-- 5 files changed, 302 insertions(+), 29 deletions(-) diff --git a/discovery/kubernetes/endpoints.go b/discovery/kubernetes/endpoints.go index 27742ab464..2413dab455 100644 --- a/discovery/kubernetes/endpoints.go +++ b/discovery/kubernetes/endpoints.go @@ -305,7 +305,11 @@ func (e *Endpoints) buildEndpoints(eps *apiv1.Endpoints) *targetgroup.Group { } if e.withNodeMetadata { - target = addNodeLabels(target, e.nodeInf, e.logger, addr.NodeName) + if addr.NodeName != nil { + target = addNodeLabels(target, e.nodeInf, e.logger, addr.NodeName) + } else if addr.TargetRef != nil && addr.TargetRef.Kind == "Node" { + target = addNodeLabels(target, e.nodeInf, e.logger, &addr.TargetRef.Name) + } } pod := e.resolvePodRef(addr.TargetRef) @@ -466,5 +470,6 @@ func addNodeLabels(tg model.LabelSet, nodeInf cache.SharedInformer, logger log.L nodeLabelset[model.LabelName(nodeLabelPrefix+ln)] = lv(v) nodeLabelset[model.LabelName(nodeLabelPresentPrefix+ln)] = presentValue } + return tg.Merge(nodeLabelset) } diff --git a/discovery/kubernetes/endpoints_test.go b/discovery/kubernetes/endpoints_test.go index 91b1b0c676..5aa58bdc49 100644 --- a/discovery/kubernetes/endpoints_test.go +++ b/discovery/kubernetes/endpoints_test.go @@ -69,6 +69,24 @@ func makeEndpoints() *v1.Endpoints { }, }, }, + { + Addresses: []v1.EndpointAddress{ + { + IP: "6.7.8.9", + TargetRef: &v1.ObjectReference{ + Kind: "Node", + Name: "barbaz", + }, + }, + }, + Ports: []v1.EndpointPort{ + { + Name: "testport", + Port: 9002, + Protocol: v1.ProtocolTCP, + }, + }, + }, }, } } @@ -106,6 +124,14 @@ func TestEndpointsDiscoveryBeforeRun(t *testing.T) { "__meta_kubernetes_endpoint_port_protocol": "TCP", "__meta_kubernetes_endpoint_ready": "false", }, + { + "__address__": "6.7.8.9:9002", + "__meta_kubernetes_endpoint_address_target_kind": "Node", + "__meta_kubernetes_endpoint_address_target_name": "barbaz", + "__meta_kubernetes_endpoint_port_name": "testport", + "__meta_kubernetes_endpoint_port_protocol": "TCP", + "__meta_kubernetes_endpoint_ready": "true", + }, }, Labels: model.LabelSet{ "__meta_kubernetes_namespace": "default", @@ -398,6 +424,14 @@ func TestEndpointsDiscoveryWithService(t *testing.T) { "__meta_kubernetes_endpoint_port_protocol": "TCP", "__meta_kubernetes_endpoint_ready": "false", }, + { + "__address__": "6.7.8.9:9002", + "__meta_kubernetes_endpoint_address_target_kind": "Node", + "__meta_kubernetes_endpoint_address_target_name": "barbaz", + "__meta_kubernetes_endpoint_port_name": "testport", + "__meta_kubernetes_endpoint_port_protocol": "TCP", + "__meta_kubernetes_endpoint_ready": "true", + }, }, Labels: model.LabelSet{ "__meta_kubernetes_namespace": "default", @@ -466,6 +500,14 @@ func TestEndpointsDiscoveryWithServiceUpdate(t *testing.T) { "__meta_kubernetes_endpoint_port_protocol": "TCP", "__meta_kubernetes_endpoint_ready": "false", }, + { + "__address__": "6.7.8.9:9002", + "__meta_kubernetes_endpoint_address_target_kind": "Node", + "__meta_kubernetes_endpoint_address_target_name": "barbaz", + "__meta_kubernetes_endpoint_port_name": "testport", + "__meta_kubernetes_endpoint_port_protocol": "TCP", + "__meta_kubernetes_endpoint_ready": "true", + }, }, Labels: model.LabelSet{ "__meta_kubernetes_namespace": "default", @@ -484,8 +526,10 @@ func TestEndpointsDiscoveryWithServiceUpdate(t *testing.T) { func TestEndpointsDiscoveryWithNodeMetadata(t *testing.T) { metadataConfig := AttachMetadataConfig{Node: true} - nodeLabels := map[string]string{"az": "us-east1"} - node := makeNode("foobar", "", "", nodeLabels, nil) + nodeLabels1 := map[string]string{"az": "us-east1"} + nodeLabels2 := map[string]string{"az": "us-west2"} + node1 := makeNode("foobar", "", "", nodeLabels1, nil) + node2 := makeNode("barbaz", "", "", nodeLabels2, nil) svc := &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name: "testendpoints", @@ -495,7 +539,7 @@ func TestEndpointsDiscoveryWithNodeMetadata(t *testing.T) { }, }, } - n, _ := makeDiscoveryWithMetadata(RoleEndpoint, NamespaceDiscovery{}, metadataConfig, makeEndpoints(), svc, node) + n, _ := makeDiscoveryWithMetadata(RoleEndpoint, NamespaceDiscovery{}, metadataConfig, makeEndpoints(), svc, node1, node2) k8sDiscoveryTest{ discovery: n, @@ -526,6 +570,17 @@ func TestEndpointsDiscoveryWithNodeMetadata(t *testing.T) { "__meta_kubernetes_endpoint_port_protocol": "TCP", "__meta_kubernetes_endpoint_ready": "false", }, + { + "__address__": "6.7.8.9:9002", + "__meta_kubernetes_endpoint_address_target_kind": "Node", + "__meta_kubernetes_endpoint_address_target_name": "barbaz", + "__meta_kubernetes_endpoint_port_name": "testport", + "__meta_kubernetes_endpoint_port_protocol": "TCP", + "__meta_kubernetes_endpoint_ready": "true", + "__meta_kubernetes_node_label_az": "us-west2", + "__meta_kubernetes_node_labelpresent_az": "true", + "__meta_kubernetes_node_name": "barbaz", + }, }, Labels: model.LabelSet{ "__meta_kubernetes_namespace": "default", @@ -541,8 +596,10 @@ func TestEndpointsDiscoveryWithNodeMetadata(t *testing.T) { } func TestEndpointsDiscoveryWithUpdatedNodeMetadata(t *testing.T) { - nodeLabels := map[string]string{"az": "us-east1"} - nodes := makeNode("foobar", "", "", nodeLabels, nil) + nodeLabels1 := map[string]string{"az": "us-east1"} + nodeLabels2 := map[string]string{"az": "us-west2"} + node1 := makeNode("foobar", "", "", nodeLabels1, nil) + node2 := makeNode("barbaz", "", "", nodeLabels2, nil) metadataConfig := AttachMetadataConfig{Node: true} svc := &v1.Service{ ObjectMeta: metav1.ObjectMeta{ @@ -553,13 +610,13 @@ func TestEndpointsDiscoveryWithUpdatedNodeMetadata(t *testing.T) { }, }, } - n, c := makeDiscoveryWithMetadata(RoleEndpoint, NamespaceDiscovery{}, metadataConfig, makeEndpoints(), nodes, svc) + n, c := makeDiscoveryWithMetadata(RoleEndpoint, NamespaceDiscovery{}, metadataConfig, makeEndpoints(), node1, node2, svc) k8sDiscoveryTest{ discovery: n, afterStart: func() { - nodes.Labels["az"] = "eu-central1" - c.CoreV1().Nodes().Update(context.Background(), nodes, metav1.UpdateOptions{}) + node1.Labels["az"] = "eu-central1" + c.CoreV1().Nodes().Update(context.Background(), node1, metav1.UpdateOptions{}) }, expectedMaxItems: 2, expectedRes: map[string]*targetgroup.Group{ @@ -572,7 +629,7 @@ func TestEndpointsDiscoveryWithUpdatedNodeMetadata(t *testing.T) { "__meta_kubernetes_endpoint_port_name": "testport", "__meta_kubernetes_endpoint_port_protocol": "TCP", "__meta_kubernetes_endpoint_ready": "true", - "__meta_kubernetes_node_label_az": "eu-central1", + "__meta_kubernetes_node_label_az": "us-east1", "__meta_kubernetes_node_labelpresent_az": "true", "__meta_kubernetes_node_name": "foobar", }, @@ -588,6 +645,17 @@ func TestEndpointsDiscoveryWithUpdatedNodeMetadata(t *testing.T) { "__meta_kubernetes_endpoint_port_protocol": "TCP", "__meta_kubernetes_endpoint_ready": "false", }, + { + "__address__": "6.7.8.9:9002", + "__meta_kubernetes_endpoint_address_target_kind": "Node", + "__meta_kubernetes_endpoint_address_target_name": "barbaz", + "__meta_kubernetes_endpoint_port_name": "testport", + "__meta_kubernetes_endpoint_port_protocol": "TCP", + "__meta_kubernetes_endpoint_ready": "true", + "__meta_kubernetes_node_label_az": "us-west2", + "__meta_kubernetes_node_labelpresent_az": "true", + "__meta_kubernetes_node_name": "barbaz", + }, }, Labels: model.LabelSet{ "__meta_kubernetes_namespace": "default", @@ -699,6 +767,14 @@ func TestEndpointsDiscoveryNamespaces(t *testing.T) { "__meta_kubernetes_endpoint_port_protocol": "TCP", "__meta_kubernetes_endpoint_ready": "false", }, + { + "__address__": "6.7.8.9:9002", + "__meta_kubernetes_endpoint_address_target_kind": "Node", + "__meta_kubernetes_endpoint_address_target_name": "barbaz", + "__meta_kubernetes_endpoint_port_name": "testport", + "__meta_kubernetes_endpoint_port_protocol": "TCP", + "__meta_kubernetes_endpoint_ready": "true", + }, }, Labels: model.LabelSet{ "__meta_kubernetes_namespace": "ns1", @@ -815,6 +891,14 @@ func TestEndpointsDiscoveryOwnNamespace(t *testing.T) { "__meta_kubernetes_endpoint_port_protocol": "TCP", "__meta_kubernetes_endpoint_ready": "false", }, + { + "__address__": "6.7.8.9:9002", + "__meta_kubernetes_endpoint_address_target_kind": "Node", + "__meta_kubernetes_endpoint_address_target_name": "barbaz", + "__meta_kubernetes_endpoint_port_name": "testport", + "__meta_kubernetes_endpoint_port_protocol": "TCP", + "__meta_kubernetes_endpoint_ready": "true", + }, }, Labels: model.LabelSet{ "__meta_kubernetes_namespace": "own-ns", diff --git a/discovery/kubernetes/endpointslice.go b/discovery/kubernetes/endpointslice.go index 841b7d4f61..c7df642525 100644 --- a/discovery/kubernetes/endpointslice.go +++ b/discovery/kubernetes/endpointslice.go @@ -339,7 +339,11 @@ func (e *EndpointSlice) buildEndpointSlice(eps endpointSliceAdaptor) *targetgrou } if e.withNodeMetadata { - target = addNodeLabels(target, e.nodeInf, e.logger, ep.nodename()) + if ep.targetRef() != nil && ep.targetRef().Kind == "Node" { + target = addNodeLabels(target, e.nodeInf, e.logger, &ep.targetRef().Name) + } else { + target = addNodeLabels(target, e.nodeInf, e.logger, ep.nodename()) + } } pod := e.resolvePodRef(ep.targetRef()) diff --git a/discovery/kubernetes/endpointslice_test.go b/discovery/kubernetes/endpointslice_test.go index f4076b943d..8104e3db3b 100644 --- a/discovery/kubernetes/endpointslice_test.go +++ b/discovery/kubernetes/endpointslice_test.go @@ -90,6 +90,17 @@ func makeEndpointSliceV1() *v1.EndpointSlice { Serving: boolptr(true), Terminating: boolptr(true), }, + }, { + Addresses: []string{"4.5.6.7"}, + Conditions: v1.EndpointConditions{ + Ready: boolptr(true), + Serving: boolptr(true), + Terminating: boolptr(false), + }, + TargetRef: &corev1.ObjectReference{ + Kind: "Node", + Name: "barbaz", + }, }, }, } @@ -130,6 +141,17 @@ func makeEndpointSliceV1beta1() *v1beta1.EndpointSlice { Serving: boolptr(true), Terminating: boolptr(true), }, + }, { + Addresses: []string{"4.5.6.7"}, + Conditions: v1beta1.EndpointConditions{ + Ready: boolptr(true), + Serving: boolptr(true), + Terminating: boolptr(false), + }, + TargetRef: &corev1.ObjectReference{ + Kind: "Node", + Name: "barbaz", + }, }, }, } @@ -183,6 +205,18 @@ func TestEndpointSliceDiscoveryBeforeRun(t *testing.T) { "__meta_kubernetes_endpointslice_port_name": "testport", "__meta_kubernetes_endpointslice_port_protocol": "TCP", }, + { + "__address__": "4.5.6.7:9000", + "__meta_kubernetes_endpointslice_address_target_kind": "Node", + "__meta_kubernetes_endpointslice_address_target_name": "barbaz", + "__meta_kubernetes_endpointslice_endpoint_conditions_ready": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_serving": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_terminating": "false", + "__meta_kubernetes_endpointslice_port": "9000", + "__meta_kubernetes_endpointslice_port_app_protocol": "http", + "__meta_kubernetes_endpointslice_port_name": "testport", + "__meta_kubernetes_endpointslice_port_protocol": "TCP", + }, }, Labels: model.LabelSet{ "__meta_kubernetes_endpointslice_address_type": "IPv4", @@ -233,6 +267,17 @@ func TestEndpointSliceDiscoveryBeforeRunV1beta1(t *testing.T) { "__meta_kubernetes_endpointslice_port_name": "testport", "__meta_kubernetes_endpointslice_port_protocol": "TCP", }, + { + "__address__": "4.5.6.7:9000", + "__meta_kubernetes_endpointslice_address_target_kind": "Node", + "__meta_kubernetes_endpointslice_address_target_name": "barbaz", + "__meta_kubernetes_endpointslice_endpoint_conditions_ready": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_serving": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_terminating": "false", + "__meta_kubernetes_endpointslice_port": "9000", + "__meta_kubernetes_endpointslice_port_name": "testport", + "__meta_kubernetes_endpointslice_port_protocol": "TCP", + }, }, Labels: model.LabelSet{ "__meta_kubernetes_endpointslice_address_type": "IPv4", @@ -419,6 +464,18 @@ func TestEndpointSliceDiscoveryDelete(t *testing.T) { "__meta_kubernetes_endpointslice_port_name": "testport", "__meta_kubernetes_endpointslice_port_protocol": "TCP", }, + { + "__address__": "4.5.6.7:9000", + "__meta_kubernetes_endpointslice_address_target_kind": "Node", + "__meta_kubernetes_endpointslice_address_target_name": "barbaz", + "__meta_kubernetes_endpointslice_endpoint_conditions_ready": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_serving": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_terminating": "false", + "__meta_kubernetes_endpointslice_port": "9000", + "__meta_kubernetes_endpointslice_port_app_protocol": "http", + "__meta_kubernetes_endpointslice_port_name": "testport", + "__meta_kubernetes_endpointslice_port_protocol": "TCP", + }, }, Labels: map[model.LabelName]model.LabelValue{ "__meta_kubernetes_endpointslice_address_type": "IPv4", @@ -503,6 +560,18 @@ func TestEndpointSliceDiscoveryUpdate(t *testing.T) { "__meta_kubernetes_endpointslice_port_name": "testport", "__meta_kubernetes_endpointslice_port_protocol": "TCP", }, + { + "__address__": "4.5.6.7:9000", + "__meta_kubernetes_endpointslice_address_target_kind": "Node", + "__meta_kubernetes_endpointslice_address_target_name": "barbaz", + "__meta_kubernetes_endpointslice_endpoint_conditions_ready": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_serving": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_terminating": "false", + "__meta_kubernetes_endpointslice_port": "9000", + "__meta_kubernetes_endpointslice_port_app_protocol": "http", + "__meta_kubernetes_endpointslice_port_name": "testport", + "__meta_kubernetes_endpointslice_port_protocol": "TCP", + }, }, Labels: model.LabelSet{ "__meta_kubernetes_endpointslice_address_type": "IPv4", @@ -576,6 +645,18 @@ func TestEndpointSliceDiscoveryEmptyEndpoints(t *testing.T) { "__meta_kubernetes_endpointslice_port_name": "testport", "__meta_kubernetes_endpointslice_port_protocol": "TCP", }, + { + "__address__": "4.5.6.7:9000", + "__meta_kubernetes_endpointslice_address_target_kind": "Node", + "__meta_kubernetes_endpointslice_address_target_name": "barbaz", + "__meta_kubernetes_endpointslice_endpoint_conditions_ready": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_serving": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_terminating": "false", + "__meta_kubernetes_endpointslice_port": "9000", + "__meta_kubernetes_endpointslice_port_app_protocol": "http", + "__meta_kubernetes_endpointslice_port_name": "testport", + "__meta_kubernetes_endpointslice_port_protocol": "TCP", + }, }, Labels: model.LabelSet{ "__meta_kubernetes_endpointslice_address_type": "IPv4", @@ -644,6 +725,18 @@ func TestEndpointSliceDiscoveryWithService(t *testing.T) { "__meta_kubernetes_endpointslice_port_name": "testport", "__meta_kubernetes_endpointslice_port_protocol": "TCP", }, + { + "__address__": "4.5.6.7:9000", + "__meta_kubernetes_endpointslice_address_target_kind": "Node", + "__meta_kubernetes_endpointslice_address_target_name": "barbaz", + "__meta_kubernetes_endpointslice_endpoint_conditions_ready": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_serving": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_terminating": "false", + "__meta_kubernetes_endpointslice_port": "9000", + "__meta_kubernetes_endpointslice_port_app_protocol": "http", + "__meta_kubernetes_endpointslice_port_name": "testport", + "__meta_kubernetes_endpointslice_port_protocol": "TCP", + }, }, Labels: model.LabelSet{ "__meta_kubernetes_endpointslice_address_type": "IPv4", @@ -728,6 +821,18 @@ func TestEndpointSliceDiscoveryWithServiceUpdate(t *testing.T) { "__meta_kubernetes_endpointslice_port_protocol": "TCP", "__meta_kubernetes_endpointslice_port_app_protocol": "http", }, + { + "__address__": "4.5.6.7:9000", + "__meta_kubernetes_endpointslice_address_target_kind": "Node", + "__meta_kubernetes_endpointslice_address_target_name": "barbaz", + "__meta_kubernetes_endpointslice_endpoint_conditions_ready": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_serving": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_terminating": "false", + "__meta_kubernetes_endpointslice_port": "9000", + "__meta_kubernetes_endpointslice_port_app_protocol": "http", + "__meta_kubernetes_endpointslice_port_name": "testport", + "__meta_kubernetes_endpointslice_port_protocol": "TCP", + }, }, Labels: model.LabelSet{ "__meta_kubernetes_endpointslice_address_type": "IPv4", @@ -747,7 +852,8 @@ func TestEndpointSliceDiscoveryWithServiceUpdate(t *testing.T) { func TestEndpointsSlicesDiscoveryWithNodeMetadata(t *testing.T) { metadataConfig := AttachMetadataConfig{Node: true} - nodeLabels := map[string]string{"az": "us-east1"} + nodeLabels1 := map[string]string{"az": "us-east1"} + nodeLabels2 := map[string]string{"az": "us-west2"} svc := &corev1.Service{ ObjectMeta: metav1.ObjectMeta{ Name: "testendpoints", @@ -757,7 +863,7 @@ func TestEndpointsSlicesDiscoveryWithNodeMetadata(t *testing.T) { }, }, } - objs := []runtime.Object{makeEndpointSliceV1(), makeNode("foobar", "", "", nodeLabels, nil), svc} + objs := []runtime.Object{makeEndpointSliceV1(), makeNode("foobar", "", "", nodeLabels1, nil), makeNode("barbaz", "", "", nodeLabels2, nil), svc} n, _ := makeDiscoveryWithMetadata(RoleEndpointSlice, NamespaceDiscovery{}, metadataConfig, objs...) k8sDiscoveryTest{ @@ -804,6 +910,21 @@ func TestEndpointsSlicesDiscoveryWithNodeMetadata(t *testing.T) { "__meta_kubernetes_endpointslice_port_name": "testport", "__meta_kubernetes_endpointslice_port_protocol": "TCP", }, + { + "__address__": "4.5.6.7:9000", + "__meta_kubernetes_endpointslice_address_target_kind": "Node", + "__meta_kubernetes_endpointslice_address_target_name": "barbaz", + "__meta_kubernetes_endpointslice_endpoint_conditions_ready": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_serving": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_terminating": "false", + "__meta_kubernetes_endpointslice_port": "9000", + "__meta_kubernetes_endpointslice_port_app_protocol": "http", + "__meta_kubernetes_endpointslice_port_name": "testport", + "__meta_kubernetes_endpointslice_port_protocol": "TCP", + "__meta_kubernetes_node_label_az": "us-west2", + "__meta_kubernetes_node_labelpresent_az": "true", + "__meta_kubernetes_node_name": "barbaz", + }, }, Labels: model.LabelSet{ "__meta_kubernetes_endpointslice_address_type": "IPv4", @@ -821,7 +942,8 @@ func TestEndpointsSlicesDiscoveryWithNodeMetadata(t *testing.T) { func TestEndpointsSlicesDiscoveryWithUpdatedNodeMetadata(t *testing.T) { metadataConfig := AttachMetadataConfig{Node: true} - nodeLabels := map[string]string{"az": "us-east1"} + nodeLabels1 := map[string]string{"az": "us-east1"} + nodeLabels2 := map[string]string{"az": "us-west2"} svc := &corev1.Service{ ObjectMeta: metav1.ObjectMeta{ Name: "testendpoints", @@ -831,16 +953,17 @@ func TestEndpointsSlicesDiscoveryWithUpdatedNodeMetadata(t *testing.T) { }, }, } - node := makeNode("foobar", "", "", nodeLabels, nil) - objs := []runtime.Object{makeEndpointSliceV1(), node, svc} + node1 := makeNode("foobar", "", "", nodeLabels1, nil) + node2 := makeNode("barbaz", "", "", nodeLabels2, nil) + objs := []runtime.Object{makeEndpointSliceV1(), node1, node2, svc} n, c := makeDiscoveryWithMetadata(RoleEndpointSlice, NamespaceDiscovery{}, metadataConfig, objs...) k8sDiscoveryTest{ discovery: n, expectedMaxItems: 2, afterStart: func() { - node.Labels["az"] = "us-central1" - c.CoreV1().Nodes().Update(context.Background(), node, metav1.UpdateOptions{}) + node1.Labels["az"] = "us-central1" + c.CoreV1().Nodes().Update(context.Background(), node1, metav1.UpdateOptions{}) }, expectedRes: map[string]*targetgroup.Group{ "endpointslice/default/testendpoints": { @@ -859,7 +982,7 @@ func TestEndpointsSlicesDiscoveryWithUpdatedNodeMetadata(t *testing.T) { "__meta_kubernetes_endpointslice_port_app_protocol": "http", "__meta_kubernetes_endpointslice_port_name": "testport", "__meta_kubernetes_endpointslice_port_protocol": "TCP", - "__meta_kubernetes_node_label_az": "us-central1", + "__meta_kubernetes_node_label_az": "us-east1", "__meta_kubernetes_node_labelpresent_az": "true", "__meta_kubernetes_node_name": "foobar", }, @@ -883,6 +1006,21 @@ func TestEndpointsSlicesDiscoveryWithUpdatedNodeMetadata(t *testing.T) { "__meta_kubernetes_endpointslice_port_name": "testport", "__meta_kubernetes_endpointslice_port_protocol": "TCP", }, + { + "__address__": "4.5.6.7:9000", + "__meta_kubernetes_endpointslice_address_target_kind": "Node", + "__meta_kubernetes_endpointslice_address_target_name": "barbaz", + "__meta_kubernetes_endpointslice_endpoint_conditions_ready": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_serving": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_terminating": "false", + "__meta_kubernetes_endpointslice_port": "9000", + "__meta_kubernetes_endpointslice_port_app_protocol": "http", + "__meta_kubernetes_endpointslice_port_name": "testport", + "__meta_kubernetes_endpointslice_port_protocol": "TCP", + "__meta_kubernetes_node_label_az": "us-west2", + "__meta_kubernetes_node_labelpresent_az": "true", + "__meta_kubernetes_node_name": "barbaz", + }, }, Labels: model.LabelSet{ "__meta_kubernetes_endpointslice_address_type": "IPv4", @@ -1007,6 +1145,18 @@ func TestEndpointSliceDiscoveryNamespaces(t *testing.T) { "__meta_kubernetes_endpointslice_port_protocol": "TCP", "__meta_kubernetes_endpointslice_port_app_protocol": "http", }, + { + "__address__": "4.5.6.7:9000", + "__meta_kubernetes_endpointslice_address_target_kind": "Node", + "__meta_kubernetes_endpointslice_address_target_name": "barbaz", + "__meta_kubernetes_endpointslice_endpoint_conditions_ready": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_serving": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_terminating": "false", + "__meta_kubernetes_endpointslice_port": "9000", + "__meta_kubernetes_endpointslice_port_app_protocol": "http", + "__meta_kubernetes_endpointslice_port_name": "testport", + "__meta_kubernetes_endpointslice_port_protocol": "TCP", + }, }, Labels: model.LabelSet{ "__meta_kubernetes_endpointslice_address_type": "IPv4", @@ -1139,6 +1289,18 @@ func TestEndpointSliceDiscoveryOwnNamespace(t *testing.T) { "__meta_kubernetes_endpointslice_port_protocol": "TCP", "__meta_kubernetes_endpointslice_port_app_protocol": "http", }, + { + "__address__": "4.5.6.7:9000", + "__meta_kubernetes_endpointslice_address_target_kind": "Node", + "__meta_kubernetes_endpointslice_address_target_name": "barbaz", + "__meta_kubernetes_endpointslice_endpoint_conditions_ready": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_serving": "true", + "__meta_kubernetes_endpointslice_endpoint_conditions_terminating": "false", + "__meta_kubernetes_endpointslice_port": "9000", + "__meta_kubernetes_endpointslice_port_app_protocol": "http", + "__meta_kubernetes_endpointslice_port_name": "testport", + "__meta_kubernetes_endpointslice_port_protocol": "TCP", + }, }, Labels: model.LabelSet{ "__meta_kubernetes_endpointslice_address_type": "IPv4", diff --git a/discovery/kubernetes/kubernetes.go b/discovery/kubernetes/kubernetes.go index a44bd513ce..e87a1c9b24 100644 --- a/discovery/kubernetes/kubernetes.go +++ b/discovery/kubernetes/kubernetes.go @@ -761,15 +761,21 @@ func (d *Discovery) newEndpointsByNodeInformer(plw *cache.ListWatch) cache.Share indexers[nodeIndex] = func(obj interface{}) ([]string, error) { e, ok := obj.(*apiv1.Endpoints) if !ok { - return nil, fmt.Errorf("object is not a pod") + return nil, fmt.Errorf("object is not endpoints") } var nodes []string for _, target := range e.Subsets { for _, addr := range target.Addresses { - if addr.NodeName == nil { - continue + if addr.TargetRef != nil { + switch addr.TargetRef.Kind { + case "Pod": + if addr.NodeName != nil { + nodes = append(nodes, *addr.NodeName) + } + case "Node": + nodes = append(nodes, addr.TargetRef.Name) + } } - nodes = append(nodes, *addr.NodeName) } } return nodes, nil @@ -789,17 +795,29 @@ func (d *Discovery) newEndpointSlicesByNodeInformer(plw *cache.ListWatch, object switch e := obj.(type) { case *disv1.EndpointSlice: for _, target := range e.Endpoints { - if target.NodeName == nil { - continue + if target.TargetRef != nil { + switch target.TargetRef.Kind { + case "Pod": + if target.NodeName != nil { + nodes = append(nodes, *target.NodeName) + } + case "Node": + nodes = append(nodes, target.TargetRef.Name) + } } - nodes = append(nodes, *target.NodeName) } case *disv1beta1.EndpointSlice: for _, target := range e.Endpoints { - if target.NodeName == nil { - continue + if target.TargetRef != nil { + switch target.TargetRef.Kind { + case "Pod": + if target.NodeName != nil { + nodes = append(nodes, *target.NodeName) + } + case "Node": + nodes = append(nodes, target.TargetRef.Name) + } } - nodes = append(nodes, *target.NodeName) } default: return nil, fmt.Errorf("object is not an endpointslice") From 9e500345f39ce310a96650b8500e825012fa579f Mon Sep 17 00:00:00 2001 From: beorn7 Date: Thu, 11 May 2023 01:59:21 +0200 Subject: [PATCH 154/231] textparse/scrape: Add option to scrape both classic and native histograms So far, if a target exposes a histogram with both classic and native buckets, a native-histogram enabled Prometheus would ignore the classic buckets. With the new scrape config option `scrape_classic_histograms` set, both buckets will be ingested, creating all the series of a classic histogram in parallel to the native histogram series. For example, a histogram `foo` would create a native histogram series `foo` and classic series called `foo_sum`, `foo_count`, and `foo_bucket`. This feature can be used in a migration strategy from classic to native histograms, where it is desired to have a transition period during which both native and classic histograms are present. Note that two bugs in classic histogram parsing were found and fixed as a byproduct of testing the new feature: 1. Series created from classic _gauge_ histograms didn't get the _sum/_count/_bucket prefix set. 2. Values of classic _float_ histograms weren't parsed properly. Signed-off-by: beorn7 --- config/config.go | 17 +- docs/configuration/configuration.md | 4 + model/textparse/interface.go | 6 +- model/textparse/interface_test.go | 2 +- model/textparse/protobufparse.go | 49 +- model/textparse/protobufparse_test.go | 1393 ++++++++++++++++++------- promql/fuzz.go | 2 +- scrape/scrape.go | 123 ++- scrape/scrape_test.go | 36 +- web/federate_test.go | 2 +- 10 files changed, 1148 insertions(+), 486 deletions(-) diff --git a/config/config.go b/config/config.go index 1b6a6cf6b6..d0ba03ab29 100644 --- a/config/config.go +++ b/config/config.go @@ -146,13 +146,14 @@ var ( // DefaultScrapeConfig is the default scrape configuration. DefaultScrapeConfig = ScrapeConfig{ - // ScrapeTimeout and ScrapeInterval default to the - // configured globals. - MetricsPath: "/metrics", - Scheme: "http", - HonorLabels: false, - HonorTimestamps: true, - HTTPClientConfig: config.DefaultHTTPClientConfig, + // ScrapeTimeout and ScrapeInterval default to the configured + // globals. + ScrapeClassicHistograms: false, + MetricsPath: "/metrics", + Scheme: "http", + HonorLabels: false, + HonorTimestamps: true, + HTTPClientConfig: config.DefaultHTTPClientConfig, } // DefaultAlertmanagerConfig is the default alertmanager configuration. @@ -467,6 +468,8 @@ type ScrapeConfig struct { ScrapeInterval model.Duration `yaml:"scrape_interval,omitempty"` // The timeout for scraping targets of this config. ScrapeTimeout model.Duration `yaml:"scrape_timeout,omitempty"` + // Whether to scrape a classic histogram that is also exposed as a native histogram. + ScrapeClassicHistograms bool `yaml:"scrape_classic_histograms,omitempty"` // The HTTP resource path on which to fetch metrics from targets. MetricsPath string `yaml:"metrics_path,omitempty"` // The URL scheme with which to fetch metrics from targets. diff --git a/docs/configuration/configuration.md b/docs/configuration/configuration.md index 0a8c4a5cdf..c74c9d478f 100644 --- a/docs/configuration/configuration.md +++ b/docs/configuration/configuration.md @@ -134,6 +134,10 @@ job_name: # Per-scrape timeout when scraping this job. [ scrape_timeout: | default = ] +# Whether to scrape a classic histogram that is also exposed as a native +# histogram (has no effect without --enable-feature=native-histograms). +[ scrape_classic_histograms: | default = false ] + # The HTTP resource path on which to fetch metrics from targets. [ metrics_path: | default = /metrics ] diff --git a/model/textparse/interface.go b/model/textparse/interface.go index 9efd942e83..efa581410f 100644 --- a/model/textparse/interface.go +++ b/model/textparse/interface.go @@ -71,7 +71,7 @@ type Parser interface { // // This function always returns a valid parser, but might additionally // return an error if the content type cannot be parsed. -func New(b []byte, contentType string) (Parser, error) { +func New(b []byte, contentType string, parseClassicHistograms bool) (Parser, error) { if contentType == "" { return NewPromParser(b), nil } @@ -84,7 +84,7 @@ func New(b []byte, contentType string) (Parser, error) { case "application/openmetrics-text": return NewOpenMetricsParser(b), nil case "application/vnd.google.protobuf": - return NewProtobufParser(b), nil + return NewProtobufParser(b, parseClassicHistograms), nil default: return NewPromParser(b), nil } @@ -100,7 +100,7 @@ const ( EntrySeries Entry = 2 // A series with a simple float64 as value. EntryComment Entry = 3 EntryUnit Entry = 4 - EntryHistogram Entry = 5 // A series with a sparse histogram as a value. + EntryHistogram Entry = 5 // A series with a native histogram as a value. ) // MetricType represents metric type values. diff --git a/model/textparse/interface_test.go b/model/textparse/interface_test.go index d94467d4db..de140d6819 100644 --- a/model/textparse/interface_test.go +++ b/model/textparse/interface_test.go @@ -91,7 +91,7 @@ func TestNewParser(t *testing.T) { tt := tt // Copy to local variable before going parallel. t.Parallel() - p, err := New([]byte{}, tt.contentType) + p, err := New([]byte{}, tt.contentType, false) tt.validateParser(t, p) if tt.err == "" { require.NoError(t, err) diff --git a/model/textparse/protobufparse.go b/model/textparse/protobufparse.go index eca145955e..b831251ad0 100644 --- a/model/textparse/protobufparse.go +++ b/model/textparse/protobufparse.go @@ -52,8 +52,10 @@ type ProtobufParser struct { // fieldPos is the position within a Summary or (legacy) Histogram. -2 // is the count. -1 is the sum. Otherwise it is the index within // quantiles/buckets. - fieldPos int - fieldsDone bool // true if no more fields of a Summary or (legacy) Histogram to be processed. + fieldPos int + fieldsDone bool // true if no more fields of a Summary or (legacy) Histogram to be processed. + redoClassic bool // true after parsing a native histogram if we need to parse it again as a classit histogram. + // state is marked by the entry we are processing. EntryInvalid implies // that we have to decode the next MetricFamily. state Entry @@ -62,17 +64,22 @@ type ProtobufParser struct { mf *dto.MetricFamily + // Wether to also parse a classic histogram that is also present as a + // native histogram. + parseClassicHistograms bool + // The following are just shenanigans to satisfy the Parser interface. metricBytes *bytes.Buffer // A somewhat fluid representation of the current metric. } // NewProtobufParser returns a parser for the payload in the byte slice. -func NewProtobufParser(b []byte) Parser { +func NewProtobufParser(b []byte, parseClassicHistograms bool) Parser { return &ProtobufParser{ - in: b, - state: EntryInvalid, - mf: &dto.MetricFamily{}, - metricBytes: &bytes.Buffer{}, + in: b, + state: EntryInvalid, + mf: &dto.MetricFamily{}, + metricBytes: &bytes.Buffer{}, + parseClassicHistograms: parseClassicHistograms, } } @@ -106,19 +113,28 @@ func (p *ProtobufParser) Series() ([]byte, *int64, float64) { v = s.GetQuantile()[p.fieldPos].GetValue() } case dto.MetricType_HISTOGRAM, dto.MetricType_GAUGE_HISTOGRAM: - // This should only happen for a legacy histogram. + // This should only happen for a classic histogram. h := m.GetHistogram() switch p.fieldPos { case -2: - v = float64(h.GetSampleCount()) + v = h.GetSampleCountFloat() + if v == 0 { + v = float64(h.GetSampleCount()) + } case -1: v = h.GetSampleSum() default: bb := h.GetBucket() if p.fieldPos >= len(bb) { - v = float64(h.GetSampleCount()) + v = h.GetSampleCountFloat() + if v == 0 { + v = float64(h.GetSampleCount()) + } } else { - v = float64(bb[p.fieldPos].GetCumulativeCount()) + v = bb[p.fieldPos].GetCumulativeCountFloat() + if v == 0 { + v = float64(bb[p.fieldPos].GetCumulativeCount()) + } } } default: @@ -149,6 +165,9 @@ func (p *ProtobufParser) Histogram() ([]byte, *int64, *histogram.Histogram, *his ts = m.GetTimestampMs() h = m.GetHistogram() ) + if p.parseClassicHistograms && len(h.GetBucket()) > 0 { + p.redoClassic = true + } if h.GetSampleCountFloat() > 0 || h.GetZeroCountFloat() > 0 { // It is a float histogram. fh := histogram.FloatHistogram{ @@ -376,6 +395,12 @@ func (p *ProtobufParser) Next() (Entry, error) { return EntryInvalid, err } case EntryHistogram, EntrySeries: + if p.redoClassic { + p.redoClassic = false + p.state = EntrySeries + p.fieldPos = -3 + p.fieldsDone = false + } t := p.mf.GetType() if p.state == EntrySeries && !p.fieldsDone && (t == dto.MetricType_SUMMARY || @@ -432,7 +457,7 @@ func (p *ProtobufParser) updateMetricBytes() error { // state. func (p *ProtobufParser) getMagicName() string { t := p.mf.GetType() - if p.state == EntryHistogram || (t != dto.MetricType_HISTOGRAM && t != dto.MetricType_SUMMARY) { + if p.state == EntryHistogram || (t != dto.MetricType_HISTOGRAM && t != dto.MetricType_GAUGE_HISTOGRAM && t != dto.MetricType_SUMMARY) { return p.mf.GetName() } if p.fieldPos == -2 { diff --git a/model/textparse/protobufparse_test.go b/model/textparse/protobufparse_test.go index 90c6a90f32..882cce59d3 100644 --- a/model/textparse/protobufparse_test.go +++ b/model/textparse/protobufparse_test.go @@ -30,8 +30,8 @@ import ( dto "github.com/prometheus/prometheus/prompb/io/prometheus/client" ) -func TestProtobufParse(t *testing.T) { - textMetricFamilies := []string{ +func createTestProtoBuf(t *testing.T) *bytes.Buffer { + testMetricFamilies := []string{ `name: "go_build_info" help: "Build information about the main Go module." type: GAUGE @@ -231,8 +231,7 @@ help: "Test float histogram with many buckets removed to keep it manageable in s type: HISTOGRAM metric: < histogram: < - sample_count: 175 - sample_count_float: 175.0 + sample_count_float: 175.0 sample_sum: 0.0008280461746287094 bucket: < cumulative_count_float: 2.0 @@ -302,8 +301,7 @@ help: "Like test_float_histogram but as gauge histogram." type: GAUGE_HISTOGRAM metric: < histogram: < - sample_count: 175 - sample_count_float: 175.0 + sample_count_float: 175.0 sample_sum: 0.0008280461746287094 bucket: < cumulative_count_float: 2.0 @@ -450,9 +448,9 @@ metric: < } varintBuf := make([]byte, binary.MaxVarintLen32) - inputBuf := &bytes.Buffer{} + buf := &bytes.Buffer{} - for _, tmf := range textMetricFamilies { + for _, tmf := range testMetricFamilies { pb := &dto.MetricFamily{} // From text to proto message. require.NoError(t, proto.UnmarshalText(tmf, pb)) @@ -462,11 +460,15 @@ metric: < // Write first length, then binary protobuf. varintLength := binary.PutUvarint(varintBuf, uint64(len(protoBuf))) - inputBuf.Write(varintBuf[:varintLength]) - inputBuf.Write(protoBuf) + buf.Write(varintBuf[:varintLength]) + buf.Write(protoBuf) } - exp := []struct { + return buf +} + +func TestProtobufParse(t *testing.T) { + type parseResult struct { lset labels.Labels m string t int64 @@ -478,417 +480,1006 @@ metric: < shs *histogram.Histogram fhs *histogram.FloatHistogram e []exemplar.Exemplar + } + + inputBuf := createTestProtoBuf(t) + + scenarios := []struct { + name string + parser Parser + expected []parseResult }{ { - m: "go_build_info", - help: "Build information about the main Go module.", - }, - { - m: "go_build_info", - typ: MetricTypeGauge, - }, - { - m: "go_build_info\xFFchecksum\xFF\xFFpath\xFFgithub.com/prometheus/client_golang\xFFversion\xFF(devel)", - v: 1, - lset: labels.FromStrings( - "__name__", "go_build_info", - "checksum", "", - "path", "github.com/prometheus/client_golang", - "version", "(devel)", - ), - }, - { - m: "go_memstats_alloc_bytes_total", - help: "Total number of bytes allocated, even if freed.", - }, - { - m: "go_memstats_alloc_bytes_total", - typ: MetricTypeCounter, - }, - { - m: "go_memstats_alloc_bytes_total", - v: 1.546544e+06, - lset: labels.FromStrings( - "__name__", "go_memstats_alloc_bytes_total", - ), - e: []exemplar.Exemplar{ - {Labels: labels.FromStrings("dummyID", "42"), Value: 12, HasTs: true, Ts: 1625851151233}, - }, - }, - { - m: "something_untyped", - help: "Just to test the untyped type.", - }, - { - m: "something_untyped", - typ: MetricTypeUnknown, - }, - { - m: "something_untyped", - t: 1234567, - v: 42, - lset: labels.FromStrings( - "__name__", "something_untyped", - ), - }, - { - m: "test_histogram", - help: "Test histogram with many buckets removed to keep it manageable in size.", - }, - { - m: "test_histogram", - typ: MetricTypeHistogram, - }, - { - m: "test_histogram", - t: 1234568, - shs: &histogram.Histogram{ - Count: 175, - ZeroCount: 2, - Sum: 0.0008280461746287094, - ZeroThreshold: 2.938735877055719e-39, - Schema: 3, - PositiveSpans: []histogram.Span{ - {Offset: -161, Length: 1}, - {Offset: 8, Length: 3}, + name: "ignore classic buckets of native histograms", + parser: NewProtobufParser(inputBuf.Bytes(), false), + expected: []parseResult{ + { + m: "go_build_info", + help: "Build information about the main Go module.", }, - NegativeSpans: []histogram.Span{ - {Offset: -162, Length: 1}, - {Offset: 23, Length: 4}, + { + m: "go_build_info", + typ: MetricTypeGauge, }, - PositiveBuckets: []int64{1, 2, -1, -1}, - NegativeBuckets: []int64{1, 3, -2, -1, 1}, - }, - lset: labels.FromStrings( - "__name__", "test_histogram", - ), - e: []exemplar.Exemplar{ - {Labels: labels.FromStrings("dummyID", "59727"), Value: -0.00039, HasTs: true, Ts: 1625851155146}, - {Labels: labels.FromStrings("dummyID", "5617"), Value: -0.00029, HasTs: false}, - }, - }, - { - m: "test_gauge_histogram", - help: "Like test_histogram but as gauge histogram.", - }, - { - m: "test_gauge_histogram", - typ: MetricTypeGaugeHistogram, - }, - { - m: "test_gauge_histogram", - t: 1234568, - shs: &histogram.Histogram{ - CounterResetHint: histogram.GaugeType, - Count: 175, - ZeroCount: 2, - Sum: 0.0008280461746287094, - ZeroThreshold: 2.938735877055719e-39, - Schema: 3, - PositiveSpans: []histogram.Span{ - {Offset: -161, Length: 1}, - {Offset: 8, Length: 3}, + { + m: "go_build_info\xFFchecksum\xFF\xFFpath\xFFgithub.com/prometheus/client_golang\xFFversion\xFF(devel)", + v: 1, + lset: labels.FromStrings( + "__name__", "go_build_info", + "checksum", "", + "path", "github.com/prometheus/client_golang", + "version", "(devel)", + ), }, - NegativeSpans: []histogram.Span{ - {Offset: -162, Length: 1}, - {Offset: 23, Length: 4}, + { + m: "go_memstats_alloc_bytes_total", + help: "Total number of bytes allocated, even if freed.", }, - PositiveBuckets: []int64{1, 2, -1, -1}, - NegativeBuckets: []int64{1, 3, -2, -1, 1}, - }, - lset: labels.FromStrings( - "__name__", "test_gauge_histogram", - ), - e: []exemplar.Exemplar{ - {Labels: labels.FromStrings("dummyID", "59727"), Value: -0.00039, HasTs: true, Ts: 1625851155146}, - {Labels: labels.FromStrings("dummyID", "5617"), Value: -0.00029, HasTs: false}, - }, - }, - { - m: "test_float_histogram", - help: "Test float histogram with many buckets removed to keep it manageable in size.", - }, - { - m: "test_float_histogram", - typ: MetricTypeHistogram, - }, - { - m: "test_float_histogram", - t: 1234568, - fhs: &histogram.FloatHistogram{ - Count: 175.0, - ZeroCount: 2.0, - Sum: 0.0008280461746287094, - ZeroThreshold: 2.938735877055719e-39, - Schema: 3, - PositiveSpans: []histogram.Span{ - {Offset: -161, Length: 1}, - {Offset: 8, Length: 3}, + { + m: "go_memstats_alloc_bytes_total", + typ: MetricTypeCounter, }, - NegativeSpans: []histogram.Span{ - {Offset: -162, Length: 1}, - {Offset: 23, Length: 4}, + { + m: "go_memstats_alloc_bytes_total", + v: 1.546544e+06, + lset: labels.FromStrings( + "__name__", "go_memstats_alloc_bytes_total", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "42"), Value: 12, HasTs: true, Ts: 1625851151233}, + }, }, - PositiveBuckets: []float64{1.0, 2.0, -1.0, -1.0}, - NegativeBuckets: []float64{1.0, 3.0, -2.0, -1.0, 1.0}, - }, - lset: labels.FromStrings( - "__name__", "test_float_histogram", - ), - e: []exemplar.Exemplar{ - {Labels: labels.FromStrings("dummyID", "59727"), Value: -0.00039, HasTs: true, Ts: 1625851155146}, - {Labels: labels.FromStrings("dummyID", "5617"), Value: -0.00029, HasTs: false}, - }, - }, - { - m: "test_gauge_float_histogram", - help: "Like test_float_histogram but as gauge histogram.", - }, - { - m: "test_gauge_float_histogram", - typ: MetricTypeGaugeHistogram, - }, - { - m: "test_gauge_float_histogram", - t: 1234568, - fhs: &histogram.FloatHistogram{ - CounterResetHint: histogram.GaugeType, - Count: 175.0, - ZeroCount: 2.0, - Sum: 0.0008280461746287094, - ZeroThreshold: 2.938735877055719e-39, - Schema: 3, - PositiveSpans: []histogram.Span{ - {Offset: -161, Length: 1}, - {Offset: 8, Length: 3}, + { + m: "something_untyped", + help: "Just to test the untyped type.", }, - NegativeSpans: []histogram.Span{ - {Offset: -162, Length: 1}, - {Offset: 23, Length: 4}, + { + m: "something_untyped", + typ: MetricTypeUnknown, + }, + { + m: "something_untyped", + t: 1234567, + v: 42, + lset: labels.FromStrings( + "__name__", "something_untyped", + ), + }, + { + m: "test_histogram", + help: "Test histogram with many buckets removed to keep it manageable in size.", + }, + { + m: "test_histogram", + typ: MetricTypeHistogram, + }, + { + m: "test_histogram", + t: 1234568, + shs: &histogram.Histogram{ + Count: 175, + ZeroCount: 2, + Sum: 0.0008280461746287094, + ZeroThreshold: 2.938735877055719e-39, + Schema: 3, + PositiveSpans: []histogram.Span{ + {Offset: -161, Length: 1}, + {Offset: 8, Length: 3}, + }, + NegativeSpans: []histogram.Span{ + {Offset: -162, Length: 1}, + {Offset: 23, Length: 4}, + }, + PositiveBuckets: []int64{1, 2, -1, -1}, + NegativeBuckets: []int64{1, 3, -2, -1, 1}, + }, + lset: labels.FromStrings( + "__name__", "test_histogram", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "59727"), Value: -0.00039, HasTs: true, Ts: 1625851155146}, + {Labels: labels.FromStrings("dummyID", "5617"), Value: -0.00029, HasTs: false}, + }, + }, + { + m: "test_gauge_histogram", + help: "Like test_histogram but as gauge histogram.", + }, + { + m: "test_gauge_histogram", + typ: MetricTypeGaugeHistogram, + }, + { + m: "test_gauge_histogram", + t: 1234568, + shs: &histogram.Histogram{ + CounterResetHint: histogram.GaugeType, + Count: 175, + ZeroCount: 2, + Sum: 0.0008280461746287094, + ZeroThreshold: 2.938735877055719e-39, + Schema: 3, + PositiveSpans: []histogram.Span{ + {Offset: -161, Length: 1}, + {Offset: 8, Length: 3}, + }, + NegativeSpans: []histogram.Span{ + {Offset: -162, Length: 1}, + {Offset: 23, Length: 4}, + }, + PositiveBuckets: []int64{1, 2, -1, -1}, + NegativeBuckets: []int64{1, 3, -2, -1, 1}, + }, + lset: labels.FromStrings( + "__name__", "test_gauge_histogram", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "59727"), Value: -0.00039, HasTs: true, Ts: 1625851155146}, + {Labels: labels.FromStrings("dummyID", "5617"), Value: -0.00029, HasTs: false}, + }, + }, + { + m: "test_float_histogram", + help: "Test float histogram with many buckets removed to keep it manageable in size.", + }, + { + m: "test_float_histogram", + typ: MetricTypeHistogram, + }, + { + m: "test_float_histogram", + t: 1234568, + fhs: &histogram.FloatHistogram{ + Count: 175.0, + ZeroCount: 2.0, + Sum: 0.0008280461746287094, + ZeroThreshold: 2.938735877055719e-39, + Schema: 3, + PositiveSpans: []histogram.Span{ + {Offset: -161, Length: 1}, + {Offset: 8, Length: 3}, + }, + NegativeSpans: []histogram.Span{ + {Offset: -162, Length: 1}, + {Offset: 23, Length: 4}, + }, + PositiveBuckets: []float64{1.0, 2.0, -1.0, -1.0}, + NegativeBuckets: []float64{1.0, 3.0, -2.0, -1.0, 1.0}, + }, + lset: labels.FromStrings( + "__name__", "test_float_histogram", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "59727"), Value: -0.00039, HasTs: true, Ts: 1625851155146}, + {Labels: labels.FromStrings("dummyID", "5617"), Value: -0.00029, HasTs: false}, + }, + }, + { + m: "test_gauge_float_histogram", + help: "Like test_float_histogram but as gauge histogram.", + }, + { + m: "test_gauge_float_histogram", + typ: MetricTypeGaugeHistogram, + }, + { + m: "test_gauge_float_histogram", + t: 1234568, + fhs: &histogram.FloatHistogram{ + CounterResetHint: histogram.GaugeType, + Count: 175.0, + ZeroCount: 2.0, + Sum: 0.0008280461746287094, + ZeroThreshold: 2.938735877055719e-39, + Schema: 3, + PositiveSpans: []histogram.Span{ + {Offset: -161, Length: 1}, + {Offset: 8, Length: 3}, + }, + NegativeSpans: []histogram.Span{ + {Offset: -162, Length: 1}, + {Offset: 23, Length: 4}, + }, + PositiveBuckets: []float64{1.0, 2.0, -1.0, -1.0}, + NegativeBuckets: []float64{1.0, 3.0, -2.0, -1.0, 1.0}, + }, + lset: labels.FromStrings( + "__name__", "test_gauge_float_histogram", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "59727"), Value: -0.00039, HasTs: true, Ts: 1625851155146}, + {Labels: labels.FromStrings("dummyID", "5617"), Value: -0.00029, HasTs: false}, + }, + }, + { + m: "test_histogram2", + help: "Similar histogram as before but now without sparse buckets.", + }, + { + m: "test_histogram2", + typ: MetricTypeHistogram, + }, + { + m: "test_histogram2_count", + v: 175, + lset: labels.FromStrings( + "__name__", "test_histogram2_count", + ), + }, + { + m: "test_histogram2_sum", + v: 0.000828, + lset: labels.FromStrings( + "__name__", "test_histogram2_sum", + ), + }, + { + m: "test_histogram2_bucket\xffle\xff-0.00048", + v: 2, + lset: labels.FromStrings( + "__name__", "test_histogram2_bucket", + "le", "-0.00048", + ), + }, + { + m: "test_histogram2_bucket\xffle\xff-0.00038", + v: 4, + lset: labels.FromStrings( + "__name__", "test_histogram2_bucket", + "le", "-0.00038", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "59727"), Value: -0.00038, HasTs: true, Ts: 1625851153146}, + }, + }, + { + m: "test_histogram2_bucket\xffle\xff1.0", + v: 16, + lset: labels.FromStrings( + "__name__", "test_histogram2_bucket", + "le", "1.0", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "5617"), Value: -0.000295, HasTs: false}, + }, + }, + { + m: "test_histogram2_bucket\xffle\xff+Inf", + v: 175, + lset: labels.FromStrings( + "__name__", "test_histogram2_bucket", + "le", "+Inf", + ), + }, + { + m: "rpc_durations_seconds", + help: "RPC latency distributions.", + }, + { + m: "rpc_durations_seconds", + typ: MetricTypeSummary, + }, + { + m: "rpc_durations_seconds_count\xffservice\xffexponential", + v: 262, + lset: labels.FromStrings( + "__name__", "rpc_durations_seconds_count", + "service", "exponential", + ), + }, + { + m: "rpc_durations_seconds_sum\xffservice\xffexponential", + v: 0.00025551262820703587, + lset: labels.FromStrings( + "__name__", "rpc_durations_seconds_sum", + "service", "exponential", + ), + }, + { + m: "rpc_durations_seconds\xffservice\xffexponential\xffquantile\xff0.5", + v: 6.442786329648548e-07, + lset: labels.FromStrings( + "__name__", "rpc_durations_seconds", + "quantile", "0.5", + "service", "exponential", + ), + }, + { + m: "rpc_durations_seconds\xffservice\xffexponential\xffquantile\xff0.9", + v: 1.9435742936658396e-06, + lset: labels.FromStrings( + "__name__", "rpc_durations_seconds", + "quantile", "0.9", + "service", "exponential", + ), + }, + { + m: "rpc_durations_seconds\xffservice\xffexponential\xffquantile\xff0.99", + v: 4.0471608667037015e-06, + lset: labels.FromStrings( + "__name__", "rpc_durations_seconds", + "quantile", "0.99", + "service", "exponential", + ), + }, + { + m: "without_quantiles", + help: "A summary without quantiles.", + }, + { + m: "without_quantiles", + typ: MetricTypeSummary, + }, + { + m: "without_quantiles_count", + v: 42, + lset: labels.FromStrings( + "__name__", "without_quantiles_count", + ), + }, + { + m: "without_quantiles_sum", + v: 1.234, + lset: labels.FromStrings( + "__name__", "without_quantiles_sum", + ), }, - PositiveBuckets: []float64{1.0, 2.0, -1.0, -1.0}, - NegativeBuckets: []float64{1.0, 3.0, -2.0, -1.0, 1.0}, - }, - lset: labels.FromStrings( - "__name__", "test_gauge_float_histogram", - ), - e: []exemplar.Exemplar{ - {Labels: labels.FromStrings("dummyID", "59727"), Value: -0.00039, HasTs: true, Ts: 1625851155146}, - {Labels: labels.FromStrings("dummyID", "5617"), Value: -0.00029, HasTs: false}, }, }, { - m: "test_histogram2", - help: "Similar histogram as before but now without sparse buckets.", - }, - { - m: "test_histogram2", - typ: MetricTypeHistogram, - }, - { - m: "test_histogram2_count", - v: 175, - lset: labels.FromStrings( - "__name__", "test_histogram2_count", - ), - }, - { - m: "test_histogram2_sum", - v: 0.000828, - lset: labels.FromStrings( - "__name__", "test_histogram2_sum", - ), - }, - { - m: "test_histogram2_bucket\xffle\xff-0.00048", - v: 2, - lset: labels.FromStrings( - "__name__", "test_histogram2_bucket", - "le", "-0.00048", - ), - }, - { - m: "test_histogram2_bucket\xffle\xff-0.00038", - v: 4, - lset: labels.FromStrings( - "__name__", "test_histogram2_bucket", - "le", "-0.00038", - ), - e: []exemplar.Exemplar{ - {Labels: labels.FromStrings("dummyID", "59727"), Value: -0.00038, HasTs: true, Ts: 1625851153146}, + name: "parse classic and native buckets", + parser: NewProtobufParser(inputBuf.Bytes(), true), + expected: []parseResult{ + { // 0 + m: "go_build_info", + help: "Build information about the main Go module.", + }, + { // 1 + m: "go_build_info", + typ: MetricTypeGauge, + }, + { // 2 + m: "go_build_info\xFFchecksum\xFF\xFFpath\xFFgithub.com/prometheus/client_golang\xFFversion\xFF(devel)", + v: 1, + lset: labels.FromStrings( + "__name__", "go_build_info", + "checksum", "", + "path", "github.com/prometheus/client_golang", + "version", "(devel)", + ), + }, + { // 3 + m: "go_memstats_alloc_bytes_total", + help: "Total number of bytes allocated, even if freed.", + }, + { // 4 + m: "go_memstats_alloc_bytes_total", + typ: MetricTypeCounter, + }, + { // 5 + m: "go_memstats_alloc_bytes_total", + v: 1.546544e+06, + lset: labels.FromStrings( + "__name__", "go_memstats_alloc_bytes_total", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "42"), Value: 12, HasTs: true, Ts: 1625851151233}, + }, + }, + { // 6 + m: "something_untyped", + help: "Just to test the untyped type.", + }, + { // 7 + m: "something_untyped", + typ: MetricTypeUnknown, + }, + { // 8 + m: "something_untyped", + t: 1234567, + v: 42, + lset: labels.FromStrings( + "__name__", "something_untyped", + ), + }, + { // 9 + m: "test_histogram", + help: "Test histogram with many buckets removed to keep it manageable in size.", + }, + { // 10 + m: "test_histogram", + typ: MetricTypeHistogram, + }, + { // 11 + m: "test_histogram", + t: 1234568, + shs: &histogram.Histogram{ + Count: 175, + ZeroCount: 2, + Sum: 0.0008280461746287094, + ZeroThreshold: 2.938735877055719e-39, + Schema: 3, + PositiveSpans: []histogram.Span{ + {Offset: -161, Length: 1}, + {Offset: 8, Length: 3}, + }, + NegativeSpans: []histogram.Span{ + {Offset: -162, Length: 1}, + {Offset: 23, Length: 4}, + }, + PositiveBuckets: []int64{1, 2, -1, -1}, + NegativeBuckets: []int64{1, 3, -2, -1, 1}, + }, + lset: labels.FromStrings( + "__name__", "test_histogram", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "59727"), Value: -0.00039, HasTs: true, Ts: 1625851155146}, + {Labels: labels.FromStrings("dummyID", "5617"), Value: -0.00029, HasTs: false}, + }, + }, + { // 12 + m: "test_histogram_count", + t: 1234568, + v: 175, + lset: labels.FromStrings( + "__name__", "test_histogram_count", + ), + }, + { // 13 + m: "test_histogram_sum", + t: 1234568, + v: 0.0008280461746287094, + lset: labels.FromStrings( + "__name__", "test_histogram_sum", + ), + }, + { // 14 + m: "test_histogram_bucket\xffle\xff-0.0004899999999999998", + t: 1234568, + v: 2, + lset: labels.FromStrings( + "__name__", "test_histogram_bucket", + "le", "-0.0004899999999999998", + ), + }, + { // 15 + m: "test_histogram_bucket\xffle\xff-0.0003899999999999998", + t: 1234568, + v: 4, + lset: labels.FromStrings( + "__name__", "test_histogram_bucket", + "le", "-0.0003899999999999998", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "59727"), Value: -0.00039, HasTs: true, Ts: 1625851155146}, + }, + }, + { // 16 + m: "test_histogram_bucket\xffle\xff-0.0002899999999999998", + t: 1234568, + v: 16, + lset: labels.FromStrings( + "__name__", "test_histogram_bucket", + "le", "-0.0002899999999999998", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "5617"), Value: -0.00029, HasTs: false}, + }, + }, + { // 17 + m: "test_histogram_bucket\xffle\xff+Inf", + t: 1234568, + v: 175, + lset: labels.FromStrings( + "__name__", "test_histogram_bucket", + "le", "+Inf", + ), + }, + { // 18 + m: "test_gauge_histogram", + help: "Like test_histogram but as gauge histogram.", + }, + { // 19 + m: "test_gauge_histogram", + typ: MetricTypeGaugeHistogram, + }, + { // 20 + m: "test_gauge_histogram", + t: 1234568, + shs: &histogram.Histogram{ + CounterResetHint: histogram.GaugeType, + Count: 175, + ZeroCount: 2, + Sum: 0.0008280461746287094, + ZeroThreshold: 2.938735877055719e-39, + Schema: 3, + PositiveSpans: []histogram.Span{ + {Offset: -161, Length: 1}, + {Offset: 8, Length: 3}, + }, + NegativeSpans: []histogram.Span{ + {Offset: -162, Length: 1}, + {Offset: 23, Length: 4}, + }, + PositiveBuckets: []int64{1, 2, -1, -1}, + NegativeBuckets: []int64{1, 3, -2, -1, 1}, + }, + lset: labels.FromStrings( + "__name__", "test_gauge_histogram", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "59727"), Value: -0.00039, HasTs: true, Ts: 1625851155146}, + {Labels: labels.FromStrings("dummyID", "5617"), Value: -0.00029, HasTs: false}, + }, + }, + { // 21 + m: "test_gauge_histogram_count", + t: 1234568, + v: 175, + lset: labels.FromStrings( + "__name__", "test_gauge_histogram_count", + ), + }, + { // 22 + m: "test_gauge_histogram_sum", + t: 1234568, + v: 0.0008280461746287094, + lset: labels.FromStrings( + "__name__", "test_gauge_histogram_sum", + ), + }, + { // 23 + m: "test_gauge_histogram_bucket\xffle\xff-0.0004899999999999998", + t: 1234568, + v: 2, + lset: labels.FromStrings( + "__name__", "test_gauge_histogram_bucket", + "le", "-0.0004899999999999998", + ), + }, + { // 24 + m: "test_gauge_histogram_bucket\xffle\xff-0.0003899999999999998", + t: 1234568, + v: 4, + lset: labels.FromStrings( + "__name__", "test_gauge_histogram_bucket", + "le", "-0.0003899999999999998", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "59727"), Value: -0.00039, HasTs: true, Ts: 1625851155146}, + }, + }, + { // 25 + m: "test_gauge_histogram_bucket\xffle\xff-0.0002899999999999998", + t: 1234568, + v: 16, + lset: labels.FromStrings( + "__name__", "test_gauge_histogram_bucket", + "le", "-0.0002899999999999998", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "5617"), Value: -0.00029, HasTs: false}, + }, + }, + { // 26 + m: "test_gauge_histogram_bucket\xffle\xff+Inf", + t: 1234568, + v: 175, + lset: labels.FromStrings( + "__name__", "test_gauge_histogram_bucket", + "le", "+Inf", + ), + }, + { // 27 + m: "test_float_histogram", + help: "Test float histogram with many buckets removed to keep it manageable in size.", + }, + { // 28 + m: "test_float_histogram", + typ: MetricTypeHistogram, + }, + { // 29 + m: "test_float_histogram", + t: 1234568, + fhs: &histogram.FloatHistogram{ + Count: 175.0, + ZeroCount: 2.0, + Sum: 0.0008280461746287094, + ZeroThreshold: 2.938735877055719e-39, + Schema: 3, + PositiveSpans: []histogram.Span{ + {Offset: -161, Length: 1}, + {Offset: 8, Length: 3}, + }, + NegativeSpans: []histogram.Span{ + {Offset: -162, Length: 1}, + {Offset: 23, Length: 4}, + }, + PositiveBuckets: []float64{1.0, 2.0, -1.0, -1.0}, + NegativeBuckets: []float64{1.0, 3.0, -2.0, -1.0, 1.0}, + }, + lset: labels.FromStrings( + "__name__", "test_float_histogram", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "59727"), Value: -0.00039, HasTs: true, Ts: 1625851155146}, + {Labels: labels.FromStrings("dummyID", "5617"), Value: -0.00029, HasTs: false}, + }, + }, + { // 30 + m: "test_float_histogram_count", + t: 1234568, + v: 175, + lset: labels.FromStrings( + "__name__", "test_float_histogram_count", + ), + }, + { // 31 + m: "test_float_histogram_sum", + t: 1234568, + v: 0.0008280461746287094, + lset: labels.FromStrings( + "__name__", "test_float_histogram_sum", + ), + }, + { // 32 + m: "test_float_histogram_bucket\xffle\xff-0.0004899999999999998", + t: 1234568, + v: 2, + lset: labels.FromStrings( + "__name__", "test_float_histogram_bucket", + "le", "-0.0004899999999999998", + ), + }, + { // 33 + m: "test_float_histogram_bucket\xffle\xff-0.0003899999999999998", + t: 1234568, + v: 4, + lset: labels.FromStrings( + "__name__", "test_float_histogram_bucket", + "le", "-0.0003899999999999998", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "59727"), Value: -0.00039, HasTs: true, Ts: 1625851155146}, + }, + }, + { // 34 + m: "test_float_histogram_bucket\xffle\xff-0.0002899999999999998", + t: 1234568, + v: 16, + lset: labels.FromStrings( + "__name__", "test_float_histogram_bucket", + "le", "-0.0002899999999999998", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "5617"), Value: -0.00029, HasTs: false}, + }, + }, + { // 35 + m: "test_float_histogram_bucket\xffle\xff+Inf", + t: 1234568, + v: 175, + lset: labels.FromStrings( + "__name__", "test_float_histogram_bucket", + "le", "+Inf", + ), + }, + { // 36 + m: "test_gauge_float_histogram", + help: "Like test_float_histogram but as gauge histogram.", + }, + { // 37 + m: "test_gauge_float_histogram", + typ: MetricTypeGaugeHistogram, + }, + { // 38 + m: "test_gauge_float_histogram", + t: 1234568, + fhs: &histogram.FloatHistogram{ + CounterResetHint: histogram.GaugeType, + Count: 175.0, + ZeroCount: 2.0, + Sum: 0.0008280461746287094, + ZeroThreshold: 2.938735877055719e-39, + Schema: 3, + PositiveSpans: []histogram.Span{ + {Offset: -161, Length: 1}, + {Offset: 8, Length: 3}, + }, + NegativeSpans: []histogram.Span{ + {Offset: -162, Length: 1}, + {Offset: 23, Length: 4}, + }, + PositiveBuckets: []float64{1.0, 2.0, -1.0, -1.0}, + NegativeBuckets: []float64{1.0, 3.0, -2.0, -1.0, 1.0}, + }, + lset: labels.FromStrings( + "__name__", "test_gauge_float_histogram", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "59727"), Value: -0.00039, HasTs: true, Ts: 1625851155146}, + {Labels: labels.FromStrings("dummyID", "5617"), Value: -0.00029, HasTs: false}, + }, + }, + { // 39 + m: "test_gauge_float_histogram_count", + t: 1234568, + v: 175, + lset: labels.FromStrings( + "__name__", "test_gauge_float_histogram_count", + ), + }, + { // 40 + m: "test_gauge_float_histogram_sum", + t: 1234568, + v: 0.0008280461746287094, + lset: labels.FromStrings( + "__name__", "test_gauge_float_histogram_sum", + ), + }, + { // 41 + m: "test_gauge_float_histogram_bucket\xffle\xff-0.0004899999999999998", + t: 1234568, + v: 2, + lset: labels.FromStrings( + "__name__", "test_gauge_float_histogram_bucket", + "le", "-0.0004899999999999998", + ), + }, + { // 42 + m: "test_gauge_float_histogram_bucket\xffle\xff-0.0003899999999999998", + t: 1234568, + v: 4, + lset: labels.FromStrings( + "__name__", "test_gauge_float_histogram_bucket", + "le", "-0.0003899999999999998", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "59727"), Value: -0.00039, HasTs: true, Ts: 1625851155146}, + }, + }, + { // 43 + m: "test_gauge_float_histogram_bucket\xffle\xff-0.0002899999999999998", + t: 1234568, + v: 16, + lset: labels.FromStrings( + "__name__", "test_gauge_float_histogram_bucket", + "le", "-0.0002899999999999998", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "5617"), Value: -0.00029, HasTs: false}, + }, + }, + { // 44 + m: "test_gauge_float_histogram_bucket\xffle\xff+Inf", + t: 1234568, + v: 175, + lset: labels.FromStrings( + "__name__", "test_gauge_float_histogram_bucket", + "le", "+Inf", + ), + }, + { // 45 + m: "test_histogram2", + help: "Similar histogram as before but now without sparse buckets.", + }, + { // 46 + m: "test_histogram2", + typ: MetricTypeHistogram, + }, + { // 47 + m: "test_histogram2_count", + v: 175, + lset: labels.FromStrings( + "__name__", "test_histogram2_count", + ), + }, + { // 48 + m: "test_histogram2_sum", + v: 0.000828, + lset: labels.FromStrings( + "__name__", "test_histogram2_sum", + ), + }, + { // 49 + m: "test_histogram2_bucket\xffle\xff-0.00048", + v: 2, + lset: labels.FromStrings( + "__name__", "test_histogram2_bucket", + "le", "-0.00048", + ), + }, + { // 50 + m: "test_histogram2_bucket\xffle\xff-0.00038", + v: 4, + lset: labels.FromStrings( + "__name__", "test_histogram2_bucket", + "le", "-0.00038", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "59727"), Value: -0.00038, HasTs: true, Ts: 1625851153146}, + }, + }, + { // 51 + m: "test_histogram2_bucket\xffle\xff1.0", + v: 16, + lset: labels.FromStrings( + "__name__", "test_histogram2_bucket", + "le", "1.0", + ), + e: []exemplar.Exemplar{ + {Labels: labels.FromStrings("dummyID", "5617"), Value: -0.000295, HasTs: false}, + }, + }, + { // 52 + m: "test_histogram2_bucket\xffle\xff+Inf", + v: 175, + lset: labels.FromStrings( + "__name__", "test_histogram2_bucket", + "le", "+Inf", + ), + }, + { // 53 + m: "rpc_durations_seconds", + help: "RPC latency distributions.", + }, + { // 54 + m: "rpc_durations_seconds", + typ: MetricTypeSummary, + }, + { // 55 + m: "rpc_durations_seconds_count\xffservice\xffexponential", + v: 262, + lset: labels.FromStrings( + "__name__", "rpc_durations_seconds_count", + "service", "exponential", + ), + }, + { // 56 + m: "rpc_durations_seconds_sum\xffservice\xffexponential", + v: 0.00025551262820703587, + lset: labels.FromStrings( + "__name__", "rpc_durations_seconds_sum", + "service", "exponential", + ), + }, + { // 57 + m: "rpc_durations_seconds\xffservice\xffexponential\xffquantile\xff0.5", + v: 6.442786329648548e-07, + lset: labels.FromStrings( + "__name__", "rpc_durations_seconds", + "quantile", "0.5", + "service", "exponential", + ), + }, + { // 58 + m: "rpc_durations_seconds\xffservice\xffexponential\xffquantile\xff0.9", + v: 1.9435742936658396e-06, + lset: labels.FromStrings( + "__name__", "rpc_durations_seconds", + "quantile", "0.9", + "service", "exponential", + ), + }, + { // 59 + m: "rpc_durations_seconds\xffservice\xffexponential\xffquantile\xff0.99", + v: 4.0471608667037015e-06, + lset: labels.FromStrings( + "__name__", "rpc_durations_seconds", + "quantile", "0.99", + "service", "exponential", + ), + }, + { // 60 + m: "without_quantiles", + help: "A summary without quantiles.", + }, + { // 61 + m: "without_quantiles", + typ: MetricTypeSummary, + }, + { // 62 + m: "without_quantiles_count", + v: 42, + lset: labels.FromStrings( + "__name__", "without_quantiles_count", + ), + }, + { // 63 + m: "without_quantiles_sum", + v: 1.234, + lset: labels.FromStrings( + "__name__", "without_quantiles_sum", + ), + }, }, }, - { - m: "test_histogram2_bucket\xffle\xff1.0", - v: 16, - lset: labels.FromStrings( - "__name__", "test_histogram2_bucket", - "le", "1.0", - ), - e: []exemplar.Exemplar{ - {Labels: labels.FromStrings("dummyID", "5617"), Value: -0.000295, HasTs: false}, - }, - }, - { - m: "test_histogram2_bucket\xffle\xff+Inf", - v: 175, - lset: labels.FromStrings( - "__name__", "test_histogram2_bucket", - "le", "+Inf", - ), - }, - { - m: "rpc_durations_seconds", - help: "RPC latency distributions.", - }, - { - m: "rpc_durations_seconds", - typ: MetricTypeSummary, - }, - { - m: "rpc_durations_seconds_count\xffservice\xffexponential", - v: 262, - lset: labels.FromStrings( - "__name__", "rpc_durations_seconds_count", - "service", "exponential", - ), - }, - { - m: "rpc_durations_seconds_sum\xffservice\xffexponential", - v: 0.00025551262820703587, - lset: labels.FromStrings( - "__name__", "rpc_durations_seconds_sum", - "service", "exponential", - ), - }, - { - m: "rpc_durations_seconds\xffservice\xffexponential\xffquantile\xff0.5", - v: 6.442786329648548e-07, - lset: labels.FromStrings( - "__name__", "rpc_durations_seconds", - "quantile", "0.5", - "service", "exponential", - ), - }, - { - m: "rpc_durations_seconds\xffservice\xffexponential\xffquantile\xff0.9", - v: 1.9435742936658396e-06, - lset: labels.FromStrings( - "__name__", "rpc_durations_seconds", - "quantile", "0.9", - "service", "exponential", - ), - }, - { - m: "rpc_durations_seconds\xffservice\xffexponential\xffquantile\xff0.99", - v: 4.0471608667037015e-06, - lset: labels.FromStrings( - "__name__", "rpc_durations_seconds", - "quantile", "0.99", - "service", "exponential", - ), - }, - { - m: "without_quantiles", - help: "A summary without quantiles.", - }, - { - m: "without_quantiles", - typ: MetricTypeSummary, - }, - { - m: "without_quantiles_count", - v: 42, - lset: labels.FromStrings( - "__name__", "without_quantiles_count", - ), - }, - { - m: "without_quantiles_sum", - v: 1.234, - lset: labels.FromStrings( - "__name__", "without_quantiles_sum", - ), - }, } - p := NewProtobufParser(inputBuf.Bytes()) - i := 0 + for _, scenario := range scenarios { + t.Run(scenario.name, func(t *testing.T) { + var ( + i int + res labels.Labels + p = scenario.parser + exp = scenario.expected + ) - var res labels.Labels + for { + et, err := p.Next() + if errors.Is(err, io.EOF) { + break + } + require.NoError(t, err) - for { - et, err := p.Next() - if errors.Is(err, io.EOF) { - break - } - require.NoError(t, err) + switch et { + case EntrySeries: + m, ts, v := p.Series() - switch et { - case EntrySeries: - m, ts, v := p.Series() + var e exemplar.Exemplar + p.Metric(&res) + found := p.Exemplar(&e) + require.Equal(t, exp[i].m, string(m)) + if ts != nil { + require.Equal(t, exp[i].t, *ts) + } else { + require.Equal(t, exp[i].t, int64(0)) + } + require.Equal(t, exp[i].v, v) + require.Equal(t, exp[i].lset, res) + if len(exp[i].e) == 0 { + require.Equal(t, false, found) + } else { + require.Equal(t, true, found) + require.Equal(t, exp[i].e[0], e) + } - var e exemplar.Exemplar - p.Metric(&res) - found := p.Exemplar(&e) - require.Equal(t, exp[i].m, string(m)) - if ts != nil { - require.Equal(t, exp[i].t, *ts) - } else { - require.Equal(t, exp[i].t, int64(0)) + case EntryHistogram: + m, ts, shs, fhs := p.Histogram() + p.Metric(&res) + require.Equal(t, exp[i].m, string(m)) + if ts != nil { + require.Equal(t, exp[i].t, *ts) + } else { + require.Equal(t, exp[i].t, int64(0)) + } + require.Equal(t, exp[i].lset, res) + require.Equal(t, exp[i].m, string(m)) + if shs != nil { + require.Equal(t, exp[i].shs, shs) + } else { + require.Equal(t, exp[i].fhs, fhs) + } + j := 0 + for e := (exemplar.Exemplar{}); p.Exemplar(&e); j++ { + require.Equal(t, exp[i].e[j], e) + e = exemplar.Exemplar{} + } + require.Equal(t, len(exp[i].e), j, "not enough exemplars found") + + case EntryType: + m, typ := p.Type() + require.Equal(t, exp[i].m, string(m)) + require.Equal(t, exp[i].typ, typ) + + case EntryHelp: + m, h := p.Help() + require.Equal(t, exp[i].m, string(m)) + require.Equal(t, exp[i].help, string(h)) + + case EntryUnit: + m, u := p.Unit() + require.Equal(t, exp[i].m, string(m)) + require.Equal(t, exp[i].unit, string(u)) + + case EntryComment: + require.Equal(t, exp[i].comment, string(p.Comment())) + } + + i++ } - require.Equal(t, exp[i].v, v) - require.Equal(t, exp[i].lset, res) - if len(exp[i].e) == 0 { - require.Equal(t, false, found) - } else { - require.Equal(t, true, found) - require.Equal(t, exp[i].e[0], e) - } - - case EntryHistogram: - m, ts, shs, fhs := p.Histogram() - p.Metric(&res) - require.Equal(t, exp[i].m, string(m)) - if ts != nil { - require.Equal(t, exp[i].t, *ts) - } else { - require.Equal(t, exp[i].t, int64(0)) - } - require.Equal(t, exp[i].lset, res) - require.Equal(t, exp[i].m, string(m)) - if shs != nil { - require.Equal(t, exp[i].shs, shs) - } else { - require.Equal(t, exp[i].fhs, fhs) - } - j := 0 - for e := (exemplar.Exemplar{}); p.Exemplar(&e); j++ { - require.Equal(t, exp[i].e[j], e) - e = exemplar.Exemplar{} - } - require.Equal(t, len(exp[i].e), j, "not enough exemplars found") - - case EntryType: - m, typ := p.Type() - require.Equal(t, exp[i].m, string(m)) - require.Equal(t, exp[i].typ, typ) - - case EntryHelp: - m, h := p.Help() - require.Equal(t, exp[i].m, string(m)) - require.Equal(t, exp[i].help, string(h)) - - case EntryUnit: - m, u := p.Unit() - require.Equal(t, exp[i].m, string(m)) - require.Equal(t, exp[i].unit, string(u)) - - case EntryComment: - require.Equal(t, exp[i].comment, string(p.Comment())) - } - - i++ + require.Equal(t, len(exp), i) + }) } - require.Equal(t, len(exp), i) } diff --git a/promql/fuzz.go b/promql/fuzz.go index 39933378e6..aff6eb15b2 100644 --- a/promql/fuzz.go +++ b/promql/fuzz.go @@ -58,7 +58,7 @@ const ( ) func fuzzParseMetricWithContentType(in []byte, contentType string) int { - p, warning := textparse.New(in, contentType) + p, warning := textparse.New(in, contentType, false) if warning != nil { // An invalid content type is being passed, which should not happen // in this context. diff --git a/scrape/scrape.go b/scrape/scrape.go index f094ee8257..a97cbf539f 100644 --- a/scrape/scrape.go +++ b/scrape/scrape.go @@ -260,17 +260,18 @@ type labelLimits struct { } type scrapeLoopOptions struct { - target *Target - scraper scraper - sampleLimit int - bucketLimit int - labelLimits *labelLimits - honorLabels bool - honorTimestamps bool - interval time.Duration - timeout time.Duration - mrc []*relabel.Config - cache *scrapeCache + target *Target + scraper scraper + sampleLimit int + bucketLimit int + labelLimits *labelLimits + honorLabels bool + honorTimestamps bool + interval time.Duration + timeout time.Duration + scrapeClassicHistograms bool + mrc []*relabel.Config + cache *scrapeCache } const maxAheadTime = 10 * time.Minute @@ -331,6 +332,7 @@ func newScrapePool(cfg *config.ScrapeConfig, app storage.Appendable, jitterSeed opts.labelLimits, opts.interval, opts.timeout, + opts.scrapeClassicHistograms, options.ExtraMetrics, options.EnableMetadataStorage, opts.target, @@ -547,9 +549,10 @@ func (sp *scrapePool) sync(targets []*Target) { labelNameLengthLimit: int(sp.config.LabelNameLengthLimit), labelValueLengthLimit: int(sp.config.LabelValueLengthLimit), } - honorLabels = sp.config.HonorLabels - honorTimestamps = sp.config.HonorTimestamps - mrc = sp.config.MetricRelabelConfigs + honorLabels = sp.config.HonorLabels + honorTimestamps = sp.config.HonorTimestamps + mrc = sp.config.MetricRelabelConfigs + scrapeClassicHistograms = sp.config.ScrapeClassicHistograms ) sp.targetMtx.Lock() @@ -568,16 +571,17 @@ func (sp *scrapePool) sync(targets []*Target) { } s := &targetScraper{Target: t, client: sp.client, timeout: timeout, bodySizeLimit: bodySizeLimit, acceptHeader: acceptHeader} l := sp.newLoop(scrapeLoopOptions{ - target: t, - scraper: s, - sampleLimit: sampleLimit, - bucketLimit: bucketLimit, - labelLimits: labelLimits, - honorLabels: honorLabels, - honorTimestamps: honorTimestamps, - mrc: mrc, - interval: interval, - timeout: timeout, + target: t, + scraper: s, + sampleLimit: sampleLimit, + bucketLimit: bucketLimit, + labelLimits: labelLimits, + honorLabels: honorLabels, + honorTimestamps: honorTimestamps, + mrc: mrc, + interval: interval, + timeout: timeout, + scrapeClassicHistograms: scrapeClassicHistograms, }) if err != nil { l.setForcedError(err) @@ -882,20 +886,21 @@ type cacheEntry struct { } type scrapeLoop struct { - scraper scraper - l log.Logger - cache *scrapeCache - lastScrapeSize int - buffers *pool.Pool - jitterSeed uint64 - honorTimestamps bool - forcedErr error - forcedErrMtx sync.Mutex - sampleLimit int - bucketLimit int - labelLimits *labelLimits - interval time.Duration - timeout time.Duration + scraper scraper + l log.Logger + cache *scrapeCache + lastScrapeSize int + buffers *pool.Pool + jitterSeed uint64 + honorTimestamps bool + forcedErr error + forcedErrMtx sync.Mutex + sampleLimit int + bucketLimit int + labelLimits *labelLimits + interval time.Duration + timeout time.Duration + scrapeClassicHistograms bool appender func(ctx context.Context) storage.Appender sampleMutator labelsMutator @@ -1177,6 +1182,7 @@ func newScrapeLoop(ctx context.Context, labelLimits *labelLimits, interval time.Duration, timeout time.Duration, + scrapeClassicHistograms bool, reportExtraMetrics bool, appendMetadataToWAL bool, target *Target, @@ -1204,25 +1210,26 @@ func newScrapeLoop(ctx context.Context, } sl := &scrapeLoop{ - scraper: sc, - buffers: buffers, - cache: cache, - appender: appender, - sampleMutator: sampleMutator, - reportSampleMutator: reportSampleMutator, - stopped: make(chan struct{}), - jitterSeed: jitterSeed, - l: l, - parentCtx: ctx, - appenderCtx: appenderCtx, - honorTimestamps: honorTimestamps, - sampleLimit: sampleLimit, - bucketLimit: bucketLimit, - labelLimits: labelLimits, - interval: interval, - timeout: timeout, - reportExtraMetrics: reportExtraMetrics, - appendMetadataToWAL: appendMetadataToWAL, + scraper: sc, + buffers: buffers, + cache: cache, + appender: appender, + sampleMutator: sampleMutator, + reportSampleMutator: reportSampleMutator, + stopped: make(chan struct{}), + jitterSeed: jitterSeed, + l: l, + parentCtx: ctx, + appenderCtx: appenderCtx, + honorTimestamps: honorTimestamps, + sampleLimit: sampleLimit, + bucketLimit: bucketLimit, + labelLimits: labelLimits, + interval: interval, + timeout: timeout, + scrapeClassicHistograms: scrapeClassicHistograms, + reportExtraMetrics: reportExtraMetrics, + appendMetadataToWAL: appendMetadataToWAL, } sl.ctx, sl.cancel = context.WithCancel(ctx) @@ -1492,7 +1499,7 @@ type appendErrors struct { } func (sl *scrapeLoop) append(app storage.Appender, b []byte, contentType string, ts time.Time) (total, added, seriesAdded int, err error) { - p, err := textparse.New(b, contentType) + p, err := textparse.New(b, contentType, sl.scrapeClassicHistograms) if err != nil { level.Debug(sl.l).Log( "msg", "Invalid content type on scrape, using prometheus parser as fallback.", diff --git a/scrape/scrape_test.go b/scrape/scrape_test.go index 7f6dea6576..6c45c26b42 100644 --- a/scrape/scrape_test.go +++ b/scrape/scrape_test.go @@ -633,6 +633,7 @@ func TestScrapeLoopStopBeforeRun(t *testing.T) { 0, false, false, + false, nil, false, ) @@ -705,6 +706,7 @@ func TestScrapeLoopStop(t *testing.T) { time.Hour, false, false, + false, nil, false, ) @@ -781,6 +783,7 @@ func TestScrapeLoopRun(t *testing.T) { time.Hour, false, false, + false, nil, false, ) @@ -836,6 +839,7 @@ func TestScrapeLoopRun(t *testing.T) { 100*time.Millisecond, false, false, + false, nil, false, ) @@ -895,6 +899,7 @@ func TestScrapeLoopForcedErr(t *testing.T) { time.Hour, false, false, + false, nil, false, ) @@ -953,6 +958,7 @@ func TestScrapeLoopMetadata(t *testing.T) { 0, false, false, + false, nil, false, ) @@ -1010,6 +1016,7 @@ func simpleTestScrapeLoop(t testing.TB) (context.Context, *scrapeLoop) { 0, false, false, + false, nil, false, ) @@ -1070,6 +1077,7 @@ func TestScrapeLoopFailWithInvalidLabelsAfterRelabel(t *testing.T) { 0, false, false, + false, nil, false, ) @@ -1148,6 +1156,7 @@ func TestScrapeLoopRunCreatesStaleMarkersOnFailedScrape(t *testing.T) { time.Hour, false, false, + false, nil, false, ) @@ -1211,6 +1220,7 @@ func TestScrapeLoopRunCreatesStaleMarkersOnParseFailure(t *testing.T) { time.Hour, false, false, + false, nil, false, ) @@ -1277,6 +1287,7 @@ func TestScrapeLoopCache(t *testing.T) { time.Hour, false, false, + false, nil, false, ) @@ -1360,6 +1371,7 @@ func TestScrapeLoopCacheMemoryExhaustionProtection(t *testing.T) { time.Hour, false, false, + false, nil, false, ) @@ -1474,6 +1486,7 @@ func TestScrapeLoopAppend(t *testing.T) { 0, false, false, + false, nil, false, ) @@ -1563,7 +1576,7 @@ func TestScrapeLoopAppendForConflictingPrefixedLabels(t *testing.T) { return mutateSampleLabels(l, &Target{labels: labels.FromStrings(tc.targetLabels...)}, false, nil) }, nil, - func(ctx context.Context) storage.Appender { return app }, nil, 0, true, 0, 0, nil, 0, 0, false, false, nil, false, + func(ctx context.Context) storage.Appender { return app }, nil, 0, true, 0, 0, nil, 0, 0, false, false, false, nil, false, ) slApp := sl.appender(context.Background()) _, _, _, err := sl.append(slApp, []byte(tc.exposedLabels), "", time.Date(2000, 1, 1, 1, 0, 0, 0, time.UTC)) @@ -1600,6 +1613,7 @@ func TestScrapeLoopAppendCacheEntryButErrNotFound(t *testing.T) { 0, false, false, + false, nil, false, ) @@ -1607,7 +1621,7 @@ func TestScrapeLoopAppendCacheEntryButErrNotFound(t *testing.T) { fakeRef := storage.SeriesRef(1) expValue := float64(1) metric := []byte(`metric{n="1"} 1`) - p, warning := textparse.New(metric, "") + p, warning := textparse.New(metric, "", false) require.NoError(t, warning) var lset labels.Labels @@ -1658,6 +1672,7 @@ func TestScrapeLoopAppendSampleLimit(t *testing.T) { 0, false, false, + false, nil, false, ) @@ -1735,6 +1750,7 @@ func TestScrapeLoop_HistogramBucketLimit(t *testing.T) { 0, false, false, + false, nil, false, ) @@ -1833,6 +1849,7 @@ func TestScrapeLoop_ChangingMetricString(t *testing.T) { 0, false, false, + false, nil, false, ) @@ -1881,6 +1898,7 @@ func TestScrapeLoopAppendStaleness(t *testing.T) { 0, false, false, + false, nil, false, ) @@ -1932,6 +1950,7 @@ func TestScrapeLoopAppendNoStalenessIfTimestamp(t *testing.T) { 0, false, false, + false, nil, false, ) @@ -2043,6 +2062,7 @@ metric_total{n="2"} 2 # {t="2"} 2.0 20000 0, false, false, + false, nil, false, ) @@ -2108,6 +2128,7 @@ func TestScrapeLoopAppendExemplarSeries(t *testing.T) { 0, false, false, + false, nil, false, ) @@ -2160,6 +2181,7 @@ func TestScrapeLoopRunReportsTargetDownOnScrapeError(t *testing.T) { time.Hour, false, false, + false, nil, false, ) @@ -2196,6 +2218,7 @@ func TestScrapeLoopRunReportsTargetDownOnInvalidUTF8(t *testing.T) { time.Hour, false, false, + false, nil, false, ) @@ -2245,6 +2268,7 @@ func TestScrapeLoopAppendGracefullyIfAmendOrOutOfOrderOrOutOfBounds(t *testing.T 0, false, false, + false, nil, false, ) @@ -2290,6 +2314,7 @@ func TestScrapeLoopOutOfBoundsTimeError(t *testing.T) { 0, false, false, + false, nil, false, ) @@ -2562,6 +2587,7 @@ func TestScrapeLoop_RespectTimestamps(t *testing.T) { 0, false, false, + false, nil, false, ) @@ -2603,6 +2629,7 @@ func TestScrapeLoop_DiscardTimestamps(t *testing.T) { 0, false, false, + false, nil, false, ) @@ -2643,6 +2670,7 @@ func TestScrapeLoopDiscardDuplicateLabels(t *testing.T) { 0, false, false, + false, nil, false, ) @@ -2701,6 +2729,7 @@ func TestScrapeLoopDiscardUnnamedMetrics(t *testing.T) { 0, false, false, + false, nil, false, ) @@ -2964,6 +2993,7 @@ func TestScrapeAddFast(t *testing.T) { 0, false, false, + false, nil, false, ) @@ -3050,6 +3080,7 @@ func TestScrapeReportSingleAppender(t *testing.T) { time.Hour, false, false, + false, nil, false, ) @@ -3252,6 +3283,7 @@ func TestScrapeLoopLabelLimit(t *testing.T) { 0, false, false, + false, nil, false, ) diff --git a/web/federate_test.go b/web/federate_test.go index 61ef62f46d..bf7b6fefe7 100644 --- a/web/federate_test.go +++ b/web/federate_test.go @@ -378,7 +378,7 @@ func TestFederationWithNativeHistograms(t *testing.T) { body, err := io.ReadAll(res.Body) require.NoError(t, err) - p := textparse.NewProtobufParser(body) + p := textparse.NewProtobufParser(body, false) var actVec promql.Vector metricFamilies := 0 l := labels.Labels{} From 1ac5131f698ebc60f13fe2727f89b115a41f6558 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Sun, 14 May 2023 07:13:03 +0100 Subject: [PATCH 155/231] Release 2.44.0 (#12364) Signed-off-by: Bryan Boreham --- CHANGELOG.md | 11 ++--------- VERSION | 2 +- web/ui/module/codemirror-promql/package.json | 4 ++-- web/ui/module/lezer-promql/package.json | 2 +- web/ui/package-lock.json | 14 +++++++------- web/ui/react-app/package.json | 4 ++-- 6 files changed, 15 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ebc602800d..f7dc32273e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,14 +1,6 @@ # Changelog -## 2.44.0-rc.2 / 2023-05-07 - -* [ENHANCEMENT] Storage: Optimise buffer used to iterate through samples. #12326 - -## 2.44.0-rc.1 / 2023-05-03 - -* [BUGFIX] Labels: Set after Del would be ignored, which broke some relabeling rules. #12322 - -## 2.44.0-rc.0 / 2023-04-22 +## 2.44.0 / 2023-05-13 This version is built with Go tag `stringlabels`, to use the smaller data structure for Labels that was optional in the previous release. For more @@ -18,6 +10,7 @@ details about this code change see #10991. * [FEATURE] Remote-read: Handle native histograms. #12085, #12192 * [FEATURE] Promtool: Health and readiness check of prometheus server in CLI. #12096 * [FEATURE] PromQL: Add `query_samples_total` metric, the total number of samples loaded by all queries. #12251 +* [ENHANCEMENT] Storage: Optimise buffer used to iterate through samples. #12326 * [ENHANCEMENT] Scrape: Reduce memory allocations on target labels. #12084 * [ENHANCEMENT] PromQL: Use faster heap method for `topk()` / `bottomk()`. #12190 * [ENHANCEMENT] Rules API: Allow filtering by rule name. #12270 diff --git a/VERSION b/VERSION index 3f6719d550..3e197472e2 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.44.0-rc.2 +2.44.0 diff --git a/web/ui/module/codemirror-promql/package.json b/web/ui/module/codemirror-promql/package.json index 048b1aab46..e819319ee6 100644 --- a/web/ui/module/codemirror-promql/package.json +++ b/web/ui/module/codemirror-promql/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/codemirror-promql", - "version": "0.44.0-rc.2", + "version": "0.44.0", "description": "a CodeMirror mode for the PromQL language", "types": "dist/esm/index.d.ts", "module": "dist/esm/index.js", @@ -29,7 +29,7 @@ }, "homepage": "https://github.com/prometheus/prometheus/blob/main/web/ui/module/codemirror-promql/README.md", "dependencies": { - "@prometheus-io/lezer-promql": "0.44.0-rc.2", + "@prometheus-io/lezer-promql": "0.44.0", "lru-cache": "^6.0.0" }, "devDependencies": { diff --git a/web/ui/module/lezer-promql/package.json b/web/ui/module/lezer-promql/package.json index 622e9bea4a..7daa567388 100644 --- a/web/ui/module/lezer-promql/package.json +++ b/web/ui/module/lezer-promql/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/lezer-promql", - "version": "0.44.0-rc.2", + "version": "0.44.0", "description": "lezer-based PromQL grammar", "main": "dist/index.cjs", "type": "module", diff --git a/web/ui/package-lock.json b/web/ui/package-lock.json index 0cd9eb7291..1d3d1d3e34 100644 --- a/web/ui/package-lock.json +++ b/web/ui/package-lock.json @@ -28,10 +28,10 @@ }, "module/codemirror-promql": { "name": "@prometheus-io/codemirror-promql", - "version": "0.44.0-rc.2", + "version": "0.44.0", "license": "Apache-2.0", "dependencies": { - "@prometheus-io/lezer-promql": "0.44.0-rc.2", + "@prometheus-io/lezer-promql": "0.44.0", "lru-cache": "^6.0.0" }, "devDependencies": { @@ -61,7 +61,7 @@ }, "module/lezer-promql": { "name": "@prometheus-io/lezer-promql", - "version": "0.44.0-rc.2", + "version": "0.44.0", "license": "Apache-2.0", "devDependencies": { "@lezer/generator": "^1.2.2", @@ -20763,7 +20763,7 @@ }, "react-app": { "name": "@prometheus-io/app", - "version": "0.44.0-rc.2", + "version": "0.44.0", "dependencies": { "@codemirror/autocomplete": "^6.4.0", "@codemirror/commands": "^6.2.0", @@ -20781,7 +20781,7 @@ "@lezer/lr": "^1.3.1", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.44.0-rc.2", + "@prometheus-io/codemirror-promql": "0.44.0", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", "downshift": "^7.2.0", @@ -23417,7 +23417,7 @@ "@lezer/lr": "^1.3.1", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.44.0-rc.2", + "@prometheus-io/codemirror-promql": "0.44.0", "@testing-library/react-hooks": "^7.0.2", "@types/enzyme": "^3.10.12", "@types/flot": "0.0.32", @@ -23468,7 +23468,7 @@ "@lezer/common": "^1.0.2", "@lezer/highlight": "^1.1.3", "@lezer/lr": "^1.3.1", - "@prometheus-io/lezer-promql": "0.44.0-rc.2", + "@prometheus-io/lezer-promql": "0.44.0", "@types/lru-cache": "^5.1.1", "isomorphic-fetch": "^3.0.0", "lru-cache": "^6.0.0", diff --git a/web/ui/react-app/package.json b/web/ui/react-app/package.json index 1e0c4ed154..fc0d1e5fc3 100644 --- a/web/ui/react-app/package.json +++ b/web/ui/react-app/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/app", - "version": "0.44.0-rc.2", + "version": "0.44.0", "private": true, "dependencies": { "@codemirror/autocomplete": "^6.4.0", @@ -19,7 +19,7 @@ "@lezer/common": "^1.0.2", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.44.0-rc.2", + "@prometheus-io/codemirror-promql": "0.44.0", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", "downshift": "^7.2.0", From c6618729c9054e00c126e03338f670109d8de7af Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gy=C3=B6rgy=20Krajcsovits?= Date: Thu, 11 May 2023 21:02:02 +0200 Subject: [PATCH 156/231] Fix HistogramAppender.Appendable array out of bound error MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The code did not handle spans with 0 length properly. Spans with length zero are now skipped in the comparison. Span index check not done against length-1, since length is a unit32, thus subtracting 1 leads to 2^32, not -1. Fixes and unit tests for both integer and float histograms added. Signed-off-by: György Krajcsovits --- tsdb/chunkenc/float_histogram.go | 8 ++-- tsdb/chunkenc/float_histogram_test.go | 58 ++++++++++++++++++++++++ tsdb/chunkenc/histogram.go | 8 ++-- tsdb/chunkenc/histogram_meta.go | 7 +++ tsdb/chunkenc/histogram_test.go | 63 +++++++++++++++++++++++++++ 5 files changed, 136 insertions(+), 8 deletions(-) diff --git a/tsdb/chunkenc/float_histogram.go b/tsdb/chunkenc/float_histogram.go index 0349de9abe..d49885a1c5 100644 --- a/tsdb/chunkenc/float_histogram.go +++ b/tsdb/chunkenc/float_histogram.go @@ -358,9 +358,9 @@ func counterResetInAnyFloatBucket(oldBuckets []xorValue, newBuckets []float64, o if oldIdx <= newIdx { // Moving ahead old bucket and span by 1 index. - if oldInsideSpanIdx == oldSpans[oldSpanSliceIdx].Length-1 { + if oldInsideSpanIdx+1 >= oldSpans[oldSpanSliceIdx].Length { // Current span is over. - oldSpanSliceIdx++ + oldSpanSliceIdx = nextNonEmptySpanSliceIdx(oldSpanSliceIdx, oldSpans) oldInsideSpanIdx = 0 if oldSpanSliceIdx >= len(oldSpans) { // All old spans are over. @@ -377,9 +377,9 @@ func counterResetInAnyFloatBucket(oldBuckets []xorValue, newBuckets []float64, o if oldIdx > newIdx { // Moving ahead new bucket and span by 1 index. - if newInsideSpanIdx == newSpans[newSpanSliceIdx].Length-1 { + if newInsideSpanIdx+1 >= newSpans[newSpanSliceIdx].Length { // Current span is over. - newSpanSliceIdx++ + newSpanSliceIdx = nextNonEmptySpanSliceIdx(newSpanSliceIdx, newSpans) newInsideSpanIdx = 0 if newSpanSliceIdx >= len(newSpans) { // All new spans are over. diff --git a/tsdb/chunkenc/float_histogram_test.go b/tsdb/chunkenc/float_histogram_test.go index 90c16d1ea9..c662e5ffa1 100644 --- a/tsdb/chunkenc/float_histogram_test.go +++ b/tsdb/chunkenc/float_histogram_test.go @@ -365,6 +365,64 @@ func TestFloatHistogramChunkAppendable(t *testing.T) { } } +func TestFloatHistogramChunkAppendableWithEmptySpan(t *testing.T) { + h1 := &histogram.FloatHistogram{ + Schema: 0, + Count: 21, + Sum: 1234.5, + ZeroThreshold: 0.001, + ZeroCount: 4, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 4}, + {Offset: 0, Length: 0}, + {Offset: 0, Length: 3}, + }, + PositiveBuckets: []float64{1, 2, 1, 1, 1, 1, 1}, + NegativeSpans: []histogram.Span{ + {Offset: 1, Length: 4}, + {Offset: 2, Length: 0}, + {Offset: 2, Length: 3}, + }, + NegativeBuckets: []float64{1, 2, 1, 2, 2, 2, 2}, + } + h2 := &histogram.FloatHistogram{ + Schema: 0, + Count: 37, + Sum: 2345.6, + ZeroThreshold: 0.001, + ZeroCount: 5, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 4}, + {Offset: 0, Length: 0}, + {Offset: 0, Length: 3}, + }, + PositiveBuckets: []float64{1, 3, 1, 2, 1, 1, 1}, + NegativeSpans: []histogram.Span{ + {Offset: 1, Length: 4}, + {Offset: 2, Length: 0}, + {Offset: 2, Length: 3}, + }, + NegativeBuckets: []float64{1, 4, 2, 7, 5, 5, 2}, + } + + c := Chunk(NewFloatHistogramChunk()) + + // Create fresh appender and add the first histogram. + app, err := c.Appender() + require.NoError(t, err) + require.Equal(t, 0, c.NumSamples()) + + app.AppendFloatHistogram(1, h1) + require.Equal(t, 1, c.NumSamples()) + hApp, _ := app.(*FloatHistogramAppender) + + pI, nI, okToAppend, counterReset := hApp.Appendable(h2) + require.Empty(t, pI) + require.Empty(t, nI) + require.True(t, okToAppend) + require.False(t, counterReset) +} + func TestFloatHistogramChunkAppendableGauge(t *testing.T) { c := Chunk(NewFloatHistogramChunk()) diff --git a/tsdb/chunkenc/histogram.go b/tsdb/chunkenc/histogram.go index f9a63d18f7..2350b2af27 100644 --- a/tsdb/chunkenc/histogram.go +++ b/tsdb/chunkenc/histogram.go @@ -386,9 +386,9 @@ func counterResetInAnyBucket(oldBuckets, newBuckets []int64, oldSpans, newSpans if oldIdx <= newIdx { // Moving ahead old bucket and span by 1 index. - if oldInsideSpanIdx == oldSpans[oldSpanSliceIdx].Length-1 { + if oldInsideSpanIdx+1 >= oldSpans[oldSpanSliceIdx].Length { // Current span is over. - oldSpanSliceIdx++ + oldSpanSliceIdx = nextNonEmptySpanSliceIdx(oldSpanSliceIdx, oldSpans) oldInsideSpanIdx = 0 if oldSpanSliceIdx >= len(oldSpans) { // All old spans are over. @@ -405,9 +405,9 @@ func counterResetInAnyBucket(oldBuckets, newBuckets []int64, oldSpans, newSpans if oldIdx > newIdx { // Moving ahead new bucket and span by 1 index. - if newInsideSpanIdx == newSpans[newSpanSliceIdx].Length-1 { + if newInsideSpanIdx+1 >= newSpans[newSpanSliceIdx].Length { // Current span is over. - newSpanSliceIdx++ + newSpanSliceIdx = nextNonEmptySpanSliceIdx(newSpanSliceIdx, newSpans) newInsideSpanIdx = 0 if newSpanSliceIdx >= len(newSpans) { // All new spans are over. diff --git a/tsdb/chunkenc/histogram_meta.go b/tsdb/chunkenc/histogram_meta.go index 027eee1129..7a21bc20bd 100644 --- a/tsdb/chunkenc/histogram_meta.go +++ b/tsdb/chunkenc/histogram_meta.go @@ -487,3 +487,10 @@ func counterResetHint(crh CounterResetHeader, numRead uint16) histogram.CounterR return histogram.UnknownCounterReset } } + +// Handle pathological case of empty span when advancing span idx. +func nextNonEmptySpanSliceIdx(idx int, spans []histogram.Span) (newIdx int) { + for idx++; idx < len(spans) && spans[idx].Length == 0; idx++ { + } + return idx +} diff --git a/tsdb/chunkenc/histogram_test.go b/tsdb/chunkenc/histogram_test.go index 45f31a3b4d..9ef7e24a4c 100644 --- a/tsdb/chunkenc/histogram_test.go +++ b/tsdb/chunkenc/histogram_test.go @@ -14,6 +14,7 @@ package chunkenc import ( + "fmt" "testing" "github.com/stretchr/testify/require" @@ -387,6 +388,64 @@ func TestHistogramChunkAppendable(t *testing.T) { } } +func TestHistogramChunkAppendableWithEmptySpan(t *testing.T) { + h1 := &histogram.Histogram{ + Schema: 0, + Count: 21, + Sum: 1234.5, + ZeroThreshold: 0.001, + ZeroCount: 4, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 4}, + {Offset: 0, Length: 0}, + {Offset: 0, Length: 3}, + }, + PositiveBuckets: []int64{1, 1, -1, 0, 0, 0, 0}, + NegativeSpans: []histogram.Span{ + {Offset: 1, Length: 4}, + {Offset: 2, Length: 0}, + {Offset: 2, Length: 3}, + }, + NegativeBuckets: []int64{1, 1, -1, 1, 0, 0, 0}, + } + h2 := &histogram.Histogram{ + Schema: 0, + Count: 37, + Sum: 2345.6, + ZeroThreshold: 0.001, + ZeroCount: 5, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 4}, + {Offset: 0, Length: 0}, + {Offset: 0, Length: 3}, + }, + PositiveBuckets: []int64{1, 2, -2, 1, -1, 0, 0}, + NegativeSpans: []histogram.Span{ + {Offset: 1, Length: 4}, + {Offset: 2, Length: 0}, + {Offset: 2, Length: 3}, + }, + NegativeBuckets: []int64{1, 3, -2, 5, -2, 0, -3}, + } + + c := Chunk(NewHistogramChunk()) + + // Create fresh appender and add the first histogram. + app, err := c.Appender() + require.NoError(t, err) + require.Equal(t, 0, c.NumSamples()) + + app.AppendHistogram(1, h1) + require.Equal(t, 1, c.NumSamples()) + hApp, _ := app.(*HistogramAppender) + + pI, nI, okToAppend, counterReset := hApp.Appendable(h2) + require.Empty(t, pI) + require.Empty(t, nI) + require.True(t, okToAppend) + require.False(t, counterReset) +} + func TestAtFloatHistogram(t *testing.T) { input := []histogram.Histogram{ { @@ -514,6 +573,10 @@ func TestAtFloatHistogram(t *testing.T) { app, err := chk.Appender() require.NoError(t, err) for i := range input { + if i > 0 { + _, _, okToAppend, _ := app.(*HistogramAppender).Appendable(&input[i]) + require.True(t, okToAppend, fmt.Sprintf("idx: %d", i)) + } app.AppendHistogram(int64(i), &input[i]) } it := chk.Iterator(nil) From 4c898fc4a1840d76ec4ffa1307e4aaf4f50dd7b3 Mon Sep 17 00:00:00 2001 From: Akshay Siwal Date: Mon, 15 May 2023 13:18:30 +0530 Subject: [PATCH 157/231] Update getting_started.md Updating signal for graceful shutdown. Signed-off-by: Akshay Siwal --- docs/getting_started.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting_started.md b/docs/getting_started.md index 11d8d0fb82..e89ac705ee 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -264,4 +264,4 @@ process ID. While Prometheus does have recovery mechanisms in the case that there is an abrupt process failure it is recommend to use the `SIGTERM` signal to cleanly shutdown a Prometheus instance. If you're running on Linux this can be performed -by using `kill -s SIGHUP `, replacing `` with your Prometheus process ID. +by using `kill -s SIGTERM `, replacing `` with your Prometheus process ID. From c3f267d862cf91b6d55a5f2d218c5c6c0bd07ce3 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Wed, 22 Mar 2023 12:11:49 +0000 Subject: [PATCH 158/231] Alerts: more efficient generation of target labels Use a label builder instead of a slice when creating labels for the target alertmanagers. This can be passed directly to `relabel.ProcessBuilder`, skipping a copy. Signed-off-by: Bryan Boreham --- notifier/notifier.go | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/notifier/notifier.go b/notifier/notifier.go index c3b2e5c7e0..d6705edd3e 100644 --- a/notifier/notifier.go +++ b/notifier/notifier.go @@ -701,36 +701,38 @@ func postPath(pre string, v config.AlertmanagerAPIVersion) string { func AlertmanagerFromGroup(tg *targetgroup.Group, cfg *config.AlertmanagerConfig) ([]alertmanager, []alertmanager, error) { var res []alertmanager var droppedAlertManagers []alertmanager + lb := labels.NewBuilder(labels.EmptyLabels()) for _, tlset := range tg.Targets { - lbls := make([]labels.Label, 0, len(tlset)+2+len(tg.Labels)) + lb.Reset(labels.EmptyLabels()) for ln, lv := range tlset { - lbls = append(lbls, labels.Label{Name: string(ln), Value: string(lv)}) + lb.Set(string(ln), string(lv)) } // Set configured scheme as the initial scheme label for overwrite. - lbls = append(lbls, labels.Label{Name: model.SchemeLabel, Value: cfg.Scheme}) - lbls = append(lbls, labels.Label{Name: pathLabel, Value: postPath(cfg.PathPrefix, cfg.APIVersion)}) + lb.Set(model.SchemeLabel, cfg.Scheme) + lb.Set(pathLabel, postPath(cfg.PathPrefix, cfg.APIVersion)) // Combine target labels with target group labels. for ln, lv := range tg.Labels { if _, ok := tlset[ln]; !ok { - lbls = append(lbls, labels.Label{Name: string(ln), Value: string(lv)}) + lb.Set(string(ln), string(lv)) } } - lset, keep := relabel.Process(labels.New(lbls...), cfg.RelabelConfigs...) + preRelabel := lb.Labels() + keep := relabel.ProcessBuilder(lb, cfg.RelabelConfigs...) if !keep { - droppedAlertManagers = append(droppedAlertManagers, alertmanagerLabels{labels.New(lbls...)}) + droppedAlertManagers = append(droppedAlertManagers, alertmanagerLabels{preRelabel}) continue } - addr := lset.Get(model.AddressLabel) + addr := lb.Get(model.AddressLabel) if err := config.CheckTargetAddress(model.LabelValue(addr)); err != nil { return nil, nil, err } - res = append(res, alertmanagerLabels{lset}) + res = append(res, alertmanagerLabels{lb.Labels()}) } return res, droppedAlertManagers, nil } From 3711339a7d4f787b136a2ef1b282dd9b9ea161b9 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Wed, 22 Mar 2023 12:26:17 +0000 Subject: [PATCH 159/231] Alerts: more efficient relabel on Send Re-use `labels.Builder` and use `relabel.ProcessBuilder` to skip a conversion step. Signed-off-by: Bryan Boreham --- notifier/notifier.go | 33 +++++++++++++++------------------ 1 file changed, 15 insertions(+), 18 deletions(-) diff --git a/notifier/notifier.go b/notifier/notifier.go index d6705edd3e..891372c43e 100644 --- a/notifier/notifier.go +++ b/notifier/notifier.go @@ -349,19 +349,6 @@ func (n *Manager) Send(alerts ...*Alert) { n.mtx.Lock() defer n.mtx.Unlock() - // Attach external labels before relabelling and sending. - for _, a := range alerts { - lb := labels.NewBuilder(a.Labels) - - n.opts.ExternalLabels.Range(func(l labels.Label) { - if a.Labels.Get(l.Name) == "" { - lb.Set(l.Name, l.Value) - } - }) - - a.Labels = lb.Labels() - } - alerts = n.relabelAlerts(alerts) if len(alerts) == 0 { return @@ -390,15 +377,25 @@ func (n *Manager) Send(alerts ...*Alert) { n.setMore() } +// Attach external labels and process relabelling rules. func (n *Manager) relabelAlerts(alerts []*Alert) []*Alert { + lb := labels.NewBuilder(labels.EmptyLabels()) var relabeledAlerts []*Alert - for _, alert := range alerts { - labels, keep := relabel.Process(alert.Labels, n.opts.RelabelConfigs...) - if keep { - alert.Labels = labels - relabeledAlerts = append(relabeledAlerts, alert) + for _, a := range alerts { + lb.Reset(a.Labels) + n.opts.ExternalLabels.Range(func(l labels.Label) { + if a.Labels.Get(l.Name) == "" { + lb.Set(l.Name, l.Value) + } + }) + + keep := relabel.ProcessBuilder(lb, n.opts.RelabelConfigs...) + if !keep { + continue } + a.Labels = lb.Labels() + relabeledAlerts = append(relabeledAlerts, a) } return relabeledAlerts } From 0d2108ad7969ed4b6dbd9b76bc812e20d456332f Mon Sep 17 00:00:00 2001 From: Callum Styan Date: Mon, 15 May 2023 12:31:49 -0700 Subject: [PATCH 160/231] [tsdb] re-implement WAL watcher to read via a "notification" channel (#11949) * WIP implement WAL watcher reading via notifications over a channel from the TSDB code Signed-off-by: Callum Styan * Notify via head appenders Commit (finished all WAL logging) rather than on each WAL Log call Signed-off-by: Callum Styan * Fix misspelled Notify plus add a metric for dropped Write notifications Signed-off-by: Callum Styan * Update tests to handle new notification pattern Signed-off-by: Callum Styan * this test maybe needs more time on windows? Signed-off-by: Callum Styan * does this test need more time on windows as well? Signed-off-by: Callum Styan * read timeout is already a time.Duration Signed-off-by: Callum Styan * remove mistakenly commited benchmark data files Signed-off-by: Callum Styan * address some review feedback Signed-off-by: Callum Styan * fix missed changes from previous commit Signed-off-by: Callum Styan * Fix issues from wrapper function Signed-off-by: Callum Styan * try fixing race condition in test by allowing tests to overwrite the read ticker timeout instead of calling the Notify function Signed-off-by: Callum Styan * fix linting Signed-off-by: Callum Styan --------- Signed-off-by: Callum Styan --- cmd/prometheus/main.go | 1 + storage/remote/storage.go | 7 +++ tsdb/db.go | 9 ++++ tsdb/head.go | 2 + tsdb/head_append.go | 4 ++ tsdb/wlog/watcher.go | 100 +++++++++++++++++++++++++++++--------- tsdb/wlog/watcher_test.go | 39 ++++++++------- tsdb/wlog/wlog.go | 6 +++ 8 files changed, 128 insertions(+), 40 deletions(-) diff --git a/cmd/prometheus/main.go b/cmd/prometheus/main.go index c883ab47e9..778b131c8c 100644 --- a/cmd/prometheus/main.go +++ b/cmd/prometheus/main.go @@ -1053,6 +1053,7 @@ func main() { startTimeMargin := int64(2 * time.Duration(cfg.tsdb.MinBlockDuration).Seconds() * 1000) localStorage.Set(db, startTimeMargin) + db.SetWriteNotified(remoteStorage) close(dbOpen) <-cancel return nil diff --git a/storage/remote/storage.go b/storage/remote/storage.go index 88892e62e3..d01f96b3ba 100644 --- a/storage/remote/storage.go +++ b/storage/remote/storage.go @@ -76,6 +76,13 @@ func NewStorage(l log.Logger, reg prometheus.Registerer, stCallback startTimeCal return s } +func (s *Storage) Notify() { + for _, q := range s.rws.queues { + // These should all be non blocking + q.watcher.Notify() + } +} + // ApplyConfig updates the state as the new config requires. func (s *Storage) ApplyConfig(conf *config.Config) error { s.mtx.Lock() diff --git a/tsdb/db.go b/tsdb/db.go index 0cd00c2c56..a0d0a4b260 100644 --- a/tsdb/db.go +++ b/tsdb/db.go @@ -229,6 +229,8 @@ type DB struct { // out-of-order compaction and vertical queries. oooWasEnabled atomic.Bool + writeNotified wlog.WriteNotified + registerer prometheus.Registerer } @@ -802,6 +804,7 @@ func open(dir string, l log.Logger, r prometheus.Registerer, opts *Options, rngs if err != nil { return nil, err } + db.head.writeNotified = db.writeNotified // Register metrics after assigning the head block. db.metrics = newDBMetrics(db, r) @@ -2016,6 +2019,12 @@ func (db *DB) CleanTombstones() (err error) { return nil } +func (db *DB) SetWriteNotified(wn wlog.WriteNotified) { + db.writeNotified = wn + // It's possible we already created the head struct, so we should also set the WN for that. + db.head.writeNotified = wn +} + func isBlockDir(fi fs.DirEntry) bool { if !fi.IsDir() { return false diff --git a/tsdb/head.go b/tsdb/head.go index a3c1353002..aa7a3c1252 100644 --- a/tsdb/head.go +++ b/tsdb/head.go @@ -121,6 +121,8 @@ type Head struct { stats *HeadStats reg prometheus.Registerer + writeNotified wlog.WriteNotified + memTruncationInProcess atomic.Bool } diff --git a/tsdb/head_append.go b/tsdb/head_append.go index 2af01a2d63..060d32b7f7 100644 --- a/tsdb/head_append.go +++ b/tsdb/head_append.go @@ -842,6 +842,10 @@ func (a *headAppender) Commit() (err error) { return errors.Wrap(err, "write to WAL") } + if a.head.writeNotified != nil { + a.head.writeNotified.Notify() + } + // No errors logging to WAL, so pass the exemplars along to the in memory storage. for _, e := range a.exemplars { s := a.head.series.getByID(chunks.HeadSeriesRef(e.ref)) diff --git a/tsdb/wlog/watcher.go b/tsdb/wlog/watcher.go index 31f60feab5..221e9607ca 100644 --- a/tsdb/wlog/watcher.go +++ b/tsdb/wlog/watcher.go @@ -34,12 +34,16 @@ import ( ) const ( - readPeriod = 10 * time.Millisecond checkpointPeriod = 5 * time.Second segmentCheckPeriod = 100 * time.Millisecond consumer = "consumer" ) +var ( + ErrIgnorable = errors.New("ignore me") + readTimeout = 15 * time.Second +) + // WriteTo is an interface used by the Watcher to send the samples it's read // from the WAL on to somewhere else. Functions will be called concurrently // and it is left to the implementer to make sure they are safe. @@ -61,11 +65,17 @@ type WriteTo interface { SeriesReset(int) } +// Used to notifier the watcher that data has been written so that it can read. +type WriteNotified interface { + Notify() +} + type WatcherMetrics struct { recordsRead *prometheus.CounterVec recordDecodeFails *prometheus.CounterVec samplesSentPreTailing *prometheus.CounterVec currentSegment *prometheus.GaugeVec + notificationsSkipped *prometheus.CounterVec } // Watcher watches the TSDB WAL for a given WriteTo. @@ -88,9 +98,11 @@ type Watcher struct { recordDecodeFailsMetric prometheus.Counter samplesSentPreTailing prometheus.Counter currentSegmentMetric prometheus.Gauge + notificationsSkipped prometheus.Counter - quit chan struct{} - done chan struct{} + readNotify chan struct{} + quit chan struct{} + done chan struct{} // For testing, stop when we hit this segment. MaxSegment int @@ -134,6 +146,15 @@ func NewWatcherMetrics(reg prometheus.Registerer) *WatcherMetrics { }, []string{consumer}, ), + notificationsSkipped: prometheus.NewCounterVec( + prometheus.CounterOpts{ + Namespace: "prometheus", + Subsystem: "wal_watcher", + Name: "notifications_skipped_total", + Help: "The number of WAL write notifications that the Watcher has skipped due to already being in a WAL read routine.", + }, + []string{consumer}, + ), } if reg != nil { @@ -141,6 +162,7 @@ func NewWatcherMetrics(reg prometheus.Registerer) *WatcherMetrics { reg.MustRegister(m.recordDecodeFails) reg.MustRegister(m.samplesSentPreTailing) reg.MustRegister(m.currentSegment) + reg.MustRegister(m.notificationsSkipped) } return m @@ -161,13 +183,25 @@ func NewWatcher(metrics *WatcherMetrics, readerMetrics *LiveReaderMetrics, logge sendExemplars: sendExemplars, sendHistograms: sendHistograms, - quit: make(chan struct{}), - done: make(chan struct{}), + readNotify: make(chan struct{}), + quit: make(chan struct{}), + done: make(chan struct{}), MaxSegment: -1, } } +func (w *Watcher) Notify() { + select { + case w.readNotify <- struct{}{}: + return + default: // default so we can exit + // we don't need a buffered channel or any buffering since + // for each notification it recv's the watcher will read until EOF + w.notificationsSkipped.Inc() + } +} + func (w *Watcher) setMetrics() { // Setup the WAL Watchers metrics. We do this here rather than in the // constructor because of the ordering of creating Queue Managers's, @@ -177,6 +211,8 @@ func (w *Watcher) setMetrics() { w.recordDecodeFailsMetric = w.metrics.recordDecodeFails.WithLabelValues(w.name) w.samplesSentPreTailing = w.metrics.samplesSentPreTailing.WithLabelValues(w.name) w.currentSegmentMetric = w.metrics.currentSegment.WithLabelValues(w.name) + w.notificationsSkipped = w.metrics.notificationsSkipped.WithLabelValues(w.name) + } } @@ -262,7 +298,7 @@ func (w *Watcher) Run() error { // On start, after reading the existing WAL for series records, we have a pointer to what is the latest segment. // On subsequent calls to this function, currentSegment will have been incremented and we should open that segment. - if err := w.watch(currentSegment, currentSegment >= lastSegment); err != nil { + if err := w.watch(currentSegment, currentSegment >= lastSegment); err != nil && !errors.Is(err, ErrIgnorable) { return err } @@ -330,6 +366,26 @@ func (w *Watcher) segments(dir string) ([]int, error) { return refs, nil } +func (w *Watcher) readAndHandleError(r *LiveReader, segmentNum int, tail bool, size int64) error { + err := w.readSegment(r, segmentNum, tail) + + // Ignore all errors reading to end of segment whilst replaying the WAL. + if !tail { + if err != nil && errors.Cause(err) != io.EOF { + level.Warn(w.logger).Log("msg", "Ignoring error reading to end of segment, may have dropped data", "segment", segmentNum, "err", err) + } else if r.Offset() != size { + level.Warn(w.logger).Log("msg", "Expected to have read whole segment, may have dropped data", "segment", segmentNum, "read", r.Offset(), "size", size) + } + return ErrIgnorable + } + + // Otherwise, when we are tailing, non-EOFs are fatal. + if errors.Cause(err) != io.EOF { + return err + } + return nil +} + // Use tail true to indicate that the reader is currently on a segment that is // actively being written to. If false, assume it's a full segment and we're // replaying it on start to cache the series records. @@ -342,7 +398,7 @@ func (w *Watcher) watch(segmentNum int, tail bool) error { reader := NewLiveReader(w.logger, w.readerMetrics, segment) - readTicker := time.NewTicker(readPeriod) + readTicker := time.NewTicker(readTimeout) defer readTicker.Stop() checkpointTicker := time.NewTicker(checkpointPeriod) @@ -400,7 +456,6 @@ func (w *Watcher) watch(segmentNum int, tail bool) error { if last <= segmentNum { continue } - err = w.readSegment(reader, segmentNum, tail) // Ignore errors reading to end of segment whilst replaying the WAL. @@ -421,24 +476,23 @@ func (w *Watcher) watch(segmentNum int, tail bool) error { return nil + // we haven't read due to a notification in quite some time, try reading anyways case <-readTicker.C: - err = w.readSegment(reader, segmentNum, tail) - - // Ignore all errors reading to end of segment whilst replaying the WAL. - if !tail { - switch { - case err != nil && errors.Cause(err) != io.EOF: - level.Warn(w.logger).Log("msg", "Ignoring error reading to end of segment, may have dropped data", "segment", segmentNum, "err", err) - case reader.Offset() != size: - level.Warn(w.logger).Log("msg", "Expected to have read whole segment, may have dropped data", "segment", segmentNum, "read", reader.Offset(), "size", size) - } - return nil - } - - // Otherwise, when we are tailing, non-EOFs are fatal. - if errors.Cause(err) != io.EOF { + level.Debug(w.logger).Log("msg", "Watcher is reading the WAL due to timeout, haven't received any write notifications recently", "timeout", readTimeout) + err := w.readAndHandleError(reader, segmentNum, tail, size) + if err != nil { return err } + // still want to reset the ticker so we don't read too often + readTicker.Reset(readTimeout) + + case <-w.readNotify: + err := w.readAndHandleError(reader, segmentNum, tail, size) + if err != nil { + return err + } + // still want to reset the ticker so we don't read too often + readTicker.Reset(readTimeout) } } } diff --git a/tsdb/wlog/watcher_test.go b/tsdb/wlog/watcher_test.go index 530d0ffb4a..94b6a92d12 100644 --- a/tsdb/wlog/watcher_test.go +++ b/tsdb/wlog/watcher_test.go @@ -104,7 +104,7 @@ func (wtm *writeToMock) SeriesReset(index int) { } } -func (wtm *writeToMock) checkNumLabels() int { +func (wtm *writeToMock) checkNumSeries() int { wtm.seriesLock.Lock() defer wtm.seriesLock.Unlock() return len(wtm.seriesSegmentIndexes) @@ -230,9 +230,9 @@ func TestTailSamples(t *testing.T) { expectedExemplars := seriesCount * exemplarsCount expectedHistograms := seriesCount * histogramsCount retry(t, defaultRetryInterval, defaultRetries, func() bool { - return wt.checkNumLabels() >= expectedSeries + return wt.checkNumSeries() >= expectedSeries }) - require.Equal(t, expectedSeries, wt.checkNumLabels(), "did not receive the expected number of series") + require.Equal(t, expectedSeries, wt.checkNumSeries(), "did not receive the expected number of series") require.Equal(t, expectedSamples, wt.samplesAppended, "did not receive the expected number of samples") require.Equal(t, expectedExemplars, wt.exemplarsAppended, "did not receive the expected number of exemplars") require.Equal(t, expectedHistograms, wt.histogramsAppended, "did not receive the expected number of histograms") @@ -290,7 +290,7 @@ func TestReadToEndNoCheckpoint(t *testing.T) { } } require.NoError(t, w.Log(recs...)) - + readTimeout = time.Second _, _, err = Segments(w.Dir()) require.NoError(t, err) @@ -299,11 +299,10 @@ func TestReadToEndNoCheckpoint(t *testing.T) { go watcher.Start() expected := seriesCount - retry(t, defaultRetryInterval, defaultRetries, func() bool { - return wt.checkNumLabels() >= expected - }) + require.Eventually(t, func() bool { + return wt.checkNumSeries() == expected + }, 20*time.Second, 1*time.Second) watcher.Stop() - require.Equal(t, expected, wt.checkNumLabels()) }) } } @@ -383,16 +382,17 @@ func TestReadToEndWithCheckpoint(t *testing.T) { _, _, err = Segments(w.Dir()) require.NoError(t, err) + readTimeout = time.Second wt := newWriteToMock() watcher := NewWatcher(wMetrics, nil, nil, "", wt, dir, false, false) go watcher.Start() expected := seriesCount * 2 - retry(t, defaultRetryInterval, defaultRetries, func() bool { - return wt.checkNumLabels() >= expected - }) + + require.Eventually(t, func() bool { + return wt.checkNumSeries() == expected + }, 10*time.Second, 1*time.Second) watcher.Stop() - require.Equal(t, expected, wt.checkNumLabels()) }) } } @@ -460,10 +460,10 @@ func TestReadCheckpoint(t *testing.T) { expectedSeries := seriesCount retry(t, defaultRetryInterval, defaultRetries, func() bool { - return wt.checkNumLabels() >= expectedSeries + return wt.checkNumSeries() >= expectedSeries }) watcher.Stop() - require.Equal(t, expectedSeries, wt.checkNumLabels()) + require.Equal(t, expectedSeries, wt.checkNumSeries()) }) } } @@ -595,6 +595,7 @@ func TestCheckpointSeriesReset(t *testing.T) { _, _, err = Segments(w.Dir()) require.NoError(t, err) + readTimeout = time.Second wt := newWriteToMock() watcher := NewWatcher(wMetrics, nil, nil, "", wt, dir, false, false) watcher.MaxSegment = -1 @@ -602,9 +603,11 @@ func TestCheckpointSeriesReset(t *testing.T) { expected := seriesCount retry(t, defaultRetryInterval, defaultRetries, func() bool { - return wt.checkNumLabels() >= expected + return wt.checkNumSeries() >= expected }) - require.Equal(t, seriesCount, wt.checkNumLabels()) + require.Eventually(t, func() bool { + return wt.checkNumSeries() == seriesCount + }, 10*time.Second, 1*time.Second) _, err = Checkpoint(log.NewNopLogger(), w, 2, 4, func(x chunks.HeadSeriesRef) bool { return true }, 0) require.NoError(t, err) @@ -621,7 +624,9 @@ func TestCheckpointSeriesReset(t *testing.T) { // If you modify the checkpoint and truncate segment #'s run the test to see how // many series records you end up with and change the last Equals check accordingly // or modify the Equals to Assert(len(wt.seriesLabels) < seriesCount*10) - require.Equal(t, tc.segments, wt.checkNumLabels()) + require.Eventually(t, func() bool { + return wt.checkNumSeries() == tc.segments + }, 20*time.Second, 1*time.Second) }) } } diff --git a/tsdb/wlog/wlog.go b/tsdb/wlog/wlog.go index df8bab53ff..e38cb94cbd 100644 --- a/tsdb/wlog/wlog.go +++ b/tsdb/wlog/wlog.go @@ -188,6 +188,8 @@ type WL struct { compress bool snappyBuf []byte + WriteNotified WriteNotified + metrics *wlMetrics } @@ -343,6 +345,10 @@ func (w *WL) Dir() string { return w.dir } +func (w *WL) SetWriteNotified(wn WriteNotified) { + w.WriteNotified = wn +} + func (w *WL) run() { Loop: for { From 191bf9055bb51093c5f13d2ccef5a18ec4786260 Mon Sep 17 00:00:00 2001 From: zenador Date: Wed, 17 May 2023 03:15:20 +0800 Subject: [PATCH 161/231] Handle more arithmetic operators for native histograms (#12262) Handle more arithmetic operators and aggregators for native histograms This includes operators for multiplication (formerly known as scaling), division, and subtraction. Plus aggregations for average and the avg_over_time function. Stdvar and stddev will (for now) ignore histograms properly (rather than counting them but adding a 0 for them). Signed-off-by: Jeanette Tan --- docs/querying/functions.md | 5 +- docs/querying/operators.md | 26 +- model/histogram/float_histogram.go | 19 +- model/histogram/float_histogram_test.go | 149 ++++++- promql/engine.go | 145 +++++-- promql/engine_test.go | 499 ++++++++++++++++++++++-- promql/functions.go | 60 ++- 7 files changed, 803 insertions(+), 100 deletions(-) diff --git a/docs/querying/functions.md b/docs/querying/functions.md index e92a58937d..c8831c0792 100644 --- a/docs/querying/functions.md +++ b/docs/querying/functions.md @@ -589,8 +589,9 @@ over time and return an instant vector with per-series aggregation results: Note that all values in the specified interval have the same weight in the aggregation even if the values are not equally spaced throughout the interval. -`count_over_time`, `last_over_time`, and `present_over_time` handle native -histograms as expected. All other functions ignore histogram samples. +`avg_over_time`, `sum_over_time`, `count_over_time`, `last_over_time`, and +`present_over_time` handle native histograms as expected. All other functions +ignore histogram samples. ## Trigonometric Functions diff --git a/docs/querying/operators.md b/docs/querying/operators.md index 0cd5368941..c691a0f1a8 100644 --- a/docs/querying/operators.md +++ b/docs/querying/operators.md @@ -318,19 +318,23 @@ histograms is still very limited. Logical/set binary operators work as expected even if histogram samples are involved. They only check for the existence of a vector element and don't change their behavior depending on the sample type of an element (float or -histogram). +histogram). The `count` aggregation operator works similarly. -The binary `+` operator between two native histograms and the `sum` aggregation -operator to aggregate native histograms are fully supported. Even if the -histograms involved have different bucket layouts, the buckets are -automatically converted appropriately so that the operation can be +The binary `+` and `-` operators between two native histograms and the `sum` +and `avg` aggregation operators to aggregate native histograms are fully +supported. Even if the histograms involved have different bucket layouts, the +buckets are automatically converted appropriately so that the operation can be performed. (With the currently supported bucket schemas, that's always -possible.) If either operator has to sum up a mix of histogram samples and +possible.) If either operator has to aggregate a mix of histogram samples and float samples, the corresponding vector element is removed from the output vector entirely. -All other operators do not behave in a meaningful way. They either treat the -histogram sample as if it were a float sample of value 0, or (in case of -arithmetic operations between a scalar and a vector) they leave the histogram -sample unchanged. This behavior will change to a meaningful one before native -histograms are a stable feature. +The binary `*` operator works between a native histogram and a float in any +order, while the binary `/` operator can be used between a native histogram +and a float in that exact order. + +All other operators (and unmentioned cases for the above operators) do not +behave in a meaningful way. They either treat the histogram sample as if it +were a float sample of value 0, or (in case of arithmetic operations between a +scalar and a vector) they leave the histogram sample unchanged. This behavior +will change to a meaningful one before native histograms are a stable feature. diff --git a/model/histogram/float_histogram.go b/model/histogram/float_histogram.go index f95f0051c9..f4ee13facc 100644 --- a/model/histogram/float_histogram.go +++ b/model/histogram/float_histogram.go @@ -159,12 +159,12 @@ func (h *FloatHistogram) ZeroBucket() Bucket[float64] { } } -// Scale scales the FloatHistogram by the provided factor, i.e. it scales all +// Mul multiplies the FloatHistogram by the provided factor, i.e. it scales all // bucket counts including the zero bucket and the count and the sum of // observations. The bucket layout stays the same. This method changes the // receiving histogram directly (rather than acting on a copy). It returns a // pointer to the receiving histogram for convenience. -func (h *FloatHistogram) Scale(factor float64) *FloatHistogram { +func (h *FloatHistogram) Mul(factor float64) *FloatHistogram { h.ZeroCount *= factor h.Count *= factor h.Sum *= factor @@ -177,6 +177,21 @@ func (h *FloatHistogram) Scale(factor float64) *FloatHistogram { return h } +// Div works like Scale but divides instead of multiplies. +// When dividing by 0, everything will be set to Inf. +func (h *FloatHistogram) Div(scalar float64) *FloatHistogram { + h.ZeroCount /= scalar + h.Count /= scalar + h.Sum /= scalar + for i := range h.PositiveBuckets { + h.PositiveBuckets[i] /= scalar + } + for i := range h.NegativeBuckets { + h.NegativeBuckets[i] /= scalar + } + return h +} + // Add adds the provided other histogram to the receiving histogram. Count, Sum, // and buckets from the other histogram are added to the corresponding // components of the receiving histogram. Buckets in the other histogram that do diff --git a/model/histogram/float_histogram_test.go b/model/histogram/float_histogram_test.go index 58aad9645e..242ef4c92c 100644 --- a/model/histogram/float_histogram_test.go +++ b/model/histogram/float_histogram_test.go @@ -15,12 +15,13 @@ package histogram import ( "fmt" + "math" "testing" "github.com/stretchr/testify/require" ) -func TestFloatHistogramScale(t *testing.T) { +func TestFloatHistogramMul(t *testing.T) { cases := []struct { name string in *FloatHistogram @@ -33,6 +34,30 @@ func TestFloatHistogramScale(t *testing.T) { 3.1415, &FloatHistogram{}, }, + { + "zero multiplier", + &FloatHistogram{ + ZeroThreshold: 0.01, + ZeroCount: 5.5, + Count: 3493.3, + Sum: 2349209.324, + PositiveSpans: []Span{{-2, 1}, {2, 3}}, + PositiveBuckets: []float64{1, 3.3, 4.2, 0.1}, + NegativeSpans: []Span{{3, 2}, {3, 2}}, + NegativeBuckets: []float64{3.1, 3, 1.234e5, 1000}, + }, + 0, + &FloatHistogram{ + ZeroThreshold: 0.01, + ZeroCount: 0, + Count: 0, + Sum: 0, + PositiveSpans: []Span{{-2, 1}, {2, 3}}, + PositiveBuckets: []float64{0, 0, 0, 0}, + NegativeSpans: []Span{{3, 2}, {3, 2}}, + NegativeBuckets: []float64{0, 0, 0, 0}, + }, + }, { "no-op", &FloatHistogram{ @@ -81,17 +106,137 @@ func TestFloatHistogramScale(t *testing.T) { NegativeBuckets: []float64{6.2, 6, 1.234e5 * 2, 2000}, }, }, + { + "triple", + &FloatHistogram{ + ZeroThreshold: 0.01, + ZeroCount: 11, + Count: 30, + Sum: 23, + PositiveSpans: []Span{{-2, 2}, {1, 3}}, + PositiveBuckets: []float64{1, 0, 3, 4, 7}, + NegativeSpans: []Span{{3, 2}, {3, 2}}, + NegativeBuckets: []float64{3, 1, 5, 6}, + }, + 3, + &FloatHistogram{ + ZeroThreshold: 0.01, + ZeroCount: 33, + Count: 90, + Sum: 69, + PositiveSpans: []Span{{-2, 2}, {1, 3}}, + PositiveBuckets: []float64{3, 0, 9, 12, 21}, + NegativeSpans: []Span{{3, 2}, {3, 2}}, + NegativeBuckets: []float64{9, 3, 15, 18}, + }, + }, } for _, c := range cases { t.Run(c.name, func(t *testing.T) { - require.Equal(t, c.expected, c.in.Scale(c.scale)) + require.Equal(t, c.expected, c.in.Mul(c.scale)) // Has it also happened in-place? require.Equal(t, c.expected, c.in) }) } } +func TestFloatHistogramDiv(t *testing.T) { + cases := []struct { + name string + fh *FloatHistogram + s float64 + expected *FloatHistogram + }{ + { + "zero value", + &FloatHistogram{}, + 3.1415, + &FloatHistogram{}, + }, + { + "zero divisor", + &FloatHistogram{ + ZeroThreshold: 0.01, + ZeroCount: 5.5, + Count: 3493.3, + Sum: 2349209.324, + PositiveSpans: []Span{{-2, 1}, {2, 3}}, + PositiveBuckets: []float64{1, 3.3, 4.2, 0.1}, + NegativeSpans: []Span{{3, 2}, {3, 2}}, + NegativeBuckets: []float64{3.1, 3, 1.234e5, 1000}, + }, + 0, + &FloatHistogram{ + ZeroThreshold: 0.01, + ZeroCount: math.Inf(1), + Count: math.Inf(1), + Sum: math.Inf(1), + PositiveSpans: []Span{{-2, 1}, {2, 3}}, + PositiveBuckets: []float64{math.Inf(1), math.Inf(1), math.Inf(1), math.Inf(1)}, + NegativeSpans: []Span{{3, 2}, {3, 2}}, + NegativeBuckets: []float64{math.Inf(1), math.Inf(1), math.Inf(1), math.Inf(1)}, + }, + }, + { + "no-op", + &FloatHistogram{ + ZeroThreshold: 0.01, + ZeroCount: 5.5, + Count: 3493.3, + Sum: 2349209.324, + PositiveSpans: []Span{{-2, 1}, {2, 3}}, + PositiveBuckets: []float64{1, 3.3, 4.2, 0.1}, + NegativeSpans: []Span{{3, 2}, {3, 2}}, + NegativeBuckets: []float64{3.1, 3, 1.234e5, 1000}, + }, + 1, + &FloatHistogram{ + ZeroThreshold: 0.01, + ZeroCount: 5.5, + Count: 3493.3, + Sum: 2349209.324, + PositiveSpans: []Span{{-2, 1}, {2, 3}}, + PositiveBuckets: []float64{1, 3.3, 4.2, 0.1}, + NegativeSpans: []Span{{3, 2}, {3, 2}}, + NegativeBuckets: []float64{3.1, 3, 1.234e5, 1000}, + }, + }, + { + "half", + &FloatHistogram{ + ZeroThreshold: 0.01, + ZeroCount: 11, + Count: 30, + Sum: 23, + PositiveSpans: []Span{{-2, 2}, {1, 3}}, + PositiveBuckets: []float64{1, 0, 3, 4, 7}, + NegativeSpans: []Span{{3, 2}, {3, 2}}, + NegativeBuckets: []float64{3, 1, 5, 6}, + }, + 2, + &FloatHistogram{ + ZeroThreshold: 0.01, + ZeroCount: 5.5, + Count: 15, + Sum: 11.5, + PositiveSpans: []Span{{-2, 2}, {1, 3}}, + PositiveBuckets: []float64{0.5, 0, 1.5, 2, 3.5}, + NegativeSpans: []Span{{3, 2}, {3, 2}}, + NegativeBuckets: []float64{1.5, 0.5, 2.5, 3}, + }, + }, + } + + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + require.Equal(t, c.expected, c.fh.Div(c.s)) + // Has it also happened in-place? + require.Equal(t, c.expected, c.fh) + }) + } +} + func TestFloatHistogramDetectReset(t *testing.T) { cases := []struct { name string diff --git a/promql/engine.go b/promql/engine.go index ae46f60054..8a64fdf394 100644 --- a/promql/engine.go +++ b/promql/engine.go @@ -2263,14 +2263,11 @@ func (ev *evaluator) VectorBinop(op parser.ItemType, lhs, rhs Vector, matching * insertedSigs[insertSig] = struct{}{} } - if (hl != nil && hr != nil) || (hl == nil && hr == nil) { - // Both lhs and rhs are of same type. - enh.Out = append(enh.Out, Sample{ - Metric: metric, - F: floatValue, - H: histogramValue, - }) - } + enh.Out = append(enh.Out, Sample{ + Metric: metric, + F: floatValue, + H: histogramValue, + }) } return enh.Out } @@ -2337,28 +2334,33 @@ func resultMetric(lhs, rhs labels.Labels, op parser.ItemType, matching *parser.V // VectorscalarBinop evaluates a binary operation between a Vector and a Scalar. func (ev *evaluator) VectorscalarBinop(op parser.ItemType, lhs Vector, rhs Scalar, swap, returnBool bool, enh *EvalNodeHelper) Vector { for _, lhsSample := range lhs { - lv, rv := lhsSample.F, rhs.V + lf, rf := lhsSample.F, rhs.V + var rh *histogram.FloatHistogram + lh := lhsSample.H // lhs always contains the Vector. If the original position was different // swap for calculating the value. if swap { - lv, rv = rv, lv + lf, rf = rf, lf + lh, rh = rh, lh } - value, _, keep := vectorElemBinop(op, lv, rv, nil, nil) + float, histogram, keep := vectorElemBinop(op, lf, rf, lh, rh) // Catch cases where the scalar is the LHS in a scalar-vector comparison operation. // We want to always keep the vector element value as the output value, even if it's on the RHS. if op.IsComparisonOperator() && swap { - value = rv + float = rf + histogram = rh } if returnBool { if keep { - value = 1.0 + float = 1.0 } else { - value = 0.0 + float = 0.0 } keep = true } if keep { - lhsSample.F = value + lhsSample.F = float + lhsSample.H = histogram if shouldDropMetricName(op) || returnBool { lhsSample.Metric = enh.DropMetricName(lhsSample.Metric) } @@ -2413,16 +2415,33 @@ func vectorElemBinop(op parser.ItemType, lhs, rhs float64, hlhs, hrhs *histogram // The histogram being added must have the larger schema // code (i.e. the higher resolution). if hrhs.Schema >= hlhs.Schema { - return 0, hlhs.Copy().Add(hrhs), true + return 0, hlhs.Copy().Add(hrhs).Compact(0), true } - return 0, hrhs.Copy().Add(hlhs), true + return 0, hrhs.Copy().Add(hlhs).Compact(0), true } return lhs + rhs, nil, true case parser.SUB: + if hlhs != nil && hrhs != nil { + // The histogram being subtracted must have the larger schema + // code (i.e. the higher resolution). + if hrhs.Schema >= hlhs.Schema { + return 0, hlhs.Copy().Sub(hrhs).Compact(0), true + } + return 0, hrhs.Copy().Mul(-1).Add(hlhs).Compact(0), true + } return lhs - rhs, nil, true case parser.MUL: + if hlhs != nil && hrhs == nil { + return 0, hlhs.Copy().Mul(rhs), true + } + if hlhs == nil && hrhs != nil { + return 0, hrhs.Copy().Mul(lhs), true + } return lhs * rhs, nil, true case parser.DIV: + if hlhs != nil && hrhs == nil { + return 0, hlhs.Copy().Div(rhs), true + } return lhs / rhs, nil, true case parser.POW: return math.Pow(lhs, rhs), nil, true @@ -2452,7 +2471,8 @@ type groupedAggregation struct { labels labels.Labels floatValue float64 histogramValue *histogram.FloatHistogram - mean float64 + floatMean float64 + histogramMean *histogram.FloatHistogram groupCount int heap vectorByValueHeap reverseHeap vectorByReverseValueHeap @@ -2536,7 +2556,7 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without newAgg := &groupedAggregation{ labels: m, floatValue: s.F, - mean: s.F, + floatMean: s.F, groupCount: 1, } switch { @@ -2545,6 +2565,11 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without case op == parser.SUM: newAgg.histogramValue = s.H.Copy() newAgg.hasHistogram = true + case op == parser.AVG: + newAgg.histogramMean = s.H.Copy() + newAgg.hasHistogram = true + case op == parser.STDVAR || op == parser.STDDEV: + newAgg.groupCount = 0 } result[groupingKey] = newAgg @@ -2589,9 +2614,7 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without if s.H.Schema >= group.histogramValue.Schema { group.histogramValue.Add(s.H) } else { - h := s.H.Copy() - h.Add(group.histogramValue) - group.histogramValue = h + group.histogramValue = s.H.Copy().Add(group.histogramValue) } } // Otherwise the aggregation contained floats @@ -2604,25 +2627,46 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without case parser.AVG: group.groupCount++ - if math.IsInf(group.mean, 0) { - if math.IsInf(s.F, 0) && (group.mean > 0) == (s.F > 0) { - // The `mean` and `s.V` values are `Inf` of the same sign. They - // can't be subtracted, but the value of `mean` is correct - // already. - break + if s.H != nil { + group.hasHistogram = true + if group.histogramMean != nil { + left := s.H.Copy().Div(float64(group.groupCount)) + right := group.histogramMean.Copy().Div(float64(group.groupCount)) + // The histogram being added/subtracted must have + // an equal or larger schema. + if s.H.Schema >= group.histogramMean.Schema { + toAdd := right.Mul(-1).Add(left) + group.histogramMean.Add(toAdd) + } else { + toAdd := left.Sub(right) + group.histogramMean = toAdd.Add(group.histogramMean) + } } - if !math.IsInf(s.F, 0) && !math.IsNaN(s.F) { - // At this stage, the mean is an infinite. If the added - // value is neither an Inf or a Nan, we can keep that mean - // value. - // This is required because our calculation below removes - // the mean value, which would look like Inf += x - Inf and - // end up as a NaN. - break + // Otherwise the aggregation contained floats + // previously and will be invalid anyway. No + // point in copying the histogram in that case. + } else { + group.hasFloat = true + if math.IsInf(group.floatMean, 0) { + if math.IsInf(s.F, 0) && (group.floatMean > 0) == (s.F > 0) { + // The `floatMean` and `s.F` values are `Inf` of the same sign. They + // can't be subtracted, but the value of `floatMean` is correct + // already. + break + } + if !math.IsInf(s.F, 0) && !math.IsNaN(s.F) { + // At this stage, the mean is an infinite. If the added + // value is neither an Inf or a Nan, we can keep that mean + // value. + // This is required because our calculation below removes + // the mean value, which would look like Inf += x - Inf and + // end up as a NaN. + break + } } + // Divide each side of the `-` by `group.groupCount` to avoid float64 overflows. + group.floatMean += s.F/float64(group.groupCount) - group.floatMean/float64(group.groupCount) } - // Divide each side of the `-` by `group.groupCount` to avoid float64 overflows. - group.mean += s.F/float64(group.groupCount) - group.mean/float64(group.groupCount) case parser.GROUP: // Do nothing. Required to avoid the panic in `default:` below. @@ -2641,10 +2685,12 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without group.groupCount++ case parser.STDVAR, parser.STDDEV: - group.groupCount++ - delta := s.F - group.mean - group.mean += delta / float64(group.groupCount) - group.floatValue += delta * (s.F - group.mean) + if s.H == nil { // Ignore native histograms. + group.groupCount++ + delta := s.F - group.floatMean + group.floatMean += delta / float64(group.groupCount) + group.floatValue += delta * (s.F - group.floatMean) + } case parser.TOPK: // We build a heap of up to k elements, with the smallest element at heap[0]. @@ -2696,7 +2742,16 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without for _, aggr := range orderedResult { switch op { case parser.AVG: - aggr.floatValue = aggr.mean + if aggr.hasFloat && aggr.hasHistogram { + // We cannot aggregate histogram sample with a float64 sample. + // TODO(zenador): Issue warning when plumbing is in place. + continue + } + if aggr.hasHistogram { + aggr.histogramValue = aggr.histogramMean.Compact(0) + } else { + aggr.floatValue = aggr.floatMean + } case parser.COUNT, parser.COUNT_VALUES: aggr.floatValue = float64(aggr.groupCount) @@ -2739,8 +2794,12 @@ func (ev *evaluator) aggregation(op parser.ItemType, grouping []string, without case parser.SUM: if aggr.hasFloat && aggr.hasHistogram { // We cannot aggregate histogram sample with a float64 sample. + // TODO(zenador): Issue warning when plumbing is in place. continue } + if aggr.hasHistogram { + aggr.histogramValue.Compact(0) + } default: // For other aggregations, we already have the right value. } diff --git a/promql/engine_test.go b/promql/engine_test.go index cfe3694dd9..72cbf91533 100644 --- a/promql/engine_test.go +++ b/promql/engine_test.go @@ -3966,21 +3966,23 @@ func TestNativeHistogram_HistogramFraction(t *testing.T) { } } -func TestNativeHistogram_Sum_Count_AddOperator(t *testing.T) { +func TestNativeHistogram_Sum_Count_Add_AvgOperator(t *testing.T) { // TODO(codesome): Integrate histograms into the PromQL testing framework // and write more tests there. cases := []struct { - histograms []histogram.Histogram - expected histogram.FloatHistogram + histograms []histogram.Histogram + expected histogram.FloatHistogram + expectedAvg histogram.FloatHistogram }{ { histograms: []histogram.Histogram{ { - Schema: 0, - Count: 21, - Sum: 1234.5, - ZeroThreshold: 0.001, - ZeroCount: 4, + CounterResetHint: histogram.GaugeType, + Schema: 0, + Count: 21, + Sum: 1234.5, + ZeroThreshold: 0.001, + ZeroCount: 4, PositiveSpans: []histogram.Span{ {Offset: 0, Length: 2}, {Offset: 1, Length: 2}, @@ -3992,6 +3994,182 @@ func TestNativeHistogram_Sum_Count_AddOperator(t *testing.T) { }, NegativeBuckets: []int64{2, 2, -3, 8}, }, + { + CounterResetHint: histogram.GaugeType, + Schema: 0, + Count: 36, + Sum: 2345.6, + ZeroThreshold: 0.001, + ZeroCount: 5, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 4}, + {Offset: 0, Length: 0}, + {Offset: 0, Length: 3}, + }, + PositiveBuckets: []int64{1, 2, -2, 1, -1, 0, 0}, + NegativeSpans: []histogram.Span{ + {Offset: 1, Length: 4}, + {Offset: 2, Length: 0}, + {Offset: 2, Length: 3}, + }, + NegativeBuckets: []int64{1, 3, -2, 5, -2, 0, -3}, + }, + { + CounterResetHint: histogram.GaugeType, + Schema: 0, + Count: 36, + Sum: 1111.1, + ZeroThreshold: 0.001, + ZeroCount: 5, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 4}, + {Offset: 0, Length: 0}, + {Offset: 0, Length: 3}, + }, + PositiveBuckets: []int64{1, 2, -2, 1, -1, 0, 0}, + NegativeSpans: []histogram.Span{ + {Offset: 1, Length: 4}, + {Offset: 2, Length: 0}, + {Offset: 2, Length: 3}, + }, + NegativeBuckets: []int64{1, 3, -2, 5, -2, 0, -3}, + }, + { + CounterResetHint: histogram.GaugeType, + Schema: 1, // Everything is 0 just to make the count 4 so avg has nicer numbers. + }, + }, + expected: histogram.FloatHistogram{ + CounterResetHint: histogram.GaugeType, + Schema: 0, + ZeroThreshold: 0.001, + ZeroCount: 14, + Count: 93, + Sum: 4691.2, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 7}, + }, + PositiveBuckets: []float64{3, 8, 2, 5, 3, 2, 2}, + NegativeSpans: []histogram.Span{ + {Offset: 0, Length: 6}, + {Offset: 3, Length: 3}, + }, + NegativeBuckets: []float64{2, 6, 8, 4, 15, 9, 10, 10, 4}, + }, + expectedAvg: histogram.FloatHistogram{ + CounterResetHint: histogram.GaugeType, + Schema: 0, + ZeroThreshold: 0.001, + ZeroCount: 3.5, + Count: 23.25, + Sum: 1172.8, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 7}, + }, + PositiveBuckets: []float64{0.75, 2, 0.5, 1.25, 0.75, 0.5, 0.5}, + NegativeSpans: []histogram.Span{ + {Offset: 0, Length: 6}, + {Offset: 3, Length: 3}, + }, + NegativeBuckets: []float64{0.5, 1.5, 2, 1, 3.75, 2.25, 2.5, 2.5, 1}, + }, + }, + } + + idx0 := int64(0) + for _, c := range cases { + for _, floatHisto := range []bool{true, false} { + t.Run(fmt.Sprintf("floatHistogram=%t %d", floatHisto, idx0), func(t *testing.T) { + test, err := NewTest(t, "") + require.NoError(t, err) + t.Cleanup(test.Close) + + seriesName := "sparse_histogram_series" + seriesNameOverTime := "sparse_histogram_series_over_time" + + engine := test.QueryEngine() + + ts := idx0 * int64(10*time.Minute/time.Millisecond) + app := test.Storage().Appender(context.TODO()) + for idx1, h := range c.histograms { + lbls := labels.FromStrings("__name__", seriesName, "idx", fmt.Sprintf("%d", idx1)) + // Since we mutate h later, we need to create a copy here. + if floatHisto { + _, err = app.AppendHistogram(0, lbls, ts, nil, h.Copy().ToFloat()) + } else { + _, err = app.AppendHistogram(0, lbls, ts, h.Copy(), nil) + } + require.NoError(t, err) + + lbls = labels.FromStrings("__name__", seriesNameOverTime) + newTs := ts + int64(idx1)*int64(time.Minute/time.Millisecond) + // Since we mutate h later, we need to create a copy here. + if floatHisto { + _, err = app.AppendHistogram(0, lbls, newTs, nil, h.Copy().ToFloat()) + } else { + _, err = app.AppendHistogram(0, lbls, newTs, h.Copy(), nil) + } + require.NoError(t, err) + } + require.NoError(t, app.Commit()) + + queryAndCheck := func(queryString string, ts int64, exp Vector) { + qry, err := engine.NewInstantQuery(test.context, test.Queryable(), nil, queryString, timestamp.Time(ts)) + require.NoError(t, err) + + res := qry.Exec(test.Context()) + require.NoError(t, res.Err) + + vector, err := res.Vector() + require.NoError(t, err) + + require.Equal(t, exp, vector) + } + + // sum(). + queryString := fmt.Sprintf("sum(%s)", seriesName) + queryAndCheck(queryString, ts, []Sample{{T: ts, H: &c.expected, Metric: labels.EmptyLabels()}}) + + // + operator. + queryString = fmt.Sprintf(`%s{idx="0"}`, seriesName) + for idx := 1; idx < len(c.histograms); idx++ { + queryString += fmt.Sprintf(` + ignoring(idx) %s{idx="%d"}`, seriesName, idx) + } + queryAndCheck(queryString, ts, []Sample{{T: ts, H: &c.expected, Metric: labels.EmptyLabels()}}) + + // count(). + queryString = fmt.Sprintf("count(%s)", seriesName) + queryAndCheck(queryString, ts, []Sample{{T: ts, F: 4, Metric: labels.EmptyLabels()}}) + + // avg(). + queryString = fmt.Sprintf("avg(%s)", seriesName) + queryAndCheck(queryString, ts, []Sample{{T: ts, H: &c.expectedAvg, Metric: labels.EmptyLabels()}}) + + offset := int64(len(c.histograms) - 1) + newTs := ts + offset*int64(time.Minute/time.Millisecond) + + // sum_over_time(). + queryString = fmt.Sprintf("sum_over_time(%s[%dm:1m])", seriesNameOverTime, offset) + queryAndCheck(queryString, newTs, []Sample{{T: newTs, H: &c.expected, Metric: labels.EmptyLabels()}}) + + // avg_over_time(). + queryString = fmt.Sprintf("avg_over_time(%s[%dm:1m])", seriesNameOverTime, offset) + queryAndCheck(queryString, newTs, []Sample{{T: newTs, H: &c.expectedAvg, Metric: labels.EmptyLabels()}}) + }) + idx0++ + } + } +} + +func TestNativeHistogram_SubOperator(t *testing.T) { + // TODO(codesome): Integrate histograms into the PromQL testing framework + // and write more tests there. + cases := []struct { + histograms []histogram.Histogram + expected histogram.FloatHistogram + }{ + { + histograms: []histogram.Histogram{ { Schema: 0, Count: 36, @@ -4011,10 +4189,117 @@ func TestNativeHistogram_Sum_Count_AddOperator(t *testing.T) { }, NegativeBuckets: []int64{1, 3, -2, 5, -2, 0, -3}, }, + { + Schema: 0, + Count: 11, + Sum: 1234.5, + ZeroThreshold: 0.001, + ZeroCount: 3, + PositiveSpans: []histogram.Span{ + {Offset: 1, Length: 2}, + }, + PositiveBuckets: []int64{2, -1}, + NegativeSpans: []histogram.Span{ + {Offset: 2, Length: 2}, + }, + NegativeBuckets: []int64{3, -1}, + }, + }, + expected: histogram.FloatHistogram{ + Schema: 0, + Count: 25, + Sum: 1111.1, + ZeroThreshold: 0.001, + ZeroCount: 2, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 2}, + {Offset: 1, Length: 4}, + }, + PositiveBuckets: []float64{1, 1, 2, 1, 1, 1}, + NegativeSpans: []histogram.Span{ + {Offset: 1, Length: 2}, + {Offset: 1, Length: 1}, + {Offset: 4, Length: 3}, + }, + NegativeBuckets: []float64{1, 1, 7, 5, 5, 2}, + }, + }, + { + histograms: []histogram.Histogram{ { Schema: 0, Count: 36, - Sum: 1111.1, + Sum: 2345.6, + ZeroThreshold: 0.001, + ZeroCount: 5, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 4}, + {Offset: 0, Length: 0}, + {Offset: 0, Length: 3}, + }, + PositiveBuckets: []int64{1, 2, -2, 1, -1, 0, 0}, + NegativeSpans: []histogram.Span{ + {Offset: 1, Length: 4}, + {Offset: 2, Length: 0}, + {Offset: 2, Length: 3}, + }, + NegativeBuckets: []int64{1, 3, -2, 5, -2, 0, -3}, + }, + { + Schema: 1, + Count: 11, + Sum: 1234.5, + ZeroThreshold: 0.001, + ZeroCount: 3, + PositiveSpans: []histogram.Span{ + {Offset: 1, Length: 2}, + }, + PositiveBuckets: []int64{2, -1}, + NegativeSpans: []histogram.Span{ + {Offset: 2, Length: 2}, + }, + NegativeBuckets: []int64{3, -1}, + }, + }, + expected: histogram.FloatHistogram{ + Schema: 0, + Count: 25, + Sum: 1111.1, + ZeroThreshold: 0.001, + ZeroCount: 2, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 1}, + {Offset: 1, Length: 5}, + }, + PositiveBuckets: []float64{1, 1, 2, 1, 1, 1}, + NegativeSpans: []histogram.Span{ + {Offset: 1, Length: 4}, + {Offset: 4, Length: 3}, + }, + NegativeBuckets: []float64{-2, 2, 2, 7, 5, 5, 2}, + }, + }, + { + histograms: []histogram.Histogram{ + { + Schema: 1, + Count: 11, + Sum: 1234.5, + ZeroThreshold: 0.001, + ZeroCount: 3, + PositiveSpans: []histogram.Span{ + {Offset: 1, Length: 2}, + }, + PositiveBuckets: []int64{2, -1}, + NegativeSpans: []histogram.Span{ + {Offset: 2, Length: 2}, + }, + NegativeBuckets: []int64{3, -1}, + }, + { + Schema: 0, + Count: 36, + Sum: 2345.6, ZeroThreshold: 0.001, ZeroCount: 5, PositiveSpans: []histogram.Span{ @@ -4033,21 +4318,20 @@ func TestNativeHistogram_Sum_Count_AddOperator(t *testing.T) { }, expected: histogram.FloatHistogram{ Schema: 0, + Count: -25, + Sum: -1111.1, ZeroThreshold: 0.001, - ZeroCount: 14, - Count: 93, - Sum: 4691.2, + ZeroCount: -2, PositiveSpans: []histogram.Span{ - {Offset: 0, Length: 3}, - {Offset: 0, Length: 4}, + {Offset: 0, Length: 1}, + {Offset: 1, Length: 5}, }, - PositiveBuckets: []float64{3, 8, 2, 5, 3, 2, 2}, + PositiveBuckets: []float64{-1, -1, -2, -1, -1, -1}, NegativeSpans: []histogram.Span{ - {Offset: 0, Length: 4}, - {Offset: 0, Length: 2}, - {Offset: 3, Length: 3}, + {Offset: 1, Length: 4}, + {Offset: 4, Length: 3}, }, - NegativeBuckets: []float64{2, 6, 8, 4, 15, 9, 10, 10, 4}, + NegativeBuckets: []float64{2, -2, -2, -7, -5, -5, -2}, }, }, } @@ -4091,20 +4375,177 @@ func TestNativeHistogram_Sum_Count_AddOperator(t *testing.T) { require.Equal(t, exp, vector) } - // sum(). - queryString := fmt.Sprintf("sum(%s)", seriesName) - queryAndCheck(queryString, []Sample{{T: ts, H: &c.expected, Metric: labels.EmptyLabels()}}) - - // + operator. - queryString = fmt.Sprintf(`%s{idx="0"}`, seriesName) + // - operator. + queryString := fmt.Sprintf(`%s{idx="0"}`, seriesName) for idx := 1; idx < len(c.histograms); idx++ { - queryString += fmt.Sprintf(` + ignoring(idx) %s{idx="%d"}`, seriesName, idx) + queryString += fmt.Sprintf(` - ignoring(idx) %s{idx="%d"}`, seriesName, idx) } queryAndCheck(queryString, []Sample{{T: ts, H: &c.expected, Metric: labels.EmptyLabels()}}) + }) + idx0++ + } + } +} - // count(). - queryString = fmt.Sprintf("count(%s)", seriesName) - queryAndCheck(queryString, []Sample{{T: ts, F: 3, Metric: labels.EmptyLabels()}}) +func TestNativeHistogram_MulDivOperator(t *testing.T) { + // TODO(codesome): Integrate histograms into the PromQL testing framework + // and write more tests there. + originalHistogram := histogram.Histogram{ + Schema: 0, + Count: 21, + Sum: 33, + ZeroThreshold: 0.001, + ZeroCount: 3, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 3}, + }, + PositiveBuckets: []int64{3, 0, 0}, + NegativeSpans: []histogram.Span{ + {Offset: 0, Length: 3}, + }, + NegativeBuckets: []int64{3, 0, 0}, + } + + cases := []struct { + scalar float64 + histogram histogram.Histogram + expectedMul histogram.FloatHistogram + expectedDiv histogram.FloatHistogram + }{ + { + scalar: 3, + histogram: originalHistogram, + expectedMul: histogram.FloatHistogram{ + Schema: 0, + Count: 63, + Sum: 99, + ZeroThreshold: 0.001, + ZeroCount: 9, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 3}, + }, + PositiveBuckets: []float64{9, 9, 9}, + NegativeSpans: []histogram.Span{ + {Offset: 0, Length: 3}, + }, + NegativeBuckets: []float64{9, 9, 9}, + }, + expectedDiv: histogram.FloatHistogram{ + Schema: 0, + Count: 7, + Sum: 11, + ZeroThreshold: 0.001, + ZeroCount: 1, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 3}, + }, + PositiveBuckets: []float64{1, 1, 1}, + NegativeSpans: []histogram.Span{ + {Offset: 0, Length: 3}, + }, + NegativeBuckets: []float64{1, 1, 1}, + }, + }, + { + scalar: 0, + histogram: originalHistogram, + expectedMul: histogram.FloatHistogram{ + Schema: 0, + Count: 0, + Sum: 0, + ZeroThreshold: 0.001, + ZeroCount: 0, + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 3}, + }, + PositiveBuckets: []float64{0, 0, 0}, + NegativeSpans: []histogram.Span{ + {Offset: 0, Length: 3}, + }, + NegativeBuckets: []float64{0, 0, 0}, + }, + expectedDiv: histogram.FloatHistogram{ + Schema: 0, + Count: math.Inf(1), + Sum: math.Inf(1), + ZeroThreshold: 0.001, + ZeroCount: math.Inf(1), + PositiveSpans: []histogram.Span{ + {Offset: 0, Length: 3}, + }, + PositiveBuckets: []float64{math.Inf(1), math.Inf(1), math.Inf(1)}, + NegativeSpans: []histogram.Span{ + {Offset: 0, Length: 3}, + }, + NegativeBuckets: []float64{math.Inf(1), math.Inf(1), math.Inf(1)}, + }, + }, + } + + idx0 := int64(0) + for _, c := range cases { + for _, floatHisto := range []bool{true, false} { + t.Run(fmt.Sprintf("floatHistogram=%t %d", floatHisto, idx0), func(t *testing.T) { + test, err := NewTest(t, "") + require.NoError(t, err) + t.Cleanup(test.Close) + + seriesName := "sparse_histogram_series" + floatSeriesName := "float_series" + + engine := test.QueryEngine() + + ts := idx0 * int64(10*time.Minute/time.Millisecond) + app := test.Storage().Appender(context.TODO()) + h := c.histogram + lbls := labels.FromStrings("__name__", seriesName) + // Since we mutate h later, we need to create a copy here. + if floatHisto { + _, err = app.AppendHistogram(0, lbls, ts, nil, h.Copy().ToFloat()) + } else { + _, err = app.AppendHistogram(0, lbls, ts, h.Copy(), nil) + } + require.NoError(t, err) + _, err = app.Append(0, labels.FromStrings("__name__", floatSeriesName), ts, c.scalar) + require.NoError(t, err) + require.NoError(t, app.Commit()) + + queryAndCheck := func(queryString string, exp Vector) { + qry, err := engine.NewInstantQuery(test.context, test.Queryable(), nil, queryString, timestamp.Time(ts)) + require.NoError(t, err) + + res := qry.Exec(test.Context()) + require.NoError(t, res.Err) + + vector, err := res.Vector() + require.NoError(t, err) + + require.Equal(t, exp, vector) + } + + // histogram * scalar. + queryString := fmt.Sprintf(`%s * %f`, seriesName, c.scalar) + queryAndCheck(queryString, []Sample{{T: ts, H: &c.expectedMul, Metric: labels.EmptyLabels()}}) + + // scalar * histogram. + queryString = fmt.Sprintf(`%f * %s`, c.scalar, seriesName) + queryAndCheck(queryString, []Sample{{T: ts, H: &c.expectedMul, Metric: labels.EmptyLabels()}}) + + // histogram * float. + queryString = fmt.Sprintf(`%s * %s`, seriesName, floatSeriesName) + queryAndCheck(queryString, []Sample{{T: ts, H: &c.expectedMul, Metric: labels.EmptyLabels()}}) + + // float * histogram. + queryString = fmt.Sprintf(`%s * %s`, floatSeriesName, seriesName) + queryAndCheck(queryString, []Sample{{T: ts, H: &c.expectedMul, Metric: labels.EmptyLabels()}}) + + // histogram / scalar. + queryString = fmt.Sprintf(`%s / %f`, seriesName, c.scalar) + queryAndCheck(queryString, []Sample{{T: ts, H: &c.expectedDiv, Metric: labels.EmptyLabels()}}) + + // histogram / float. + queryString = fmt.Sprintf(`%s / %s`, seriesName, floatSeriesName) + queryAndCheck(queryString, []Sample{{T: ts, H: &c.expectedDiv, Metric: labels.EmptyLabels()}}) }) idx0++ } diff --git a/promql/functions.go b/promql/functions.go index df29d6c5dc..96bffab96d 100644 --- a/promql/functions.go +++ b/promql/functions.go @@ -162,7 +162,7 @@ func extrapolatedRate(vals []parser.Value, args parser.Expressions, enh *EvalNod if resultHistogram == nil { resultFloat *= factor } else { - resultHistogram.Scale(factor) + resultHistogram.Mul(factor) } return append(enh.Out, Sample{F: resultFloat, H: resultHistogram}) @@ -443,15 +443,40 @@ func aggrOverTime(vals []parser.Value, enh *EvalNodeHelper, aggrFn func(Series) return append(enh.Out, Sample{F: aggrFn(el)}) } +func aggrHistOverTime(vals []parser.Value, enh *EvalNodeHelper, aggrFn func(Series) *histogram.FloatHistogram) Vector { + el := vals[0].(Matrix)[0] + + return append(enh.Out, Sample{H: aggrFn(el)}) +} + // === avg_over_time(Matrix parser.ValueTypeMatrix) Vector === func funcAvgOverTime(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { - if len(vals[0].(Matrix)[0].Floats) == 0 { - // TODO(beorn7): The passed values only contain - // histograms. avg_over_time ignores histograms for now. If - // there are only histograms, we have to return without adding - // anything to enh.Out. + if len(vals[0].(Matrix)[0].Floats) > 0 && len(vals[0].(Matrix)[0].Histograms) > 0 { + // TODO(zenador): Add warning for mixed floats and histograms. return enh.Out } + if len(vals[0].(Matrix)[0].Floats) == 0 { + // The passed values only contain histograms. + return aggrHistOverTime(vals, enh, func(s Series) *histogram.FloatHistogram { + count := 1 + mean := s.Histograms[0].H.Copy() + for _, h := range s.Histograms[1:] { + count++ + left := h.H.Copy().Div(float64(count)) + right := mean.Copy().Div(float64(count)) + // The histogram being added/subtracted must have + // an equal or larger schema. + if h.H.Schema >= mean.Schema { + toAdd := right.Mul(-1).Add(left) + mean.Add(toAdd) + } else { + toAdd := left.Sub(right) + mean = toAdd.Add(mean) + } + } + return mean + }) + } return aggrOverTime(vals, enh, func(s Series) float64 { var mean, count, c float64 for _, f := range s.Floats { @@ -558,13 +583,26 @@ func funcMinOverTime(vals []parser.Value, args parser.Expressions, enh *EvalNode // === sum_over_time(Matrix parser.ValueTypeMatrix) Vector === func funcSumOverTime(vals []parser.Value, args parser.Expressions, enh *EvalNodeHelper) Vector { - if len(vals[0].(Matrix)[0].Floats) == 0 { - // TODO(beorn7): The passed values only contain - // histograms. sum_over_time ignores histograms for now. If - // there are only histograms, we have to return without adding - // anything to enh.Out. + if len(vals[0].(Matrix)[0].Floats) > 0 && len(vals[0].(Matrix)[0].Histograms) > 0 { + // TODO(zenador): Add warning for mixed floats and histograms. return enh.Out } + if len(vals[0].(Matrix)[0].Floats) == 0 { + // The passed values only contain histograms. + return aggrHistOverTime(vals, enh, func(s Series) *histogram.FloatHistogram { + sum := s.Histograms[0].H.Copy() + for _, h := range s.Histograms[1:] { + // The histogram being added must have + // an equal or larger schema. + if h.H.Schema >= sum.Schema { + sum.Add(h.H) + } else { + sum = h.H.Copy().Add(sum) + } + } + return sum + }) + } return aggrOverTime(vals, enh, func(s Series) float64 { var sum, c float64 for _, f := range s.Floats { From 0dc31ade41e4693b1f30ae2ca35648a92e0583f1 Mon Sep 17 00:00:00 2001 From: Julien Pivotto Date: Wed, 17 May 2023 00:14:58 +0200 Subject: [PATCH 162/231] Add support for consul path_prefix Signed-off-by: Julien Pivotto --- config/config_test.go | 1 + config/testdata/conf.good.yml | 1 + discovery/consul/consul.go | 2 ++ docs/configuration/configuration.md | 2 ++ 4 files changed, 6 insertions(+) diff --git a/config/config_test.go b/config/config_test.go index 3ee327c5fa..bde09dfece 100644 --- a/config/config_test.go +++ b/config/config_test.go @@ -363,6 +363,7 @@ var expectedConf = &Config{ ServiceDiscoveryConfigs: discovery.Configs{ &consul.SDConfig{ Server: "localhost:1234", + PathPrefix: "/consul", Token: "mysecret", Services: []string{"nginx", "cache", "mysql"}, ServiceTags: []string{"canary", "v1"}, diff --git a/config/testdata/conf.good.yml b/config/testdata/conf.good.yml index 764f1a342b..388b9de32d 100644 --- a/config/testdata/conf.good.yml +++ b/config/testdata/conf.good.yml @@ -151,6 +151,7 @@ scrape_configs: consul_sd_configs: - server: "localhost:1234" token: mysecret + path_prefix: /consul services: ["nginx", "cache", "mysql"] tags: ["canary", "v1"] node_meta: diff --git a/discovery/consul/consul.go b/discovery/consul/consul.go index c59bd1f5d8..99ea396b99 100644 --- a/discovery/consul/consul.go +++ b/discovery/consul/consul.go @@ -111,6 +111,7 @@ func init() { // SDConfig is the configuration for Consul service discovery. type SDConfig struct { Server string `yaml:"server,omitempty"` + PathPrefix string `yaml:"path_prefix,omitempty"` Token config.Secret `yaml:"token,omitempty"` Datacenter string `yaml:"datacenter,omitempty"` Namespace string `yaml:"namespace,omitempty"` @@ -211,6 +212,7 @@ func NewDiscovery(conf *SDConfig, logger log.Logger) (*Discovery, error) { clientConf := &consul.Config{ Address: conf.Server, + PathPrefix: conf.PathPrefix, Scheme: conf.Scheme, Datacenter: conf.Datacenter, Namespace: conf.Namespace, diff --git a/docs/configuration/configuration.md b/docs/configuration/configuration.md index c74c9d478f..b0b587e02a 100644 --- a/docs/configuration/configuration.md +++ b/docs/configuration/configuration.md @@ -585,6 +585,8 @@ The following meta labels are available on targets during [relabeling](#relabel_ # The information to access the Consul API. It is to be defined # as the Consul documentation requires. [ server: | default = "localhost:8500" ] +# Prefix for URIs for when consul is behind an API gateway (reverse proxy). +[ path_prefix: ] [ token: ] [ datacenter: ] # Namespaces are only supported in Consul Enterprise. From f26760cf32b9f64871568ff4e7c78cd4f325a621 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 16 May 2023 22:30:22 +0000 Subject: [PATCH 163/231] build(deps): bump github.com/hetznercloud/hcloud-go Bumps [github.com/hetznercloud/hcloud-go](https://github.com/hetznercloud/hcloud-go) from 1.43.0 to 1.45.1. - [Release notes](https://github.com/hetznercloud/hcloud-go/releases) - [Changelog](https://github.com/hetznercloud/hcloud-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/hetznercloud/hcloud-go/compare/v1.43.0...v1.45.1) --- updated-dependencies: - dependency-name: github.com/hetznercloud/hcloud-go dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- go.mod | 4 ++-- go.sum | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/go.mod b/go.mod index 36065b21d8..c902854388 100644 --- a/go.mod +++ b/go.mod @@ -29,7 +29,7 @@ require ( github.com/grpc-ecosystem/grpc-gateway v1.16.0 github.com/hashicorp/consul/api v1.20.0 github.com/hashicorp/nomad/api v0.0.0-20230418003350-3067191c5197 - github.com/hetznercloud/hcloud-go v1.43.0 + github.com/hetznercloud/hcloud-go v1.45.1 github.com/ionos-cloud/sdk-go/v6 v6.1.6 github.com/json-iterator/go v1.1.12 github.com/kolo/xmlrpc v0.0.0-20220921171641-a4b6fa1dd06b @@ -41,7 +41,7 @@ require ( github.com/ovh/go-ovh v1.4.1 github.com/pkg/errors v0.9.1 github.com/prometheus/alertmanager v0.25.0 - github.com/prometheus/client_golang v1.15.0 + github.com/prometheus/client_golang v1.15.1 github.com/prometheus/client_model v0.3.0 github.com/prometheus/common v0.42.0 github.com/prometheus/common/assets v0.2.0 diff --git a/go.sum b/go.sum index dc44888723..9ecc5c7275 100644 --- a/go.sum +++ b/go.sum @@ -456,8 +456,8 @@ github.com/hashicorp/nomad/api v0.0.0-20230418003350-3067191c5197/go.mod h1:2TCr github.com/hashicorp/serf v0.8.2/go.mod h1:6hOLApaqBFA1NXqRQAsxw9QxuDEvNxSQRwA/JwenrHc= github.com/hashicorp/serf v0.10.1 h1:Z1H2J60yRKvfDYAOZLd2MU0ND4AH/WDz7xYHDWQsIPY= github.com/hashicorp/serf v0.10.1/go.mod h1:yL2t6BqATOLGc5HF7qbFkTfXoPIY0WZdWHfEvMqbG+4= -github.com/hetznercloud/hcloud-go v1.43.0 h1:m4p5+mz32Tt+bHkNQEg9RQdtMIu+SUdMjs29LsOJjUk= -github.com/hetznercloud/hcloud-go v1.43.0/go.mod h1:DPs7Dvae8LrTVOWyq2abwQQOwfpfICAzKHm2HQMU5/E= +github.com/hetznercloud/hcloud-go v1.45.1 h1:nl0OOklFfQT5J6AaNIOhl5Ruh3fhmGmhvZEqHbibVuk= +github.com/hetznercloud/hcloud-go v1.45.1/go.mod h1:aAUGxSfSnB8/lVXHNEDxtCT1jykaul8kqjD7f5KQXF8= github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= github.com/hudl/fargo v1.3.0/go.mod h1:y3CKSmjA+wD2gak7sUSXTAoopbhU08POFhmITJgmKTg= github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc= @@ -647,8 +647,8 @@ github.com/prometheus/client_golang v1.3.0/go.mod h1:hJaj2vgQTGQmVCsAACORcieXFeD github.com/prometheus/client_golang v1.4.0/go.mod h1:e9GMxYsXl05ICDXkRhurwBS4Q3OK1iX/F2sw+iXX5zU= github.com/prometheus/client_golang v1.7.1/go.mod h1:PY5Wy2awLA44sXw4AOSfFBetzPP4j5+D6mVACh+pe2M= github.com/prometheus/client_golang v1.11.0/go.mod h1:Z6t4BnS23TR94PD6BsDNk8yVqroYurpAkEiz0P2BEV0= -github.com/prometheus/client_golang v1.15.0 h1:5fCgGYogn0hFdhyhLbw7hEsWxufKtY9klyvdNfFlFhM= -github.com/prometheus/client_golang v1.15.0/go.mod h1:e9yaBhRPU2pPNsZwE+JdQl0KEt1N9XgF6zxWmaC0xOk= +github.com/prometheus/client_golang v1.15.1 h1:8tXpTmJbyH5lydzFPoxSIJ0J46jdh3tylbvM1xCv0LI= +github.com/prometheus/client_golang v1.15.1/go.mod h1:e9yaBhRPU2pPNsZwE+JdQl0KEt1N9XgF6zxWmaC0xOk= github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo= github.com/prometheus/client_model v0.0.0-20190115171406-56726106282f/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo= github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= From 80b7f73d267a812b3689321554aec637b75f468d Mon Sep 17 00:00:00 2001 From: Xiaochao Dong Date: Wed, 17 May 2023 21:15:12 +0800 Subject: [PATCH 164/231] Copy tombstone intervals to avoid race (#12245) Signed-off-by: Xiaochao Dong (@damnever) --- tsdb/querier_test.go | 40 ++++++++++++++++++++++++++++++ tsdb/tombstones/tombstones.go | 32 +++++++++++++++++------- tsdb/tombstones/tombstones_test.go | 36 +++++++++++++++++++++++++++ 3 files changed, 99 insertions(+), 9 deletions(-) diff --git a/tsdb/querier_test.go b/tsdb/querier_test.go index 9e17f5b8d8..6802fc2504 100644 --- a/tsdb/querier_test.go +++ b/tsdb/querier_test.go @@ -501,6 +501,46 @@ func TestBlockQuerier_AgainstHeadWithOpenChunks(t *testing.T) { } } +func TestBlockQuerier_TrimmingDoesNotModifyOriginalTombstoneIntervals(t *testing.T) { + c := blockQuerierTestCase{ + mint: 2, + maxt: 6, + ms: []*labels.Matcher{labels.MustNewMatcher(labels.MatchRegexp, "a", "a")}, + exp: newMockSeriesSet([]storage.Series{ + storage.NewListSeries(labels.FromStrings("a", "a"), + []tsdbutil.Sample{sample{3, 4, nil, nil}, sample{5, 2, nil, nil}, sample{6, 3, nil, nil}}, + ), + storage.NewListSeries(labels.FromStrings("a", "a", "b", "b"), + []tsdbutil.Sample{sample{3, 3, nil, nil}, sample{5, 3, nil, nil}, sample{6, 6, nil, nil}}, + ), + }), + expChks: newMockChunkSeriesSet([]storage.ChunkSeries{ + storage.NewListChunkSeriesFromSamples(labels.FromStrings("a", "a"), + []tsdbutil.Sample{sample{3, 4, nil, nil}}, []tsdbutil.Sample{sample{5, 2, nil, nil}, sample{6, 3, nil, nil}}, + ), + storage.NewListChunkSeriesFromSamples(labels.FromStrings("a", "a", "b", "b"), + []tsdbutil.Sample{sample{3, 3, nil, nil}}, []tsdbutil.Sample{sample{5, 3, nil, nil}, sample{6, 6, nil, nil}}, + ), + }), + } + ir, cr, _, _ := createIdxChkReaders(t, testData) + stones := tombstones.NewMemTombstones() + p, err := ir.Postings("a", "a") + require.NoError(t, err) + refs, err := index.ExpandPostings(p) + require.NoError(t, err) + for _, ref := range refs { + stones.AddInterval(ref, tombstones.Interval{Mint: 1, Maxt: 2}) + } + testBlockQuerier(t, c, ir, cr, stones) + for _, ref := range refs { + intervals, err := stones.Get(ref) + require.NoError(t, err) + // Without copy, the intervals could be [math.MinInt64, 2]. + require.Equal(t, tombstones.Intervals{{Mint: 1, Maxt: 2}}, intervals) + } +} + var testData = []seriesSamples{ { lset: map[string]string{"a": "a"}, diff --git a/tsdb/tombstones/tombstones.go b/tsdb/tombstones/tombstones.go index a52e1caa97..94daf51953 100644 --- a/tsdb/tombstones/tombstones.go +++ b/tsdb/tombstones/tombstones.go @@ -18,6 +18,7 @@ import ( "fmt" "hash" "hash/crc32" + "math" "os" "path/filepath" "sort" @@ -252,7 +253,14 @@ func NewTestMemTombstones(intervals []Intervals) *MemTombstones { func (t *MemTombstones) Get(ref storage.SeriesRef) (Intervals, error) { t.mtx.RLock() defer t.mtx.RUnlock() - return t.intvlGroups[ref], nil + intervals, ok := t.intvlGroups[ref] + if !ok { + return nil, nil + } + // Make a copy to avoid race. + res := make(Intervals, len(intervals)) + copy(res, intervals) + return res, nil } func (t *MemTombstones) DeleteTombstones(refs map[storage.SeriesRef]struct{}) { @@ -349,17 +357,23 @@ func (in Intervals) Add(n Interval) Intervals { // Find min and max indexes of intervals that overlap with the new interval. // Intervals are closed [t1, t2] and t is discreet, so if neighbour intervals are 1 step difference // to the new one, we can merge those together. - mini := sort.Search(len(in), func(i int) bool { return in[i].Maxt >= n.Mint-1 }) - if mini == len(in) { - return append(in, n) + mini := 0 + if n.Mint != math.MinInt64 { // Avoid overflow. + mini = sort.Search(len(in), func(i int) bool { return in[i].Maxt >= n.Mint-1 }) + if mini == len(in) { + return append(in, n) + } } - maxi := sort.Search(len(in)-mini, func(i int) bool { return in[mini+i].Mint > n.Maxt+1 }) - if maxi == 0 { - if mini == 0 { - return append(Intervals{n}, in...) + maxi := len(in) + if n.Maxt != math.MaxInt64 { // Avoid overflow. + maxi = sort.Search(len(in)-mini, func(i int) bool { return in[mini+i].Mint > n.Maxt+1 }) + if maxi == 0 { + if mini == 0 { + return append(Intervals{n}, in...) + } + return append(in[:mini], append(Intervals{n}, in[mini:]...)...) } - return append(in[:mini], append(Intervals{n}, in[mini:]...)...) } if n.Mint < in[mini].Mint { diff --git a/tsdb/tombstones/tombstones_test.go b/tsdb/tombstones/tombstones_test.go index ffe25b0be1..36c9f1c1e3 100644 --- a/tsdb/tombstones/tombstones_test.go +++ b/tsdb/tombstones/tombstones_test.go @@ -81,6 +81,22 @@ func TestDeletingTombstones(t *testing.T) { require.Empty(t, intervals) } +func TestTombstonesGetWithCopy(t *testing.T) { + stones := NewMemTombstones() + stones.AddInterval(1, Intervals{{Mint: 1, Maxt: 2}, {Mint: 7, Maxt: 8}, {Mint: 11, Maxt: 12}}...) + + intervals0, err := stones.Get(1) + require.NoError(t, err) + require.Equal(t, Intervals{{Mint: 1, Maxt: 2}, {Mint: 7, Maxt: 8}, {Mint: 11, Maxt: 12}}, intervals0) + intervals1 := intervals0.Add(Interval{Mint: 4, Maxt: 6}) + require.Equal(t, Intervals{{Mint: 1, Maxt: 2}, {Mint: 4, Maxt: 8}, {Mint: 11, Maxt: 12}}, intervals0) // Original slice changed. + require.Equal(t, Intervals{{Mint: 1, Maxt: 2}, {Mint: 4, Maxt: 8}, {Mint: 11, Maxt: 12}}, intervals1) + + intervals2, err := stones.Get(1) + require.NoError(t, err) + require.Equal(t, Intervals{{Mint: 1, Maxt: 2}, {Mint: 7, Maxt: 8}, {Mint: 11, Maxt: 12}}, intervals2) +} + func TestTruncateBefore(t *testing.T) { cases := []struct { before Intervals @@ -210,6 +226,26 @@ func TestAddingNewIntervals(t *testing.T) { new: Interval{math.MinInt64, 10}, exp: Intervals{{math.MinInt64, math.MaxInt64}}, }, + { + exist: Intervals{{9, 10}}, + new: Interval{math.MinInt64, 7}, + exp: Intervals{{math.MinInt64, 7}, {9, 10}}, + }, + { + exist: Intervals{{9, 10}}, + new: Interval{12, math.MaxInt64}, + exp: Intervals{{9, 10}, {12, math.MaxInt64}}, + }, + { + exist: Intervals{{9, 10}}, + new: Interval{math.MinInt64, 8}, + exp: Intervals{{math.MinInt64, 10}}, + }, + { + exist: Intervals{{9, 10}}, + new: Interval{11, math.MaxInt64}, + exp: Intervals{{9, math.MaxInt64}}, + }, } for _, c := range cases { From f731a90a7f33cf19ea8a63d889a1830a05c0236d Mon Sep 17 00:00:00 2001 From: Baskar Shanmugam <123750249+codebasky@users.noreply.github.com> Date: Fri, 19 May 2023 13:06:30 +0530 Subject: [PATCH 165/231] Fix LabelValueStats in posting stats (#12342) Problem: LabelValueStats - This will provide a list of the label names and memory used in bytes. It is calculated by adding the length of all values for a given label name. But internally Prometheus stores the name and the value independently for each series. Solution: MemPostings struct maintains the values to seriesRef map which is used to get the number of series which contains the label values. Using that LabelValueStats is calculated as: seriesCnt * len(value name) Signed-off-by: Baskar Shanmugam --- tsdb/index/postings.go | 5 +++-- tsdb/index/postings_test.go | 29 +++++++++++++++++++++++++++++ 2 files changed, 32 insertions(+), 2 deletions(-) diff --git a/tsdb/index/postings.go b/tsdb/index/postings.go index b93d3a2021..200100239c 100644 --- a/tsdb/index/postings.go +++ b/tsdb/index/postings.go @@ -184,8 +184,9 @@ func (p *MemPostings) Stats(label string) *PostingsStats { if n == label { metrics.push(Stat{Name: name, Count: uint64(len(values))}) } - labelValuePairs.push(Stat{Name: n + "=" + name, Count: uint64(len(values))}) - size += uint64(len(name)) + seriesCnt := uint64(len(values)) + labelValuePairs.push(Stat{Name: n + "=" + name, Count: seriesCnt}) + size += uint64(len(name)) * seriesCnt } labelValueLength.push(Stat{Name: n, Count: size}) } diff --git a/tsdb/index/postings_test.go b/tsdb/index/postings_test.go index a34f3c12df..9d8b5ebf32 100644 --- a/tsdb/index/postings_test.go +++ b/tsdb/index/postings_test.go @@ -916,6 +916,35 @@ func BenchmarkPostings_Stats(b *testing.B) { } } +func TestMemPostingsStats(t *testing.T) { + // create a new MemPostings + p := NewMemPostings() + + // add some postings to the MemPostings + p.Add(1, labels.FromStrings("label", "value1")) + p.Add(1, labels.FromStrings("label", "value2")) + p.Add(1, labels.FromStrings("label", "value3")) + p.Add(2, labels.FromStrings("label", "value1")) + + // call the Stats method to calculate the cardinality statistics + stats := p.Stats("label") + + // assert that the expected statistics were calculated + require.Equal(t, uint64(2), stats.CardinalityMetricsStats[0].Count) + require.Equal(t, "value1", stats.CardinalityMetricsStats[0].Name) + + require.Equal(t, uint64(3), stats.CardinalityLabelStats[0].Count) + require.Equal(t, "label", stats.CardinalityLabelStats[0].Name) + + require.Equal(t, uint64(24), stats.LabelValueStats[0].Count) + require.Equal(t, "label", stats.LabelValueStats[0].Name) + + require.Equal(t, uint64(2), stats.LabelValuePairsStats[0].Count) + require.Equal(t, "label=value1", stats.LabelValuePairsStats[0].Name) + + require.Equal(t, 3, stats.NumLabelPairs) +} + func TestMemPostings_Delete(t *testing.T) { p := NewMemPostings() p.Add(1, labels.FromStrings("lbl1", "a")) From 92d69803606620505ed8f0226abb640a121ec1cd Mon Sep 17 00:00:00 2001 From: George Krajcsovits Date: Fri, 19 May 2023 10:24:06 +0200 Subject: [PATCH 166/231] Fix populateWithDelChunkSeriesIterator and gauge histograms (#12330) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Use AppendableGauge to detect corrupt chunk with gauge histograms. Detect if first sample is a gauge but the chunk is not set up to contain gauge histograms. Signed-off-by: György Krajcsovits Signed-off-by: George Krajcsovits --- tsdb/head_read.go | 8 +- tsdb/querier.go | 137 ++++++++++++++----- tsdb/querier_test.go | 274 +++++++++++++++++++++++++++++++++++++ tsdb/tsdbutil/chunks.go | 14 +- tsdb/tsdbutil/histogram.go | 10 ++ 5 files changed, 404 insertions(+), 39 deletions(-) diff --git a/tsdb/head_read.go b/tsdb/head_read.go index 9c546ab164..0e6e005ea2 100644 --- a/tsdb/head_read.go +++ b/tsdb/head_read.go @@ -331,7 +331,7 @@ func (h *headChunkReader) chunk(meta chunks.Meta, copyLastChunk bool) (chunkenc. } s.Unlock() - return &safeChunk{ + return &safeHeadChunk{ Chunk: chk, s: s, cid: cid, @@ -627,15 +627,15 @@ func (b boundedIterator) Seek(t int64) chunkenc.ValueType { return b.Iterator.Seek(t) } -// safeChunk makes sure that the chunk can be accessed without a race condition -type safeChunk struct { +// safeHeadChunk makes sure that the chunk can be accessed without a race condition +type safeHeadChunk struct { chunkenc.Chunk s *memSeries cid chunks.HeadChunkID isoState *isolationState } -func (c *safeChunk) Iterator(reuseIter chunkenc.Iterator) chunkenc.Iterator { +func (c *safeHeadChunk) Iterator(reuseIter chunkenc.Iterator) chunkenc.Iterator { c.s.Lock() it := c.s.iterator(c.cid, c.Chunk, c.isoState, reuseIter) c.s.Unlock() diff --git a/tsdb/querier.go b/tsdb/querier.go index 8b17bb745a..8bcd451f3f 100644 --- a/tsdb/querier.go +++ b/tsdb/querier.go @@ -785,14 +785,35 @@ func (p *populateWithDelChunkSeriesIterator) Next() bool { if app, err = newChunk.Appender(); err != nil { break } - if hc, ok := p.currChkMeta.Chunk.(*chunkenc.HistogramChunk); ok { + + switch hc := p.currChkMeta.Chunk.(type) { + case *chunkenc.HistogramChunk: newChunk.(*chunkenc.HistogramChunk).SetCounterResetHeader(hc.GetCounterResetHeader()) + case *safeHeadChunk: + if unwrapped, ok := hc.Chunk.(*chunkenc.HistogramChunk); ok { + newChunk.(*chunkenc.HistogramChunk).SetCounterResetHeader(unwrapped.GetCounterResetHeader()) + } else { + err = fmt.Errorf("internal error, could not unwrap safeHeadChunk to histogram chunk: %T", hc.Chunk) + } + default: + err = fmt.Errorf("internal error, unknown chunk type %T when expecting histogram", p.currChkMeta.Chunk) } + if err != nil { + break + } + var h *histogram.Histogram t, h = p.currDelIter.AtHistogram() p.curr.MinTime = t + // Detect missing gauge reset hint. + if h.CounterResetHint == histogram.GaugeType && newChunk.(*chunkenc.HistogramChunk).GetCounterResetHeader() != chunkenc.GaugeType { + err = fmt.Errorf("found gauge histogram in non gauge chunk") + break + } + app.AppendHistogram(t, h) + for vt := p.currDelIter.Next(); vt != chunkenc.ValNone; vt = p.currDelIter.Next() { if vt != chunkenc.ValHistogram { err = fmt.Errorf("found value type %v in histogram chunk", vt) @@ -801,23 +822,37 @@ func (p *populateWithDelChunkSeriesIterator) Next() bool { t, h = p.currDelIter.AtHistogram() // Defend against corrupted chunks. - pI, nI, okToAppend, counterReset := app.(*chunkenc.HistogramAppender).Appendable(h) - if len(pI)+len(nI) > 0 { - err = fmt.Errorf( - "bucket layout has changed unexpectedly: %d positive and %d negative bucket interjections required", - len(pI), len(nI), - ) - break + if h.CounterResetHint == histogram.GaugeType { + pI, nI, bpI, bnI, _, _, okToAppend := app.(*chunkenc.HistogramAppender).AppendableGauge(h) + if !okToAppend { + err = errors.New("unable to append histogram due to unexpected schema change") + break + } + if len(pI)+len(nI)+len(bpI)+len(bnI) > 0 { + err = fmt.Errorf( + "bucket layout has changed unexpectedly: forward %d positive, %d negative, backward %d positive %d negative bucket interjections required", + len(pI), len(nI), len(bpI), len(bnI), + ) + break + } + } else { + pI, nI, okToAppend, counterReset := app.(*chunkenc.HistogramAppender).Appendable(h) + if len(pI)+len(nI) > 0 { + err = fmt.Errorf( + "bucket layout has changed unexpectedly: %d positive and %d negative bucket interjections required", + len(pI), len(nI), + ) + break + } + if counterReset { + err = errors.New("detected unexpected counter reset in histogram") + break + } + if !okToAppend { + err = errors.New("unable to append histogram due to unexpected schema change") + break + } } - if counterReset { - err = errors.New("detected unexpected counter reset in histogram") - break - } - if !okToAppend { - err = errors.New("unable to append histogram due to unexpected schema change") - break - } - app.AppendHistogram(t, h) } case chunkenc.ValFloat: @@ -842,14 +877,35 @@ func (p *populateWithDelChunkSeriesIterator) Next() bool { if app, err = newChunk.Appender(); err != nil { break } - if hc, ok := p.currChkMeta.Chunk.(*chunkenc.FloatHistogramChunk); ok { + + switch hc := p.currChkMeta.Chunk.(type) { + case *chunkenc.FloatHistogramChunk: newChunk.(*chunkenc.FloatHistogramChunk).SetCounterResetHeader(hc.GetCounterResetHeader()) + case *safeHeadChunk: + if unwrapped, ok := hc.Chunk.(*chunkenc.FloatHistogramChunk); ok { + newChunk.(*chunkenc.FloatHistogramChunk).SetCounterResetHeader(unwrapped.GetCounterResetHeader()) + } else { + err = fmt.Errorf("internal error, could not unwrap safeHeadChunk to float histogram chunk: %T", hc.Chunk) + } + default: + err = fmt.Errorf("internal error, unknown chunk type %T when expecting float histogram", p.currChkMeta.Chunk) } + if err != nil { + break + } + var h *histogram.FloatHistogram t, h = p.currDelIter.AtFloatHistogram() p.curr.MinTime = t + // Detect missing gauge reset hint. + if h.CounterResetHint == histogram.GaugeType && newChunk.(*chunkenc.FloatHistogramChunk).GetCounterResetHeader() != chunkenc.GaugeType { + err = fmt.Errorf("found float gauge histogram in non gauge chunk") + break + } + app.AppendFloatHistogram(t, h) + for vt := p.currDelIter.Next(); vt != chunkenc.ValNone; vt = p.currDelIter.Next() { if vt != chunkenc.ValFloatHistogram { err = fmt.Errorf("found value type %v in histogram chunk", vt) @@ -858,21 +914,36 @@ func (p *populateWithDelChunkSeriesIterator) Next() bool { t, h = p.currDelIter.AtFloatHistogram() // Defend against corrupted chunks. - pI, nI, okToAppend, counterReset := app.(*chunkenc.FloatHistogramAppender).Appendable(h) - if len(pI)+len(nI) > 0 { - err = fmt.Errorf( - "bucket layout has changed unexpectedly: %d positive and %d negative bucket interjections required", - len(pI), len(nI), - ) - break - } - if counterReset { - err = errors.New("detected unexpected counter reset in histogram") - break - } - if !okToAppend { - err = errors.New("unable to append histogram due to unexpected schema change") - break + if h.CounterResetHint == histogram.GaugeType { + pI, nI, bpI, bnI, _, _, okToAppend := app.(*chunkenc.FloatHistogramAppender).AppendableGauge(h) + if !okToAppend { + err = errors.New("unable to append histogram due to unexpected schema change") + break + } + if len(pI)+len(nI)+len(bpI)+len(bnI) > 0 { + err = fmt.Errorf( + "bucket layout has changed unexpectedly: forward %d positive, %d negative, backward %d positive %d negative bucket interjections required", + len(pI), len(nI), len(bpI), len(bnI), + ) + break + } + } else { + pI, nI, okToAppend, counterReset := app.(*chunkenc.FloatHistogramAppender).Appendable(h) + if len(pI)+len(nI) > 0 { + err = fmt.Errorf( + "bucket layout has changed unexpectedly: %d positive and %d negative bucket interjections required", + len(pI), len(nI), + ) + break + } + if counterReset { + err = errors.New("detected unexpected counter reset in histogram") + break + } + if !okToAppend { + err = errors.New("unable to append histogram due to unexpected schema change") + break + } } app.AppendFloatHistogram(t, h) diff --git a/tsdb/querier_test.go b/tsdb/querier_test.go index 6802fc2504..5bf721a620 100644 --- a/tsdb/querier_test.go +++ b/tsdb/querier_test.go @@ -29,6 +29,7 @@ import ( "github.com/pkg/errors" "github.com/stretchr/testify/require" + "github.com/prometheus/prometheus/model/histogram" "github.com/prometheus/prometheus/model/labels" "github.com/prometheus/prometheus/storage" "github.com/prometheus/prometheus/tsdb/chunkenc" @@ -907,6 +908,202 @@ func TestPopulateWithTombSeriesIterators(t *testing.T) { sample{3, 5, nil, nil}, sample{6, 1, nil, nil}, sample{7, 89, nil, nil}, }, }, + { + name: "one histogram chunk", + chks: [][]tsdbutil.Sample{ + { + sample{1, 0, tsdbutil.GenerateTestHistogram(1), nil}, + sample{2, 0, tsdbutil.GenerateTestHistogram(2), nil}, + sample{3, 0, tsdbutil.GenerateTestHistogram(3), nil}, + sample{6, 0, tsdbutil.GenerateTestHistogram(6), nil}, + }, + }, + expected: []tsdbutil.Sample{ + sample{1, 0, tsdbutil.GenerateTestHistogram(1), nil}, + sample{2, 0, tsdbutil.SetHistogramNotCounterReset(tsdbutil.GenerateTestHistogram(2)), nil}, + sample{3, 0, tsdbutil.SetHistogramNotCounterReset(tsdbutil.GenerateTestHistogram(3)), nil}, + sample{6, 0, tsdbutil.SetHistogramNotCounterReset(tsdbutil.GenerateTestHistogram(6)), nil}, + }, + expectedChks: []chunks.Meta{ + tsdbutil.ChunkFromSamples([]tsdbutil.Sample{ + sample{1, 0, tsdbutil.GenerateTestHistogram(1), nil}, + sample{2, 0, tsdbutil.SetHistogramNotCounterReset(tsdbutil.GenerateTestHistogram(2)), nil}, + sample{3, 0, tsdbutil.SetHistogramNotCounterReset(tsdbutil.GenerateTestHistogram(3)), nil}, + sample{6, 0, tsdbutil.SetHistogramNotCounterReset(tsdbutil.GenerateTestHistogram(6)), nil}, + }), + }, + }, + { + name: "one histogram chunk intersect with deletion interval", + chks: [][]tsdbutil.Sample{ + { + sample{1, 0, tsdbutil.GenerateTestHistogram(1), nil}, + sample{2, 0, tsdbutil.GenerateTestHistogram(2), nil}, + sample{3, 0, tsdbutil.GenerateTestHistogram(3), nil}, + sample{6, 0, tsdbutil.GenerateTestHistogram(6), nil}, + }, + }, + intervals: tombstones.Intervals{{Mint: 5, Maxt: 20}}, + expected: []tsdbutil.Sample{ + sample{1, 0, tsdbutil.GenerateTestHistogram(1), nil}, + sample{2, 0, tsdbutil.SetHistogramNotCounterReset(tsdbutil.GenerateTestHistogram(2)), nil}, + sample{3, 0, tsdbutil.SetHistogramNotCounterReset(tsdbutil.GenerateTestHistogram(3)), nil}, + }, + expectedChks: []chunks.Meta{ + tsdbutil.ChunkFromSamples([]tsdbutil.Sample{ + sample{1, 0, tsdbutil.GenerateTestHistogram(1), nil}, + sample{2, 0, tsdbutil.SetHistogramNotCounterReset(tsdbutil.GenerateTestHistogram(2)), nil}, + sample{3, 0, tsdbutil.SetHistogramNotCounterReset(tsdbutil.GenerateTestHistogram(3)), nil}, + }), + }, + }, + { + name: "one float histogram chunk", + chks: [][]tsdbutil.Sample{ + { + sample{1, 0, nil, tsdbutil.GenerateTestFloatHistogram(1)}, + sample{2, 0, nil, tsdbutil.GenerateTestFloatHistogram(2)}, + sample{3, 0, nil, tsdbutil.GenerateTestFloatHistogram(3)}, + sample{6, 0, nil, tsdbutil.GenerateTestFloatHistogram(6)}, + }, + }, + expected: []tsdbutil.Sample{ + sample{1, 0, nil, tsdbutil.GenerateTestFloatHistogram(1)}, + sample{2, 0, nil, tsdbutil.SetFloatHistogramNotCounterReset(tsdbutil.GenerateTestFloatHistogram(2))}, + sample{3, 0, nil, tsdbutil.SetFloatHistogramNotCounterReset(tsdbutil.GenerateTestFloatHistogram(3))}, + sample{6, 0, nil, tsdbutil.SetFloatHistogramNotCounterReset(tsdbutil.GenerateTestFloatHistogram(6))}, + }, + expectedChks: []chunks.Meta{ + tsdbutil.ChunkFromSamples([]tsdbutil.Sample{ + sample{1, 0, nil, tsdbutil.GenerateTestFloatHistogram(1)}, + sample{2, 0, nil, tsdbutil.SetFloatHistogramNotCounterReset(tsdbutil.GenerateTestFloatHistogram(2))}, + sample{3, 0, nil, tsdbutil.SetFloatHistogramNotCounterReset(tsdbutil.GenerateTestFloatHistogram(3))}, + sample{6, 0, nil, tsdbutil.SetFloatHistogramNotCounterReset(tsdbutil.GenerateTestFloatHistogram(6))}, + }), + }, + }, + { + name: "one float histogram chunk intersect with deletion interval", + chks: [][]tsdbutil.Sample{ + { + sample{1, 0, nil, tsdbutil.GenerateTestFloatHistogram(1)}, + sample{2, 0, nil, tsdbutil.GenerateTestFloatHistogram(2)}, + sample{3, 0, nil, tsdbutil.GenerateTestFloatHistogram(3)}, + sample{6, 0, nil, tsdbutil.GenerateTestFloatHistogram(6)}, + }, + }, + intervals: tombstones.Intervals{{Mint: 5, Maxt: 20}}, + expected: []tsdbutil.Sample{ + sample{1, 0, nil, tsdbutil.GenerateTestFloatHistogram(1)}, + sample{2, 0, nil, tsdbutil.SetFloatHistogramNotCounterReset(tsdbutil.GenerateTestFloatHistogram(2))}, + sample{3, 0, nil, tsdbutil.SetFloatHistogramNotCounterReset(tsdbutil.GenerateTestFloatHistogram(3))}, + }, + expectedChks: []chunks.Meta{ + tsdbutil.ChunkFromSamples([]tsdbutil.Sample{ + sample{1, 0, nil, tsdbutil.GenerateTestFloatHistogram(1)}, + sample{2, 0, nil, tsdbutil.SetFloatHistogramNotCounterReset(tsdbutil.GenerateTestFloatHistogram(2))}, + sample{3, 0, nil, tsdbutil.SetFloatHistogramNotCounterReset(tsdbutil.GenerateTestFloatHistogram(3))}, + }), + }, + }, + { + name: "one gauge histogram chunk", + chks: [][]tsdbutil.Sample{ + { + sample{1, 0, tsdbutil.GenerateTestGaugeHistogram(1), nil}, + sample{2, 0, tsdbutil.GenerateTestGaugeHistogram(2), nil}, + sample{3, 0, tsdbutil.GenerateTestGaugeHistogram(3), nil}, + sample{6, 0, tsdbutil.GenerateTestGaugeHistogram(6), nil}, + }, + }, + expected: []tsdbutil.Sample{ + sample{1, 0, tsdbutil.GenerateTestGaugeHistogram(1), nil}, + sample{2, 0, tsdbutil.GenerateTestGaugeHistogram(2), nil}, + sample{3, 0, tsdbutil.GenerateTestGaugeHistogram(3), nil}, + sample{6, 0, tsdbutil.GenerateTestGaugeHistogram(6), nil}, + }, + expectedChks: []chunks.Meta{ + tsdbutil.ChunkFromSamples([]tsdbutil.Sample{ + sample{1, 0, tsdbutil.GenerateTestGaugeHistogram(1), nil}, + sample{2, 0, tsdbutil.GenerateTestGaugeHistogram(2), nil}, + sample{3, 0, tsdbutil.GenerateTestGaugeHistogram(3), nil}, + sample{6, 0, tsdbutil.GenerateTestGaugeHistogram(6), nil}, + }), + }, + }, + { + name: "one gauge histogram chunk intersect with deletion interval", + chks: [][]tsdbutil.Sample{ + { + sample{1, 0, tsdbutil.GenerateTestGaugeHistogram(1), nil}, + sample{2, 0, tsdbutil.GenerateTestGaugeHistogram(2), nil}, + sample{3, 0, tsdbutil.GenerateTestGaugeHistogram(3), nil}, + sample{6, 0, tsdbutil.GenerateTestGaugeHistogram(6), nil}, + }, + }, + intervals: tombstones.Intervals{{Mint: 5, Maxt: 20}}, + expected: []tsdbutil.Sample{ + sample{1, 0, tsdbutil.GenerateTestGaugeHistogram(1), nil}, + sample{2, 0, tsdbutil.GenerateTestGaugeHistogram(2), nil}, + sample{3, 0, tsdbutil.GenerateTestGaugeHistogram(3), nil}, + }, + expectedChks: []chunks.Meta{ + tsdbutil.ChunkFromSamples([]tsdbutil.Sample{ + sample{1, 0, tsdbutil.GenerateTestGaugeHistogram(1), nil}, + sample{2, 0, tsdbutil.GenerateTestGaugeHistogram(2), nil}, + sample{3, 0, tsdbutil.GenerateTestGaugeHistogram(3), nil}, + }), + }, + }, + { + name: "one gauge float histogram", + chks: [][]tsdbutil.Sample{ + { + sample{1, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(1)}, + sample{2, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(2)}, + sample{3, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(3)}, + sample{6, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(6)}, + }, + }, + expected: []tsdbutil.Sample{ + sample{1, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(1)}, + sample{2, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(2)}, + sample{3, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(3)}, + sample{6, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(6)}, + }, + expectedChks: []chunks.Meta{ + tsdbutil.ChunkFromSamples([]tsdbutil.Sample{ + sample{1, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(1)}, + sample{2, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(2)}, + sample{3, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(3)}, + sample{6, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(6)}, + }), + }, + }, + { + name: "one gauge float histogram chunk intersect with deletion interval", + chks: [][]tsdbutil.Sample{ + { + sample{1, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(1)}, + sample{2, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(2)}, + sample{3, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(3)}, + sample{6, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(6)}, + }, + }, + intervals: tombstones.Intervals{{Mint: 5, Maxt: 20}}, + expected: []tsdbutil.Sample{ + sample{1, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(1)}, + sample{2, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(2)}, + sample{3, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(3)}, + }, + expectedChks: []chunks.Meta{ + tsdbutil.ChunkFromSamples([]tsdbutil.Sample{ + sample{1, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(1)}, + sample{2, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(2)}, + sample{3, 0, nil, tsdbutil.GenerateTestGaugeFloatHistogram(3)}, + }), + }, + }, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { @@ -2411,3 +2608,80 @@ func BenchmarkHeadQuerier(b *testing.B) { require.NoError(b, ss.Err()) } } + +// This is a regression test for the case where gauge histograms were not handled by +// populateWithDelChunkSeriesIterator correctly. +func TestQueryWithDeletedHistograms(t *testing.T) { + testcases := map[string]func(int) (*histogram.Histogram, *histogram.FloatHistogram){ + "intCounter": func(i int) (*histogram.Histogram, *histogram.FloatHistogram) { + return tsdbutil.GenerateTestHistogram(i), nil + }, + "intgauge": func(i int) (*histogram.Histogram, *histogram.FloatHistogram) { + return tsdbutil.GenerateTestGaugeHistogram(rand.Int() % 1000), nil + }, + "floatCounter": func(i int) (*histogram.Histogram, *histogram.FloatHistogram) { + return nil, tsdbutil.GenerateTestFloatHistogram(i) + }, + "floatGauge": func(i int) (*histogram.Histogram, *histogram.FloatHistogram) { + return nil, tsdbutil.GenerateTestGaugeFloatHistogram(rand.Int() % 1000) + }, + } + + for name, tc := range testcases { + t.Run(name, func(t *testing.T) { + db := openTestDB(t, nil, nil) + defer func() { + require.NoError(t, db.Close()) + }() + + db.EnableNativeHistograms() + appender := db.Appender(context.Background()) + + var ( + err error + seriesRef storage.SeriesRef + ) + lbs := labels.FromStrings("__name__", "test", "type", name) + + for i := 0; i < 100; i++ { + h, fh := tc(i) + seriesRef, err = appender.AppendHistogram(seriesRef, lbs, int64(i), h, fh) + require.NoError(t, err) + } + + err = appender.Commit() + require.NoError(t, err) + + matcher, err := labels.NewMatcher(labels.MatchEqual, "__name__", "test") + require.NoError(t, err) + + // Delete the last 20. + err = db.Delete(80, 100, matcher) + require.NoError(t, err) + + chunkQuerier, err := db.ChunkQuerier(context.Background(), 0, 100) + require.NoError(t, err) + + css := chunkQuerier.Select(false, nil, matcher) + + seriesCount := 0 + for css.Next() { + seriesCount++ + series := css.At() + + sampleCount := 0 + it := series.Iterator(nil) + for it.Next() { + chk := it.At() + for cit := chk.Chunk.Iterator(nil); cit.Next() != chunkenc.ValNone; { + sampleCount++ + } + } + require.NoError(t, it.Err()) + require.Equal(t, 80, sampleCount) + } + require.NoError(t, css.Err()) + require.Equal(t, 1, seriesCount) + }) + } +} diff --git a/tsdb/tsdbutil/chunks.go b/tsdb/tsdbutil/chunks.go index 02a7dd6198..6e57016fec 100644 --- a/tsdb/tsdbutil/chunks.go +++ b/tsdb/tsdbutil/chunks.go @@ -71,9 +71,19 @@ func ChunkFromSamplesGeneric(s Samples) chunks.Meta { case chunkenc.ValFloat: ca.Append(s.Get(i).T(), s.Get(i).F()) case chunkenc.ValHistogram: - ca.AppendHistogram(s.Get(i).T(), s.Get(i).H()) + h := s.Get(i).H() + ca.AppendHistogram(s.Get(i).T(), h) + if i == 0 && h.CounterResetHint == histogram.GaugeType { + hc := c.(*chunkenc.HistogramChunk) + hc.SetCounterResetHeader(chunkenc.GaugeType) + } case chunkenc.ValFloatHistogram: - ca.AppendFloatHistogram(s.Get(i).T(), s.Get(i).FH()) + fh := s.Get(i).FH() + ca.AppendFloatHistogram(s.Get(i).T(), fh) + if i == 0 && fh.CounterResetHint == histogram.GaugeType { + hc := c.(*chunkenc.FloatHistogramChunk) + hc.SetCounterResetHeader(chunkenc.GaugeType) + } default: panic(fmt.Sprintf("unknown sample type %s", sampleType.String())) } diff --git a/tsdb/tsdbutil/histogram.go b/tsdb/tsdbutil/histogram.go index 3c276c8411..2145034e19 100644 --- a/tsdb/tsdbutil/histogram.go +++ b/tsdb/tsdbutil/histogram.go @@ -108,3 +108,13 @@ func GenerateTestGaugeFloatHistogram(i int) *histogram.FloatHistogram { h.CounterResetHint = histogram.GaugeType return h } + +func SetHistogramNotCounterReset(h *histogram.Histogram) *histogram.Histogram { + h.CounterResetHint = histogram.NotCounterReset + return h +} + +func SetFloatHistogramNotCounterReset(h *histogram.FloatHistogram) *histogram.FloatHistogram { + h.CounterResetHint = histogram.NotCounterReset + return h +} From c8e7f95a3cbfbb26eb48e86155ca1f4cfe20dc8f Mon Sep 17 00:00:00 2001 From: Matthieu MOREL Date: Sun, 21 May 2023 09:20:07 +0200 Subject: [PATCH 167/231] ci(lint): enable predeclared linter Signed-off-by: Matthieu MOREL --- .golangci.yml | 3 ++- tsdb/exemplar.go | 6 +++--- tsdb/isolation.go | 4 ++-- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/.golangci.yml b/.golangci.yml index d1cd86ed59..fc2721455c 100644 --- a/.golangci.yml +++ b/.golangci.yml @@ -13,8 +13,9 @@ linters: - gocritic - gofumpt - goimports - - revive - misspell + - predeclared + - revive - unconvert - unused diff --git a/tsdb/exemplar.go b/tsdb/exemplar.go index ad3b2ef39b..01718bb57d 100644 --- a/tsdb/exemplar.go +++ b/tsdb/exemplar.go @@ -216,9 +216,9 @@ func (ce *CircularExemplarStorage) ValidateExemplar(l labels.Labels, e exemplar. return ce.validateExemplar(seriesLabels, e, false) } -// Not thread safe. The append parameters tells us whether this is an external validation, or internal +// Not thread safe. The appended parameters tells us whether this is an external validation, or internal // as a result of an AddExemplar call, in which case we should update any relevant metrics. -func (ce *CircularExemplarStorage) validateExemplar(key []byte, e exemplar.Exemplar, append bool) error { +func (ce *CircularExemplarStorage) validateExemplar(key []byte, e exemplar.Exemplar, appended bool) error { if len(ce.exemplars) == 0 { return storage.ErrExemplarsDisabled } @@ -250,7 +250,7 @@ func (ce *CircularExemplarStorage) validateExemplar(key []byte, e exemplar.Exemp } if e.Ts <= ce.exemplars[idx.newest].exemplar.Ts { - if append { + if appended { ce.metrics.outOfOrderExemplars.Inc() } return storage.ErrOutOfOrderExemplar diff --git a/tsdb/isolation.go b/tsdb/isolation.go index 401e5885a0..e436884a8d 100644 --- a/tsdb/isolation.go +++ b/tsdb/isolation.go @@ -244,9 +244,9 @@ type txRing struct { txIDCount int // How many ids in the ring. } -func newTxRing(cap int) *txRing { +func newTxRing(capacity int) *txRing { return &txRing{ - txIDs: make([]uint64, cap), + txIDs: make([]uint64, capacity), } } From 8c5d4b4add2c528701cced254d1272824b9ec3ba Mon Sep 17 00:00:00 2001 From: Alan Protasio Date: Sun, 21 May 2023 01:41:30 -0700 Subject: [PATCH 168/231] Opmize MatchNotEqual (#12377) Signed-off-by: Alan Protasio --- tsdb/querier.go | 24 ++++++++++++++++-------- tsdb/querier_bench_test.go | 1 + 2 files changed, 17 insertions(+), 8 deletions(-) diff --git a/tsdb/querier.go b/tsdb/querier.go index 8bcd451f3f..9baf3f2429 100644 --- a/tsdb/querier.go +++ b/tsdb/querier.go @@ -361,6 +361,22 @@ func postingsForMatcher(ix IndexReader, m *labels.Matcher) (index.Postings, erro // inversePostingsForMatcher returns the postings for the series with the label name set but not matching the matcher. func inversePostingsForMatcher(ix IndexReader, m *labels.Matcher) (index.Postings, error) { + // Fast-path for MatchNotRegexp matching. + // Inverse of a MatchNotRegexp is MatchRegexp (double negation). + // Fast-path for set matching. + if m.Type == labels.MatchNotRegexp { + setMatches := findSetMatches(m.GetRegexString()) + if len(setMatches) > 0 { + return ix.Postings(m.Name, setMatches...) + } + } + + // Fast-path for MatchNotEqual matching. + // Inverse of a MatchNotEqual is MatchEqual (double negation). + if m.Type == labels.MatchNotEqual { + return ix.Postings(m.Name, m.Value) + } + vals, err := ix.LabelValues(m.Name) if err != nil { return nil, err @@ -371,14 +387,6 @@ func inversePostingsForMatcher(ix IndexReader, m *labels.Matcher) (index.Posting if m.Type == labels.MatchEqual && m.Value == "" { res = vals } else { - // Inverse of a MatchNotRegexp is MatchRegexp (double negation). - // Fast-path for set matching. - if m.Type == labels.MatchNotRegexp { - setMatches := findSetMatches(m.GetRegexString()) - if len(setMatches) > 0 { - return ix.Postings(m.Name, setMatches...) - } - } for _, val := range vals { if !m.Matches(val) { res = append(res, val) diff --git a/tsdb/querier_bench_test.go b/tsdb/querier_bench_test.go index 6e5243ef11..19619e35bb 100644 --- a/tsdb/querier_bench_test.go +++ b/tsdb/querier_bench_test.go @@ -125,6 +125,7 @@ func benchmarkPostingsForMatchers(b *testing.B, ir IndexReader) { {`n="X",j="foo"`, []*labels.Matcher{nX, jFoo}}, {`j="foo",n="1"`, []*labels.Matcher{jFoo, n1}}, {`n="1",j!="foo"`, []*labels.Matcher{n1, jNotFoo}}, + {`n="1",i!="2"`, []*labels.Matcher{n1, iNot2}}, {`n="X",j!="foo"`, []*labels.Matcher{nX, jNotFoo}}, {`i=~"1[0-9]",j=~"foo|bar"`, []*labels.Matcher{iCharSet, jFooBar}}, {`j=~"foo|bar"`, []*labels.Matcher{jFooBar}}, From 905a0bd63a12fa328eac4b7f2674356c283bacac Mon Sep 17 00:00:00 2001 From: Baskar Shanmugam <123750249+codebasky@users.noreply.github.com> Date: Mon, 22 May 2023 18:07:07 +0530 Subject: [PATCH 169/231] Added 'limit' query parameter support to /api/v1/status/tsdb endpoint (#12336) * Added 'topN' query parameter support to /api/v1/status/tsdb endpoint Signed-off-by: Baskar Shanmugam * Updated query parameter for tsdb status to 'limit' Signed-off-by: Baskar Shanmugam * Corrected Stats() parameter name from topN to limit Signed-off-by: Baskar Shanmugam * Fixed p.Stats CI failure Signed-off-by: Baskar Shanmugam --------- Signed-off-by: Baskar Shanmugam --- cmd/prometheus/main.go | 4 ++-- docs/querying/api.md | 4 ++++ tsdb/head.go | 8 ++++---- tsdb/index/postings.go | 12 +++++------- tsdb/index/postings_test.go | 4 ++-- web/api/v1/api.go | 13 ++++++++++--- web/api/v1/api_test.go | 19 +++++++++++++++---- web/federate_test.go | 2 +- web/web_test.go | 4 ++-- 9 files changed, 45 insertions(+), 25 deletions(-) diff --git a/cmd/prometheus/main.go b/cmd/prometheus/main.go index 778b131c8c..e05ac79570 100644 --- a/cmd/prometheus/main.go +++ b/cmd/prometheus/main.go @@ -1486,11 +1486,11 @@ func (s *readyStorage) Snapshot(dir string, withHead bool) error { } // Stats implements the api_v1.TSDBAdminStats interface. -func (s *readyStorage) Stats(statsByLabelName string) (*tsdb.Stats, error) { +func (s *readyStorage) Stats(statsByLabelName string, limit int) (*tsdb.Stats, error) { if x := s.get(); x != nil { switch db := x.(type) { case *tsdb.DB: - return db.Head().Stats(statsByLabelName), nil + return db.Head().Stats(statsByLabelName, limit), nil case *agent.DB: return nil, agent.ErrUnsupported default: diff --git a/docs/querying/api.md b/docs/querying/api.md index ef7fa54c67..edce366ee6 100644 --- a/docs/querying/api.md +++ b/docs/querying/api.md @@ -1074,6 +1074,10 @@ The following endpoint returns various cardinality statistics about the Promethe ``` GET /api/v1/status/tsdb ``` +URL query parameters: +- `limit=`: Limit the number of returned items to a given number for each set of statistics. By default, 10 items are returned. + +The `data` section of the query result consists of - **headStats**: This provides the following data about the head block of the TSDB: - **numSeries**: The number of series. - **chunkCount**: The number of chunks. diff --git a/tsdb/head.go b/tsdb/head.go index aa7a3c1252..f094b3662c 100644 --- a/tsdb/head.go +++ b/tsdb/head.go @@ -978,7 +978,7 @@ func (h *Head) DisableNativeHistograms() { } // PostingsCardinalityStats returns top 10 highest cardinality stats By label and value names. -func (h *Head) PostingsCardinalityStats(statsByLabelName string) *index.PostingsStats { +func (h *Head) PostingsCardinalityStats(statsByLabelName string, limit int) *index.PostingsStats { h.cardinalityMutex.Lock() defer h.cardinalityMutex.Unlock() currentTime := time.Duration(time.Now().Unix()) * time.Second @@ -989,7 +989,7 @@ func (h *Head) PostingsCardinalityStats(statsByLabelName string) *index.Postings if h.cardinalityCache != nil { return h.cardinalityCache } - h.cardinalityCache = h.postings.Stats(statsByLabelName) + h.cardinalityCache = h.postings.Stats(statsByLabelName, limit) h.lastPostingsStatsCall = time.Duration(time.Now().Unix()) * time.Second return h.cardinalityCache @@ -1329,12 +1329,12 @@ type Stats struct { // Stats returns important current HEAD statistics. Note that it is expensive to // calculate these. -func (h *Head) Stats(statsByLabelName string) *Stats { +func (h *Head) Stats(statsByLabelName string, limit int) *Stats { return &Stats{ NumSeries: h.NumSeries(), MaxTime: h.MaxTime(), MinTime: h.MinTime(), - IndexPostingStats: h.PostingsCardinalityStats(statsByLabelName), + IndexPostingStats: h.PostingsCardinalityStats(statsByLabelName, limit), } } diff --git a/tsdb/index/postings.go b/tsdb/index/postings.go index 200100239c..2ac6edbdca 100644 --- a/tsdb/index/postings.go +++ b/tsdb/index/postings.go @@ -156,10 +156,8 @@ type PostingsStats struct { } // Stats calculates the cardinality statistics from postings. -func (p *MemPostings) Stats(label string) *PostingsStats { - const maxNumOfRecords = 10 +func (p *MemPostings) Stats(label string, limit int) *PostingsStats { var size uint64 - p.mtx.RLock() metrics := &maxHeap{} @@ -168,10 +166,10 @@ func (p *MemPostings) Stats(label string) *PostingsStats { labelValuePairs := &maxHeap{} numLabelPairs := 0 - metrics.init(maxNumOfRecords) - labels.init(maxNumOfRecords) - labelValueLength.init(maxNumOfRecords) - labelValuePairs.init(maxNumOfRecords) + metrics.init(limit) + labels.init(limit) + labelValueLength.init(limit) + labelValuePairs.init(limit) for n, e := range p.m { if n == "" { diff --git a/tsdb/index/postings_test.go b/tsdb/index/postings_test.go index 9d8b5ebf32..9454def467 100644 --- a/tsdb/index/postings_test.go +++ b/tsdb/index/postings_test.go @@ -912,7 +912,7 @@ func BenchmarkPostings_Stats(b *testing.B) { } b.ResetTimer() for n := 0; n < b.N; n++ { - p.Stats("__name__") + p.Stats("__name__", 10) } } @@ -927,7 +927,7 @@ func TestMemPostingsStats(t *testing.T) { p.Add(2, labels.FromStrings("label", "value1")) // call the Stats method to calculate the cardinality statistics - stats := p.Stats("label") + stats := p.Stats("label", 10) // assert that the expected statistics were calculated require.Equal(t, uint64(2), stats.CardinalityMetricsStats[0].Count) diff --git a/web/api/v1/api.go b/web/api/v1/api.go index e1168e1a66..f7249efb04 100644 --- a/web/api/v1/api.go +++ b/web/api/v1/api.go @@ -171,7 +171,7 @@ type TSDBAdminStats interface { CleanTombstones() error Delete(mint, maxt int64, ms ...*labels.Matcher) error Snapshot(dir string, withHead bool) error - Stats(statsByLabelName string) (*tsdb.Stats, error) + Stats(statsByLabelName string, limit int) (*tsdb.Stats, error) WALReplayStatus() (tsdb.WALReplayStatus, error) } @@ -1472,8 +1472,15 @@ func TSDBStatsFromIndexStats(stats []index.Stat) []TSDBStat { return result } -func (api *API) serveTSDBStatus(*http.Request) apiFuncResult { - s, err := api.db.Stats(labels.MetricName) +func (api *API) serveTSDBStatus(r *http.Request) apiFuncResult { + limit := 10 + if s := r.FormValue("limit"); s != "" { + var err error + if limit, err = strconv.Atoi(s); err != nil || limit < 1 { + return apiFuncResult{nil, &apiError{errorBadData, errors.New("limit must be a positive number")}, nil, nil} + } + } + s, err := api.db.Stats(labels.MetricName, limit) if err != nil { return apiFuncResult{nil, &apiError{errorInternal, err}, nil, nil} } diff --git a/web/api/v1/api_test.go b/web/api/v1/api_test.go index 620acd8629..baee2189ef 100644 --- a/web/api/v1/api_test.go +++ b/web/api/v1/api_test.go @@ -2622,7 +2622,7 @@ type fakeDB struct { func (f *fakeDB) CleanTombstones() error { return f.err } func (f *fakeDB) Delete(int64, int64, ...*labels.Matcher) error { return f.err } func (f *fakeDB) Snapshot(string, bool) error { return f.err } -func (f *fakeDB) Stats(statsByLabelName string) (_ *tsdb.Stats, retErr error) { +func (f *fakeDB) Stats(statsByLabelName string, limit int) (_ *tsdb.Stats, retErr error) { dbDir, err := os.MkdirTemp("", "tsdb-api-ready") if err != nil { return nil, err @@ -2636,7 +2636,7 @@ func (f *fakeDB) Stats(statsByLabelName string) (_ *tsdb.Stats, retErr error) { opts := tsdb.DefaultHeadOptions() opts.ChunkRange = 1000 h, _ := tsdb.NewHead(nil, nil, nil, nil, opts, nil) - return h.Stats(statsByLabelName), nil + return h.Stats(statsByLabelName, limit), nil } func (f *fakeDB) WALReplayStatus() (tsdb.WALReplayStatus, error) { @@ -3283,8 +3283,19 @@ func TestTSDBStatus(t *testing.T) { { db: tsdb, endpoint: tsdbStatusAPI, - - errType: errorNone, + errType: errorNone, + }, + { + db: tsdb, + endpoint: tsdbStatusAPI, + values: map[string][]string{"limit": {"20"}}, + errType: errorNone, + }, + { + db: tsdb, + endpoint: tsdbStatusAPI, + values: map[string][]string{"limit": {"0"}}, + errType: errorBadData, }, } { tc := tc diff --git a/web/federate_test.go b/web/federate_test.go index bf7b6fefe7..6bc0ae212a 100644 --- a/web/federate_test.go +++ b/web/federate_test.go @@ -251,7 +251,7 @@ func (notReadyReadStorage) StartTime() (int64, error) { return 0, errors.Wrap(tsdb.ErrNotReady, "wrap") } -func (notReadyReadStorage) Stats(string) (*tsdb.Stats, error) { +func (notReadyReadStorage) Stats(string, int) (*tsdb.Stats, error) { return nil, errors.Wrap(tsdb.ErrNotReady, "wrap") } diff --git a/web/web_test.go b/web/web_test.go index 7da06a3060..8832c28390 100644 --- a/web/web_test.go +++ b/web/web_test.go @@ -52,8 +52,8 @@ type dbAdapter struct { *tsdb.DB } -func (a *dbAdapter) Stats(statsByLabelName string) (*tsdb.Stats, error) { - return a.Head().Stats(statsByLabelName), nil +func (a *dbAdapter) Stats(statsByLabelName string, limit int) (*tsdb.Stats, error) { + return a.Head().Stats(statsByLabelName, limit), nil } func (a *dbAdapter) WALReplayStatus() (tsdb.WALReplayStatus, error) { From 3524a16aa03d7b91a6d0484aa4ed6f1d443837f1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fran=C3=A7ois=20Gouteroux?= Date: Tue, 23 May 2023 10:29:17 +0200 Subject: [PATCH 170/231] feat: add suggested changes, tests, and stdin support MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: François Gouteroux --- cmd/promtool/main.go | 11 +-- cmd/promtool/metrics.go | 157 +++++++--------------------------- docs/command-line/promtool.md | 4 +- util/fmtutil/format.go | 142 ++++++++++++++++++++++++++++++ util/fmtutil/format_test.go | 71 +++++++++++++++ 5 files changed, 251 insertions(+), 134 deletions(-) create mode 100644 util/fmtutil/format.go create mode 100644 util/fmtutil/format_test.go diff --git a/cmd/promtool/main.go b/cmd/promtool/main.go index c4077954ea..3b5ba78e4c 100644 --- a/cmd/promtool/main.go +++ b/cmd/promtool/main.go @@ -81,6 +81,7 @@ func main() { var ( httpRoundTripper = api.DefaultRoundTripper serverURL *url.URL + remoteWriteURL *url.URL httpConfigFilePath string ) @@ -180,12 +181,12 @@ func main() { pushCmd := app.Command("push", "Push to a Prometheus server.") pushCmd.Flag("http.config.file", "HTTP client configuration file for promtool to connect to Prometheus.").PlaceHolder("").ExistingFileVar(&httpConfigFilePath) - pushMetricsCmd := pushCmd.Command("metrics", "Push metrics to a prometheus remote write.") - pushMetricsCmd.Arg("remote-write-url", "Prometheus remote write url to push metrics.").Required().URLVar(&serverURL) + pushMetricsCmd := pushCmd.Command("metrics", "Push metrics to a prometheus remote write (for testing purpose only).") + pushMetricsCmd.Arg("remote-write-url", "Prometheus remote write url to push metrics.").Required().URLVar(&remoteWriteURL) metricFiles := pushMetricsCmd.Arg( "metric-files", - "The metric files to push.", - ).Required().ExistingFiles() + "The metric files to push, default is read from standard input (STDIN).", + ).ExistingFiles() metricJobLabel := pushMetricsCmd.Flag("job-label", "Job label to attach to metrics.").Default("promtool").String() pushMetricsTimeout := pushMetricsCmd.Flag("timeout", "The time to wait for pushing metrics.").Default("30s").Duration() pushMetricsHeaders := pushMetricsCmd.Flag("header", "Prometheus remote write header.").StringMap() @@ -314,7 +315,7 @@ func main() { os.Exit(CheckMetrics(*checkMetricsExtended)) case pushMetricsCmd.FullCommand(): - os.Exit(PushMetrics(serverURL, httpRoundTripper, *pushMetricsHeaders, *pushMetricsTimeout, *metricJobLabel, *metricFiles...)) + os.Exit(PushMetrics(remoteWriteURL, httpRoundTripper, *pushMetricsHeaders, *pushMetricsTimeout, *metricJobLabel, *metricFiles...)) case queryInstantCmd.FullCommand(): os.Exit(QueryInstant(serverURL, httpRoundTripper, *queryInstantExpr, *queryInstantTime, p)) diff --git a/cmd/promtool/metrics.go b/cmd/promtool/metrics.go index 21fcd3e662..c845b5a587 100644 --- a/cmd/promtool/metrics.go +++ b/cmd/promtool/metrics.go @@ -21,22 +21,18 @@ import ( "net/http" "net/url" "os" - "sort" "time" "github.com/golang/snappy" - dto "github.com/prometheus/client_model/go" config_util "github.com/prometheus/common/config" - "github.com/prometheus/common/expfmt" "github.com/prometheus/common/model" - "github.com/prometheus/prometheus/prompb" "github.com/prometheus/prometheus/storage/remote" + "github.com/prometheus/prometheus/util/fmtutil" ) -// Push metrics to a prometheus remote write. +// Push metrics to a prometheus remote write (for testing purpose only). func PushMetrics(url *url.URL, roundTripper http.RoundTripper, headers map[string]string, timeout time.Duration, jobLabel string, files ...string) int { - // remote write should respect specification: https://prometheus.io/docs/concepts/remote_write_spec/ failed := false addressURL, err := url.Parse(url.String()) @@ -67,18 +63,36 @@ func PushMetrics(url *url.URL, roundTripper http.RoundTripper, headers map[strin headers: headers, } - for _, f := range files { + // add empty string to avoid matching filename + if len(files) == 0 { + files = append(files, "") + } + + for _, file := range files { var data []byte var err error - data, err = os.ReadFile(f) - if err != nil { - fmt.Fprintln(os.Stderr, err) - failed = true - continue - } - fmt.Printf("Parsing metric file %s\n", f) - metricsData, err := parseMetricsTextAndFormat(bytes.NewReader(data), jobLabel) + // if file is an empty string it is a stdin + if file == "" { + data, err = io.ReadAll(os.Stdin) + if err != nil { + fmt.Fprintln(os.Stderr, err) + failed = true + break + } + + fmt.Printf("Parsing stdin\n") + } else { + data, err = os.ReadFile(file) + if err != nil { + fmt.Fprintln(os.Stderr, err) + failed = true + continue + } + + fmt.Printf("Parsing metric file %s\n", file) + } + metricsData, err := fmtutil.ParseMetricsTextAndFormat(bytes.NewReader(data), jobLabel) if err != nil { fmt.Fprintln(os.Stderr, err) failed = true @@ -100,7 +114,7 @@ func PushMetrics(url *url.URL, roundTripper http.RoundTripper, headers map[strin failed = true continue } - fmt.Printf("Successfully pushed metric file %s\n", f) + fmt.Printf("Successfully pushed metric file %s\n", file) } if failed { @@ -121,114 +135,3 @@ func (s *setHeadersTransport) RoundTrip(req *http.Request) (*http.Response, erro } return s.RoundTripper.RoundTrip(req) } - -var MetricMetadataTypeValue = map[string]int32{ - "UNKNOWN": 0, - "COUNTER": 1, - "GAUGE": 2, - "HISTOGRAM": 3, - "GAUGEHISTOGRAM": 4, - "SUMMARY": 5, - "INFO": 6, - "STATESET": 7, -} - -// formatMetrics convert metric family to a writerequest -func formatMetrics(mf map[string]*dto.MetricFamily, jobLabel string) (*prompb.WriteRequest, error) { - wr := &prompb.WriteRequest{} - - // build metric list - sortedMetricNames := make([]string, 0, len(mf)) - for metric := range mf { - sortedMetricNames = append(sortedMetricNames, metric) - } - // sort metrics name in lexicographical order - sort.Strings(sortedMetricNames) - - for _, metricName := range sortedMetricNames { - // Set metadata writerequest - mtype := MetricMetadataTypeValue[mf[metricName].Type.String()] - metadata := prompb.MetricMetadata{ - MetricFamilyName: mf[metricName].GetName(), - Type: prompb.MetricMetadata_MetricType(mtype), - Help: mf[metricName].GetHelp(), - } - wr.Metadata = append(wr.Metadata, metadata) - - for _, metric := range mf[metricName].Metric { - var timeserie prompb.TimeSeries - - // build labels map - labels := make(map[string]string, len(metric.Label)+2) - labels[model.MetricNameLabel] = metricName - labels[model.JobLabel] = jobLabel - - for _, label := range metric.Label { - labelname := label.GetName() - if labelname == model.JobLabel { - labelname = fmt.Sprintf("%s%s", model.ExportedLabelPrefix, labelname) - } - labels[labelname] = label.GetValue() - } - - // build labels name list - sortedLabelNames := make([]string, 0, len(labels)) - for label := range labels { - sortedLabelNames = append(sortedLabelNames, label) - } - // sort labels name in lexicographical order - sort.Strings(sortedLabelNames) - - for _, label := range sortedLabelNames { - timeserie.Labels = append(timeserie.Labels, prompb.Label{ - Name: label, - Value: labels[label], - }) - } - - timeserie.Samples = []prompb.Sample{ - { - Timestamp: time.Now().UnixNano() / int64(time.Millisecond), - Value: getMetricsValue(metric), - }, - } - - wr.Timeseries = append(wr.Timeseries, timeserie) - } - } - return wr, nil -} - -// parseMetricsTextReader consumes an io.Reader and returns the MetricFamily -func parseMetricsTextReader(input io.Reader) (map[string]*dto.MetricFamily, error) { - var parser expfmt.TextParser - mf, err := parser.TextToMetricFamilies(input) - if err != nil { - return nil, err - } - return mf, nil -} - -// getMetricsValue return the value of a timeserie without the need to give value type -func getMetricsValue(m *dto.Metric) float64 { - switch { - case m.Gauge != nil: - return m.GetGauge().GetValue() - case m.Counter != nil: - return m.GetCounter().GetValue() - case m.Untyped != nil: - return m.GetUntyped().GetValue() - default: - return 0. - } -} - -// parseMetricsTextAndFormat return the data in the expected prometheus metrics write request format -func parseMetricsTextAndFormat(input io.Reader, jobLabel string) (*prompb.WriteRequest, error) { - mf, err := parseMetricsTextReader(input) - if err != nil { - return nil, err - } - - return formatMetrics(mf, jobLabel) -} diff --git a/docs/command-line/promtool.md b/docs/command-line/promtool.md index ac159a9214..024c71e51a 100644 --- a/docs/command-line/promtool.md +++ b/docs/command-line/promtool.md @@ -390,7 +390,7 @@ Push to a Prometheus server. ##### `promtool push metrics` -Push metrics to a prometheus remote write. +Push metrics to a prometheus remote write (for testing purpose only). @@ -410,7 +410,7 @@ Push metrics to a prometheus remote write. | Argument | Description | Required | | --- | --- | --- | | remote-write-url | Prometheus remote write url to push metrics. | Yes | -| metric-files | The metric files to push. | Yes | +| metric-files | The metric files to push, default is read from standard input (STDIN). | | diff --git a/util/fmtutil/format.go b/util/fmtutil/format.go new file mode 100644 index 0000000000..9a06d6bb15 --- /dev/null +++ b/util/fmtutil/format.go @@ -0,0 +1,142 @@ +// Copyright 2023 The Prometheus Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package fmtutil + +import ( + "fmt" + "io" + "sort" + "time" + + dto "github.com/prometheus/client_model/go" + "github.com/prometheus/common/expfmt" + "github.com/prometheus/common/model" + + "github.com/prometheus/prometheus/prompb" +) + +var MetricMetadataTypeValue = map[string]int32{ + "UNKNOWN": 0, + "COUNTER": 1, + "GAUGE": 2, + "HISTOGRAM": 3, + "GAUGEHISTOGRAM": 4, + "SUMMARY": 5, + "INFO": 6, + "STATESET": 7, +} + +// FormatMetrics convert metric family to a writerequest. +func FormatMetrics(mf map[string]*dto.MetricFamily, jobLabel string) (*prompb.WriteRequest, error) { + wr := &prompb.WriteRequest{} + + // build metric list + sortedMetricNames := make([]string, 0, len(mf)) + for metric := range mf { + sortedMetricNames = append(sortedMetricNames, metric) + } + // sort metrics name in lexicographical order + sort.Strings(sortedMetricNames) + + for _, metricName := range sortedMetricNames { + // Set metadata writerequest + mtype := MetricMetadataTypeValue[mf[metricName].Type.String()] + metadata := prompb.MetricMetadata{ + MetricFamilyName: mf[metricName].GetName(), + Type: prompb.MetricMetadata_MetricType(mtype), + Help: mf[metricName].GetHelp(), + } + wr.Metadata = append(wr.Metadata, metadata) + + for _, metric := range mf[metricName].Metric { + var timeserie prompb.TimeSeries + + // build labels map + labels := make(map[string]string, len(metric.Label)+2) + labels[model.MetricNameLabel] = metricName + labels[model.JobLabel] = jobLabel + + for _, label := range metric.Label { + labelname := label.GetName() + if labelname == model.JobLabel { + labelname = fmt.Sprintf("%s%s", model.ExportedLabelPrefix, labelname) + } + labels[labelname] = label.GetValue() + } + + // build labels name list + sortedLabelNames := make([]string, 0, len(labels)) + for label := range labels { + sortedLabelNames = append(sortedLabelNames, label) + } + // sort labels name in lexicographical order + sort.Strings(sortedLabelNames) + + for _, label := range sortedLabelNames { + timeserie.Labels = append(timeserie.Labels, prompb.Label{ + Name: label, + Value: labels[label], + }) + } + + timestamp := metric.GetTimestampMs() + if timestamp == 0 { + timestamp = time.Now().UnixNano() / int64(time.Millisecond) + } + + timeserie.Samples = []prompb.Sample{ + { + Timestamp: timestamp, + Value: getMetricsValue(metric), + }, + } + + wr.Timeseries = append(wr.Timeseries, timeserie) + } + } + return wr, nil +} + +// getMetricsValue return the value of a timeserie without the need to give value type +func getMetricsValue(m *dto.Metric) float64 { + switch { + case m.Gauge != nil: + return m.GetGauge().GetValue() + case m.Counter != nil: + return m.GetCounter().GetValue() + case m.Untyped != nil: + return m.GetUntyped().GetValue() + default: + return 0. + } +} + +// ParseMetricsTextReader consumes an io.Reader and returns the MetricFamily. +func ParseMetricsTextReader(input io.Reader) (map[string]*dto.MetricFamily, error) { + var parser expfmt.TextParser + mf, err := parser.TextToMetricFamilies(input) + if err != nil { + return nil, err + } + return mf, nil +} + +// ParseMetricsTextAndFormat return the data in the expected prometheus metrics write request format. +func ParseMetricsTextAndFormat(input io.Reader, jobLabel string) (*prompb.WriteRequest, error) { + mf, err := ParseMetricsTextReader(input) + if err != nil { + return nil, err + } + return FormatMetrics(mf, jobLabel) +} diff --git a/util/fmtutil/format_test.go b/util/fmtutil/format_test.go new file mode 100644 index 0000000000..ef3b7fcd40 --- /dev/null +++ b/util/fmtutil/format_test.go @@ -0,0 +1,71 @@ +// Copyright 2023 The Prometheus Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package fmtutil + +import ( + "bytes" + "testing" + + "github.com/stretchr/testify/require" + + "github.com/prometheus/prometheus/prompb" +) + +var writeRequestFixture = &prompb.WriteRequest{ + Metadata: []prompb.MetricMetadata{ + { + MetricFamilyName: "test_metric1", + Type: 2, + Help: "this is a test metric", + }, + }, + Timeseries: []prompb.TimeSeries{ + { + Labels: []prompb.Label{ + {Name: "__name__", Value: "test_metric1"}, + {Name: "b", Value: "c"}, + {Name: "baz", Value: "qux"}, + {Name: "d", Value: "e"}, + {Name: "foo", Value: "bar"}, + {Name: "job", Value: "promtool"}, + }, + Samples: []prompb.Sample{{Value: 1, Timestamp: 1}}, + }, + { + Labels: []prompb.Label{ + {Name: "__name__", Value: "test_metric1"}, + {Name: "b", Value: "c"}, + {Name: "baz", Value: "qux"}, + {Name: "d", Value: "e"}, + {Name: "foo", Value: "bar"}, + {Name: "job", Value: "promtool"}, + }, + Samples: []prompb.Sample{{Value: 2, Timestamp: 1}}, + }, + }, +} + +func TestParseMetricsTextAndFormat(t *testing.T) { + input := bytes.NewReader([]byte(` + # HELP test_metric1 this is a test metric + # TYPE test_metric1 gauge + test_metric1{b="c",baz="qux",d="e",foo="bar"} 1 1 + test_metric1{b="c",baz="qux",d="e",foo="bar"} 2 1 + `)) + + expected, err := ParseMetricsTextAndFormat(input, "promtool") + require.NoError(t, err) + + require.Equal(t, writeRequestFixture, expected) +} From 934c5ddb8d223c6363033a48919e014c63e15ccb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fran=C3=A7ois=20Gouteroux?= Date: Wed, 24 May 2023 10:55:49 +0200 Subject: [PATCH 171/231] feat: make push metrics labels generic and repeatable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: François Gouteroux --- cmd/promtool/main.go | 4 ++-- cmd/promtool/metrics.go | 25 +++++++++++++++---------- docs/command-line/promtool.md | 2 +- util/fmtutil/format.go | 15 ++++++++++----- util/fmtutil/format_test.go | 3 ++- 5 files changed, 30 insertions(+), 19 deletions(-) diff --git a/cmd/promtool/main.go b/cmd/promtool/main.go index 3b5ba78e4c..c76790e13b 100644 --- a/cmd/promtool/main.go +++ b/cmd/promtool/main.go @@ -187,7 +187,7 @@ func main() { "metric-files", "The metric files to push, default is read from standard input (STDIN).", ).ExistingFiles() - metricJobLabel := pushMetricsCmd.Flag("job-label", "Job label to attach to metrics.").Default("promtool").String() + pushMetricsLabels := pushMetricsCmd.Flag("label", "Label to attach to metrics. Can be specified multiple times.").Default("job=promtool").StringMap() pushMetricsTimeout := pushMetricsCmd.Flag("timeout", "The time to wait for pushing metrics.").Default("30s").Duration() pushMetricsHeaders := pushMetricsCmd.Flag("header", "Prometheus remote write header.").StringMap() @@ -315,7 +315,7 @@ func main() { os.Exit(CheckMetrics(*checkMetricsExtended)) case pushMetricsCmd.FullCommand(): - os.Exit(PushMetrics(remoteWriteURL, httpRoundTripper, *pushMetricsHeaders, *pushMetricsTimeout, *metricJobLabel, *metricFiles...)) + os.Exit(PushMetrics(remoteWriteURL, httpRoundTripper, *pushMetricsHeaders, *pushMetricsTimeout, *pushMetricsLabels, *metricFiles...)) case queryInstantCmd.FullCommand(): os.Exit(QueryInstant(serverURL, httpRoundTripper, *queryInstantExpr, *queryInstantTime, p)) diff --git a/cmd/promtool/metrics.go b/cmd/promtool/metrics.go index c845b5a587..8abe32cf41 100644 --- a/cmd/promtool/metrics.go +++ b/cmd/promtool/metrics.go @@ -32,7 +32,7 @@ import ( ) // Push metrics to a prometheus remote write (for testing purpose only). -func PushMetrics(url *url.URL, roundTripper http.RoundTripper, headers map[string]string, timeout time.Duration, jobLabel string, files ...string) int { +func PushMetrics(url *url.URL, roundTripper http.RoundTripper, headers map[string]string, timeout time.Duration, labels map[string]string, files ...string) int { failed := false addressURL, err := url.Parse(url.String()) @@ -76,32 +76,32 @@ func PushMetrics(url *url.URL, roundTripper http.RoundTripper, headers map[strin if file == "" { data, err = io.ReadAll(os.Stdin) if err != nil { - fmt.Fprintln(os.Stderr, err) + fmt.Fprintln(os.Stderr, " FAILED:", err) failed = true break } - fmt.Printf("Parsing stdin\n") + fmt.Printf("Parsing input from stdin\n") } else { data, err = os.ReadFile(file) if err != nil { - fmt.Fprintln(os.Stderr, err) + fmt.Fprintln(os.Stderr, " FAILED:", err) failed = true continue } - fmt.Printf("Parsing metric file %s\n", file) + fmt.Printf("Parsing input from metric file %s\n", file) } - metricsData, err := fmtutil.ParseMetricsTextAndFormat(bytes.NewReader(data), jobLabel) + metricsData, err := fmtutil.ParseMetricsTextAndFormat(bytes.NewReader(data), labels) if err != nil { - fmt.Fprintln(os.Stderr, err) + fmt.Fprintln(os.Stderr, " FAILED:", err) failed = true continue } raw, err := metricsData.Marshal() if err != nil { - fmt.Fprintln(os.Stderr, err) + fmt.Fprintln(os.Stderr, " FAILED:", err) failed = true continue } @@ -110,11 +110,16 @@ func PushMetrics(url *url.URL, roundTripper http.RoundTripper, headers map[strin compressed := snappy.Encode(nil, raw) err = client.Store(context.Background(), compressed) if err != nil { - fmt.Fprintln(os.Stderr, err) + fmt.Fprintln(os.Stderr, " FAILED:", err) failed = true continue } - fmt.Printf("Successfully pushed metric file %s\n", file) + + if file == "" { + fmt.Printf(" SUCCESS: metric pushed to remote write.\n") + } else { + fmt.Printf(" SUCCESS: metric file %s pushed to remote write.\n", file) + } } if failed { diff --git a/docs/command-line/promtool.md b/docs/command-line/promtool.md index 024c71e51a..c78900b991 100644 --- a/docs/command-line/promtool.md +++ b/docs/command-line/promtool.md @@ -398,7 +398,7 @@ Push metrics to a prometheus remote write (for testing purpose only). | Flag | Description | Default | | --- | --- | --- | -| --job-label | Job label to attach to metrics. | `promtool` | +| --label | Label to attach to metrics. Can be specified multiple times. | `job=promtool` | | --timeout | The time to wait for pushing metrics. | `30s` | | --header | Prometheus remote write header. | | diff --git a/util/fmtutil/format.go b/util/fmtutil/format.go index 9a06d6bb15..b5bb9469ce 100644 --- a/util/fmtutil/format.go +++ b/util/fmtutil/format.go @@ -38,7 +38,7 @@ var MetricMetadataTypeValue = map[string]int32{ } // FormatMetrics convert metric family to a writerequest. -func FormatMetrics(mf map[string]*dto.MetricFamily, jobLabel string) (*prompb.WriteRequest, error) { +func FormatMetrics(mf map[string]*dto.MetricFamily, extraLabels map[string]string) (*prompb.WriteRequest, error) { wr := &prompb.WriteRequest{} // build metric list @@ -63,10 +63,15 @@ func FormatMetrics(mf map[string]*dto.MetricFamily, jobLabel string) (*prompb.Wr var timeserie prompb.TimeSeries // build labels map - labels := make(map[string]string, len(metric.Label)+2) + labels := make(map[string]string, len(metric.Label)+len(extraLabels)) labels[model.MetricNameLabel] = metricName - labels[model.JobLabel] = jobLabel + // add extra labels + for key, value := range extraLabels { + labels[key] = value + } + + // add metric labels for _, label := range metric.Label { labelname := label.GetName() if labelname == model.JobLabel { @@ -133,10 +138,10 @@ func ParseMetricsTextReader(input io.Reader) (map[string]*dto.MetricFamily, erro } // ParseMetricsTextAndFormat return the data in the expected prometheus metrics write request format. -func ParseMetricsTextAndFormat(input io.Reader, jobLabel string) (*prompb.WriteRequest, error) { +func ParseMetricsTextAndFormat(input io.Reader, labels map[string]string) (*prompb.WriteRequest, error) { mf, err := ParseMetricsTextReader(input) if err != nil { return nil, err } - return FormatMetrics(mf, jobLabel) + return FormatMetrics(mf, labels) } diff --git a/util/fmtutil/format_test.go b/util/fmtutil/format_test.go index ef3b7fcd40..9deed2de90 100644 --- a/util/fmtutil/format_test.go +++ b/util/fmtutil/format_test.go @@ -63,8 +63,9 @@ func TestParseMetricsTextAndFormat(t *testing.T) { test_metric1{b="c",baz="qux",d="e",foo="bar"} 1 1 test_metric1{b="c",baz="qux",d="e",foo="bar"} 2 1 `)) + labels := map[string]string{"job": "promtool"} - expected, err := ParseMetricsTextAndFormat(input, "promtool") + expected, err := ParseMetricsTextAndFormat(input, labels) require.NoError(t, err) require.Equal(t, writeRequestFixture, expected) From 37e5249e33e4c2d7bc49c4ac587faf12eb6785b7 Mon Sep 17 00:00:00 2001 From: zenador Date: Wed, 24 May 2023 19:00:21 +0800 Subject: [PATCH 172/231] Use DefaultSamplesPerChunk in tsdb (#12387) Signed-off-by: Jeanette Tan --- tsdb/db.go | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tsdb/db.go b/tsdb/db.go index a0d0a4b260..12974150be 100644 --- a/tsdb/db.go +++ b/tsdb/db.go @@ -640,6 +640,9 @@ func validateOpts(opts *Options, rngs []int64) (*Options, []int64) { if opts.HeadChunksWriteQueueSize < 0 { opts.HeadChunksWriteQueueSize = chunks.DefaultWriteQueueSize } + if opts.SamplesPerChunk <= 0 { + opts.SamplesPerChunk = DefaultSamplesPerChunk + } if opts.MaxBlockChunkSegmentSize <= 0 { opts.MaxBlockChunkSegmentSize = chunks.DefaultChunkSegmentSize } From 89af3517304bcd48a8f20686c0e3317544b6985d Mon Sep 17 00:00:00 2001 From: Justin Lei <97976793+leizor@users.noreply.github.com> Date: Thu, 25 May 2023 02:18:41 -0700 Subject: [PATCH 173/231] Remove samplesPerChunk from memSeries (#12390) Signed-off-by: Justin Lei --- tsdb/head.go | 14 +++++----- tsdb/head_append.go | 24 ++++++++--------- tsdb/head_test.go | 66 ++++++++++++++++++++++----------------------- tsdb/head_wal.go | 6 ++--- 4 files changed, 54 insertions(+), 56 deletions(-) diff --git a/tsdb/head.go b/tsdb/head.go index f094b3662c..a1d61fd6ab 100644 --- a/tsdb/head.go +++ b/tsdb/head.go @@ -1614,7 +1614,7 @@ func (h *Head) getOrCreate(hash uint64, lset labels.Labels) (*memSeries, bool, e func (h *Head) getOrCreateWithID(id chunks.HeadSeriesRef, hash uint64, lset labels.Labels) (*memSeries, bool, error) { s, created, err := h.series.getOrSet(hash, lset, func() *memSeries { - return newMemSeries(lset, id, h.opts.IsolationDisabled, h.opts.SamplesPerChunk) + return newMemSeries(lset, id, h.opts.IsolationDisabled) }) if err != nil { return nil, false, err @@ -1922,8 +1922,7 @@ type memSeries struct { mmMaxTime int64 // Max time of any mmapped chunk, only used during WAL replay. - samplesPerChunk int // Target number of samples per chunk. - nextAt int64 // Timestamp at which to cut the next chunk. + nextAt int64 // Timestamp at which to cut the next chunk. // We keep the last value here (in addition to appending it to the chunk) so we can check for duplicates. lastValue float64 @@ -1951,12 +1950,11 @@ type memSeriesOOOFields struct { firstOOOChunkID chunks.HeadChunkID // HeadOOOChunkID for oooMmappedChunks[0]. } -func newMemSeries(lset labels.Labels, id chunks.HeadSeriesRef, isolationDisabled bool, samplesPerChunk int) *memSeries { +func newMemSeries(lset labels.Labels, id chunks.HeadSeriesRef, isolationDisabled bool) *memSeries { s := &memSeries{ - lset: lset, - ref: id, - nextAt: math.MinInt64, - samplesPerChunk: samplesPerChunk, + lset: lset, + ref: id, + nextAt: math.MinInt64, } if !isolationDisabled { s.txs = newTxRing(4) diff --git a/tsdb/head_append.go b/tsdb/head_append.go index 060d32b7f7..44847dceb2 100644 --- a/tsdb/head_append.go +++ b/tsdb/head_append.go @@ -987,7 +987,7 @@ func (a *headAppender) Commit() (err error) { samplesAppended-- } default: - ok, chunkCreated = series.append(s.T, s.V, a.appendID, a.head.chunkDiskMapper, chunkRange) + ok, chunkCreated = series.append(s.T, s.V, a.appendID, a.head.chunkDiskMapper, chunkRange, a.head.opts.SamplesPerChunk) if ok { if s.T < inOrderMint { inOrderMint = s.T @@ -1016,7 +1016,7 @@ func (a *headAppender) Commit() (err error) { for i, s := range a.histograms { series = a.histogramSeries[i] series.Lock() - ok, chunkCreated := series.appendHistogram(s.T, s.H, a.appendID, a.head.chunkDiskMapper, chunkRange) + ok, chunkCreated := series.appendHistogram(s.T, s.H, a.appendID, a.head.chunkDiskMapper, chunkRange, a.head.opts.SamplesPerChunk) series.cleanupAppendIDsBelow(a.cleanupAppendIDsBelow) series.pendingCommit = false series.Unlock() @@ -1042,7 +1042,7 @@ func (a *headAppender) Commit() (err error) { for i, s := range a.floatHistograms { series = a.floatHistogramSeries[i] series.Lock() - ok, chunkCreated := series.appendFloatHistogram(s.T, s.FH, a.appendID, a.head.chunkDiskMapper, chunkRange) + ok, chunkCreated := series.appendFloatHistogram(s.T, s.FH, a.appendID, a.head.chunkDiskMapper, chunkRange, a.head.opts.SamplesPerChunk) series.cleanupAppendIDsBelow(a.cleanupAppendIDsBelow) series.pendingCommit = false series.Unlock() @@ -1122,8 +1122,8 @@ func (s *memSeries) insert(t int64, v float64, chunkDiskMapper *chunks.ChunkDisk // the appendID for isolation. (The appendID can be zero, which results in no // isolation for this append.) // It is unsafe to call this concurrently with s.iterator(...) without holding the series lock. -func (s *memSeries) append(t int64, v float64, appendID uint64, chunkDiskMapper *chunks.ChunkDiskMapper, chunkRange int64) (sampleInOrder, chunkCreated bool) { - c, sampleInOrder, chunkCreated := s.appendPreprocessor(t, chunkenc.EncXOR, chunkDiskMapper, chunkRange) +func (s *memSeries) append(t int64, v float64, appendID uint64, chunkDiskMapper *chunks.ChunkDiskMapper, chunkRange int64, samplesPerChunk int) (sampleInOrder, chunkCreated bool) { + c, sampleInOrder, chunkCreated := s.appendPreprocessor(t, chunkenc.EncXOR, chunkDiskMapper, chunkRange, samplesPerChunk) if !sampleInOrder { return sampleInOrder, chunkCreated } @@ -1144,7 +1144,7 @@ func (s *memSeries) append(t int64, v float64, appendID uint64, chunkDiskMapper // appendHistogram adds the histogram. // It is unsafe to call this concurrently with s.iterator(...) without holding the series lock. -func (s *memSeries) appendHistogram(t int64, h *histogram.Histogram, appendID uint64, chunkDiskMapper *chunks.ChunkDiskMapper, chunkRange int64) (sampleInOrder, chunkCreated bool) { +func (s *memSeries) appendHistogram(t int64, h *histogram.Histogram, appendID uint64, chunkDiskMapper *chunks.ChunkDiskMapper, chunkRange int64, samplesPerChunk int) (sampleInOrder, chunkCreated bool) { // Head controls the execution of recoding, so that we own the proper // chunk reference afterwards. We check for Appendable from appender before // appendPreprocessor because in case it ends up creating a new chunk, @@ -1157,7 +1157,7 @@ func (s *memSeries) appendHistogram(t int64, h *histogram.Histogram, appendID ui pMergedSpans, nMergedSpans []histogram.Span okToAppend, counterReset, gauge bool ) - c, sampleInOrder, chunkCreated := s.appendPreprocessor(t, chunkenc.EncHistogram, chunkDiskMapper, chunkRange) + c, sampleInOrder, chunkCreated := s.appendPreprocessor(t, chunkenc.EncHistogram, chunkDiskMapper, chunkRange, samplesPerChunk) if !sampleInOrder { return sampleInOrder, chunkCreated } @@ -1238,7 +1238,7 @@ func (s *memSeries) appendHistogram(t int64, h *histogram.Histogram, appendID ui // appendFloatHistogram adds the float histogram. // It is unsafe to call this concurrently with s.iterator(...) without holding the series lock. -func (s *memSeries) appendFloatHistogram(t int64, fh *histogram.FloatHistogram, appendID uint64, chunkDiskMapper *chunks.ChunkDiskMapper, chunkRange int64) (sampleInOrder, chunkCreated bool) { +func (s *memSeries) appendFloatHistogram(t int64, fh *histogram.FloatHistogram, appendID uint64, chunkDiskMapper *chunks.ChunkDiskMapper, chunkRange int64, samplesPerChunk int) (sampleInOrder, chunkCreated bool) { // Head controls the execution of recoding, so that we own the proper // chunk reference afterwards. We check for Appendable from appender before // appendPreprocessor because in case it ends up creating a new chunk, @@ -1251,7 +1251,7 @@ func (s *memSeries) appendFloatHistogram(t int64, fh *histogram.FloatHistogram, pMergedSpans, nMergedSpans []histogram.Span okToAppend, counterReset, gauge bool ) - c, sampleInOrder, chunkCreated := s.appendPreprocessor(t, chunkenc.EncFloatHistogram, chunkDiskMapper, chunkRange) + c, sampleInOrder, chunkCreated := s.appendPreprocessor(t, chunkenc.EncFloatHistogram, chunkDiskMapper, chunkRange, samplesPerChunk) if !sampleInOrder { return sampleInOrder, chunkCreated } @@ -1334,7 +1334,7 @@ func (s *memSeries) appendFloatHistogram(t int64, fh *histogram.FloatHistogram, // It is unsafe to call this concurrently with s.iterator(...) without holding the series lock. // This should be called only when appending data. func (s *memSeries) appendPreprocessor( - t int64, e chunkenc.Encoding, chunkDiskMapper *chunks.ChunkDiskMapper, chunkRange int64, + t int64, e chunkenc.Encoding, chunkDiskMapper *chunks.ChunkDiskMapper, chunkRange int64, samplesPerChunk int, ) (c *memChunk, sampleInOrder, chunkCreated bool) { c = s.head() @@ -1372,7 +1372,7 @@ func (s *memSeries) appendPreprocessor( // for this chunk that will try to make samples equally distributed within // the remaining chunks in the current chunk range. // At latest it must happen at the timestamp set when the chunk was cut. - if numSamples == s.samplesPerChunk/4 { + if numSamples == samplesPerChunk/4 { s.nextAt = computeChunkEndTime(c.minTime, c.maxTime, s.nextAt) } // If numSamples > samplesPerChunk*2 then our previous prediction was invalid, @@ -1380,7 +1380,7 @@ func (s *memSeries) appendPreprocessor( // Since we assume that the rate is higher, we're being conservative and cutting at 2*samplesPerChunk // as we expect more chunks to come. // Note that next chunk will have its nextAt recalculated for the new rate. - if t >= s.nextAt || numSamples >= s.samplesPerChunk*2 { + if t >= s.nextAt || numSamples >= samplesPerChunk*2 { c = s.cutNewHeadChunk(t, e, chunkDiskMapper, chunkRange) chunkCreated = true } diff --git a/tsdb/head_test.go b/tsdb/head_test.go index af3df378e7..14468e0716 100644 --- a/tsdb/head_test.go +++ b/tsdb/head_test.go @@ -285,8 +285,8 @@ func BenchmarkLoadWAL(b *testing.B) { require.NoError(b, err) for k := 0; k < c.batches*c.seriesPerBatch; k++ { // Create one mmapped chunk per series, with one sample at the given time. - s := newMemSeries(labels.Labels{}, chunks.HeadSeriesRef(k)*101, defaultIsolationDisabled, DefaultSamplesPerChunk) - s.append(c.mmappedChunkT, 42, 0, chunkDiskMapper, c.mmappedChunkT) + s := newMemSeries(labels.Labels{}, chunks.HeadSeriesRef(k)*101, defaultIsolationDisabled) + s.append(c.mmappedChunkT, 42, 0, chunkDiskMapper, c.mmappedChunkT, DefaultSamplesPerChunk) s.mmapCurrentHeadChunk(chunkDiskMapper) } require.NoError(b, chunkDiskMapper.Close()) @@ -807,10 +807,10 @@ func TestMemSeries_truncateChunks(t *testing.T) { }, } - s := newMemSeries(labels.FromStrings("a", "b"), 1, defaultIsolationDisabled, DefaultSamplesPerChunk) + s := newMemSeries(labels.FromStrings("a", "b"), 1, defaultIsolationDisabled) for i := 0; i < 4000; i += 5 { - ok, _ := s.append(int64(i), float64(i), 0, chunkDiskMapper, chunkRange) + ok, _ := s.append(int64(i), float64(i), 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "sample append failed") } @@ -1338,24 +1338,24 @@ func TestMemSeries_append(t *testing.T) { }() const chunkRange = 500 - s := newMemSeries(labels.Labels{}, 1, defaultIsolationDisabled, DefaultSamplesPerChunk) + s := newMemSeries(labels.Labels{}, 1, defaultIsolationDisabled) // Add first two samples at the very end of a chunk range and the next two // on and after it. // New chunk must correctly be cut at 1000. - ok, chunkCreated := s.append(998, 1, 0, chunkDiskMapper, chunkRange) + ok, chunkCreated := s.append(998, 1, 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "append failed") require.True(t, chunkCreated, "first sample created chunk") - ok, chunkCreated = s.append(999, 2, 0, chunkDiskMapper, chunkRange) + ok, chunkCreated = s.append(999, 2, 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "append failed") require.False(t, chunkCreated, "second sample should use same chunk") - ok, chunkCreated = s.append(1000, 3, 0, chunkDiskMapper, chunkRange) + ok, chunkCreated = s.append(1000, 3, 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "append failed") require.True(t, chunkCreated, "expected new chunk on boundary") - ok, chunkCreated = s.append(1001, 4, 0, chunkDiskMapper, chunkRange) + ok, chunkCreated = s.append(1001, 4, 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "append failed") require.False(t, chunkCreated, "second sample should use same chunk") @@ -1368,7 +1368,7 @@ func TestMemSeries_append(t *testing.T) { // Fill the range [1000,2000) with many samples. Intermediate chunks should be cut // at approximately 120 samples per chunk. for i := 1; i < 1000; i++ { - ok, _ := s.append(1001+int64(i), float64(i), 0, chunkDiskMapper, chunkRange) + ok, _ := s.append(1001+int64(i), float64(i), 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "append failed") } @@ -1392,7 +1392,7 @@ func TestMemSeries_appendHistogram(t *testing.T) { }() chunkRange := int64(1000) - s := newMemSeries(labels.Labels{}, 1, defaultIsolationDisabled, DefaultSamplesPerChunk) + s := newMemSeries(labels.Labels{}, 1, defaultIsolationDisabled) histograms := tsdbutil.GenerateTestHistograms(4) histogramWithOneMoreBucket := histograms[3].Copy() @@ -1404,19 +1404,19 @@ func TestMemSeries_appendHistogram(t *testing.T) { // Add first two samples at the very end of a chunk range and the next two // on and after it. // New chunk must correctly be cut at 1000. - ok, chunkCreated := s.appendHistogram(998, histograms[0], 0, chunkDiskMapper, chunkRange) + ok, chunkCreated := s.appendHistogram(998, histograms[0], 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "append failed") require.True(t, chunkCreated, "first sample created chunk") - ok, chunkCreated = s.appendHistogram(999, histograms[1], 0, chunkDiskMapper, chunkRange) + ok, chunkCreated = s.appendHistogram(999, histograms[1], 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "append failed") require.False(t, chunkCreated, "second sample should use same chunk") - ok, chunkCreated = s.appendHistogram(1000, histograms[2], 0, chunkDiskMapper, chunkRange) + ok, chunkCreated = s.appendHistogram(1000, histograms[2], 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "append failed") require.True(t, chunkCreated, "expected new chunk on boundary") - ok, chunkCreated = s.appendHistogram(1001, histograms[3], 0, chunkDiskMapper, chunkRange) + ok, chunkCreated = s.appendHistogram(1001, histograms[3], 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "append failed") require.False(t, chunkCreated, "second sample should use same chunk") @@ -1426,7 +1426,7 @@ func TestMemSeries_appendHistogram(t *testing.T) { require.Equal(t, int64(1000), s.headChunk.minTime, "wrong chunk range") require.Equal(t, int64(1001), s.headChunk.maxTime, "wrong chunk range") - ok, chunkCreated = s.appendHistogram(1002, histogramWithOneMoreBucket, 0, chunkDiskMapper, chunkRange) + ok, chunkCreated = s.appendHistogram(1002, histogramWithOneMoreBucket, 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "append failed") require.False(t, chunkCreated, "third sample should trigger a re-encoded chunk") @@ -1448,7 +1448,7 @@ func TestMemSeries_append_atVariableRate(t *testing.T) { }) chunkRange := DefaultBlockDuration - s := newMemSeries(labels.Labels{}, 1, defaultIsolationDisabled, DefaultSamplesPerChunk) + s := newMemSeries(labels.Labels{}, 1, defaultIsolationDisabled) // At this slow rate, we will fill the chunk in two block durations. slowRate := (DefaultBlockDuration * 2) / samplesPerChunk @@ -1456,7 +1456,7 @@ func TestMemSeries_append_atVariableRate(t *testing.T) { var nextTs int64 var totalAppendedSamples int for i := 0; i < samplesPerChunk/4; i++ { - ok, _ := s.append(nextTs, float64(i), 0, chunkDiskMapper, chunkRange) + ok, _ := s.append(nextTs, float64(i), 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.Truef(t, ok, "slow sample %d was not appended", i) nextTs += slowRate totalAppendedSamples++ @@ -1465,12 +1465,12 @@ func TestMemSeries_append_atVariableRate(t *testing.T) { // Suddenly, the rate increases and we receive a sample every millisecond. for i := 0; i < math.MaxUint16; i++ { - ok, _ := s.append(nextTs, float64(i), 0, chunkDiskMapper, chunkRange) + ok, _ := s.append(nextTs, float64(i), 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.Truef(t, ok, "quick sample %d was not appended", i) nextTs++ totalAppendedSamples++ } - ok, chunkCreated := s.append(DefaultBlockDuration, float64(0), 0, chunkDiskMapper, chunkRange) + ok, chunkCreated := s.append(DefaultBlockDuration, float64(0), 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "new chunk sample was not appended") require.True(t, chunkCreated, "sample at block duration timestamp should create a new chunk") @@ -1495,18 +1495,18 @@ func TestGCChunkAccess(t *testing.T) { s, _, _ := h.getOrCreate(1, labels.FromStrings("a", "1")) // Appending 2 samples for the first chunk. - ok, chunkCreated := s.append(0, 0, 0, h.chunkDiskMapper, chunkRange) + ok, chunkCreated := s.append(0, 0, 0, h.chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "series append failed") require.True(t, chunkCreated, "chunks was not created") - ok, chunkCreated = s.append(999, 999, 0, h.chunkDiskMapper, chunkRange) + ok, chunkCreated = s.append(999, 999, 0, h.chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "series append failed") require.False(t, chunkCreated, "chunks was created") // A new chunks should be created here as it's beyond the chunk range. - ok, chunkCreated = s.append(1000, 1000, 0, h.chunkDiskMapper, chunkRange) + ok, chunkCreated = s.append(1000, 1000, 0, h.chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "series append failed") require.True(t, chunkCreated, "chunks was not created") - ok, chunkCreated = s.append(1999, 1999, 0, h.chunkDiskMapper, chunkRange) + ok, chunkCreated = s.append(1999, 1999, 0, h.chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "series append failed") require.False(t, chunkCreated, "chunks was created") @@ -1548,18 +1548,18 @@ func TestGCSeriesAccess(t *testing.T) { s, _, _ := h.getOrCreate(1, labels.FromStrings("a", "1")) // Appending 2 samples for the first chunk. - ok, chunkCreated := s.append(0, 0, 0, h.chunkDiskMapper, chunkRange) + ok, chunkCreated := s.append(0, 0, 0, h.chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "series append failed") require.True(t, chunkCreated, "chunks was not created") - ok, chunkCreated = s.append(999, 999, 0, h.chunkDiskMapper, chunkRange) + ok, chunkCreated = s.append(999, 999, 0, h.chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "series append failed") require.False(t, chunkCreated, "chunks was created") // A new chunks should be created here as it's beyond the chunk range. - ok, chunkCreated = s.append(1000, 1000, 0, h.chunkDiskMapper, chunkRange) + ok, chunkCreated = s.append(1000, 1000, 0, h.chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "series append failed") require.True(t, chunkCreated, "chunks was not created") - ok, chunkCreated = s.append(1999, 1999, 0, h.chunkDiskMapper, chunkRange) + ok, chunkCreated = s.append(1999, 1999, 0, h.chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "series append failed") require.False(t, chunkCreated, "chunks was created") @@ -1795,10 +1795,10 @@ func TestHeadReadWriterRepair(t *testing.T) { require.True(t, created, "series was not created") for i := 0; i < 7; i++ { - ok, chunkCreated := s.append(int64(i*chunkRange), float64(i*chunkRange), 0, h.chunkDiskMapper, chunkRange) + ok, chunkCreated := s.append(int64(i*chunkRange), float64(i*chunkRange), 0, h.chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "series append failed") require.True(t, chunkCreated, "chunk was not created") - ok, chunkCreated = s.append(int64(i*chunkRange)+chunkRange-1, float64(i*chunkRange), 0, h.chunkDiskMapper, chunkRange) + ok, chunkCreated = s.append(int64(i*chunkRange)+chunkRange-1, float64(i*chunkRange), 0, h.chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "series append failed") require.False(t, chunkCreated, "chunk was created") h.chunkDiskMapper.CutNewFile() @@ -2146,7 +2146,7 @@ func TestIsolationAppendIDZeroIsNoop(t *testing.T) { s, _, _ := h.getOrCreate(1, labels.FromStrings("a", "1")) - ok, _ := s.append(0, 0, 0, h.chunkDiskMapper, h.chunkRange.Load()) + ok, _ := s.append(0, 0, 0, h.chunkDiskMapper, h.chunkRange.Load(), DefaultSamplesPerChunk) require.True(t, ok, "Series append failed.") require.Equal(t, 0, s.txs.txIDCount, "Series should not have an appendID after append with appendID=0.") } @@ -2610,10 +2610,10 @@ func TestIteratorSeekIntoBuffer(t *testing.T) { }() const chunkRange = 500 - s := newMemSeries(labels.Labels{}, 1, defaultIsolationDisabled, DefaultSamplesPerChunk) + s := newMemSeries(labels.Labels{}, 1, defaultIsolationDisabled) for i := 0; i < 7; i++ { - ok, _ := s.append(int64(i), float64(i), 0, chunkDiskMapper, chunkRange) + ok, _ := s.append(int64(i), float64(i), 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) require.True(t, ok, "sample append failed") } diff --git a/tsdb/head_wal.go b/tsdb/head_wal.go index 6e81f17932..9741d1da04 100644 --- a/tsdb/head_wal.go +++ b/tsdb/head_wal.go @@ -588,7 +588,7 @@ func (wp *walSubsetProcessor) processWALSamples(h *Head, mmappedChunks, oooMmapp if s.T <= ms.mmMaxTime { continue } - if _, chunkCreated := ms.append(s.T, s.V, 0, h.chunkDiskMapper, chunkRange); chunkCreated { + if _, chunkCreated := ms.append(s.T, s.V, 0, h.chunkDiskMapper, chunkRange, h.opts.SamplesPerChunk); chunkCreated { h.metrics.chunksCreated.Inc() h.metrics.chunks.Inc() } @@ -618,9 +618,9 @@ func (wp *walSubsetProcessor) processWALSamples(h *Head, mmappedChunks, oooMmapp } var chunkCreated bool if s.h != nil { - _, chunkCreated = ms.appendHistogram(s.t, s.h, 0, h.chunkDiskMapper, chunkRange) + _, chunkCreated = ms.appendHistogram(s.t, s.h, 0, h.chunkDiskMapper, chunkRange, h.opts.SamplesPerChunk) } else { - _, chunkCreated = ms.appendFloatHistogram(s.t, s.fh, 0, h.chunkDiskMapper, chunkRange) + _, chunkCreated = ms.appendFloatHistogram(s.t, s.fh, 0, h.chunkDiskMapper, chunkRange, h.opts.SamplesPerChunk) } if chunkCreated { h.metrics.chunksCreated.Inc() From cb045c0e4b94bbf3eee174d91b5ef2b8553948d5 Mon Sep 17 00:00:00 2001 From: Julius Volz Date: Thu, 25 May 2023 11:49:43 +0200 Subject: [PATCH 174/231] Fix wording from "jitterSeed" -> "offsetSeed" for server-wide scrape offsets In digital communication, "jitter" usually refers to how much a signal deviates from true periodicity, see https://en.wikipedia.org/wiki/Jitter. The way we are using the "jitterSeed" in Prometheus does not affect the true periodicity at all, but just introduces a constant phase shift (or offset) within the period. So it would be more correct and less confusing to call the "jitterSeed" an "offsetSeed" instead. Signed-off-by: Julius Volz --- scrape/manager.go | 12 ++++++------ scrape/manager_test.go | 18 +++++++++--------- scrape/scrape.go | 14 +++++++------- scrape/target_test.go | 4 ++-- 4 files changed, 24 insertions(+), 24 deletions(-) diff --git a/scrape/manager.go b/scrape/manager.go index d75fe30cf5..d7cf6792c2 100644 --- a/scrape/manager.go +++ b/scrape/manager.go @@ -150,7 +150,7 @@ type Manager struct { append storage.Appendable graceShut chan struct{} - jitterSeed uint64 // Global jitterSeed seed is used to spread scrape workload across HA setup. + offsetSeed uint64 // Global offsetSeed seed is used to spread scrape workload across HA setup. mtxScrape sync.Mutex // Guards the fields below. scrapeConfigs map[string]*config.ScrapeConfig scrapePools map[string]*scrapePool @@ -214,7 +214,7 @@ func (m *Manager) reload() { level.Error(m.logger).Log("msg", "error reloading target set", "err", "invalid config id:"+setName) continue } - sp, err := newScrapePool(scrapeConfig, m.append, m.jitterSeed, log.With(m.logger, "scrape_pool", setName), m.opts) + sp, err := newScrapePool(scrapeConfig, m.append, m.offsetSeed, log.With(m.logger, "scrape_pool", setName), m.opts) if err != nil { level.Error(m.logger).Log("msg", "error creating new scrape pool", "err", err, "scrape_pool", setName) continue @@ -234,8 +234,8 @@ func (m *Manager) reload() { wg.Wait() } -// setJitterSeed calculates a global jitterSeed per server relying on extra label set. -func (m *Manager) setJitterSeed(labels labels.Labels) error { +// setOffsetSeed calculates a global offsetSeed per server relying on extra label set. +func (m *Manager) setOffsetSeed(labels labels.Labels) error { h := fnv.New64a() hostname, err := osutil.GetFQDN() if err != nil { @@ -244,7 +244,7 @@ func (m *Manager) setJitterSeed(labels labels.Labels) error { if _, err := fmt.Fprintf(h, "%s%s", hostname, labels.String()); err != nil { return err } - m.jitterSeed = h.Sum64() + m.offsetSeed = h.Sum64() return nil } @@ -281,7 +281,7 @@ func (m *Manager) ApplyConfig(cfg *config.Config) error { } m.scrapeConfigs = c - if err := m.setJitterSeed(cfg.GlobalConfig.ExternalLabels); err != nil { + if err := m.setOffsetSeed(cfg.GlobalConfig.ExternalLabels); err != nil { return err } diff --git a/scrape/manager_test.go b/scrape/manager_test.go index d05d25fa2c..50f6320137 100644 --- a/scrape/manager_test.go +++ b/scrape/manager_test.go @@ -596,7 +596,7 @@ func TestManagerTargetsUpdates(t *testing.T) { } } -func TestSetJitter(t *testing.T) { +func TestSetOffsetSeed(t *testing.T) { getConfig := func(prometheus string) *config.Config { cfgText := ` global: @@ -617,24 +617,24 @@ global: // Load the first config. cfg1 := getConfig("ha1") - if err := scrapeManager.setJitterSeed(cfg1.GlobalConfig.ExternalLabels); err != nil { + if err := scrapeManager.setOffsetSeed(cfg1.GlobalConfig.ExternalLabels); err != nil { t.Error(err) } - jitter1 := scrapeManager.jitterSeed + offsetSeed1 := scrapeManager.offsetSeed - if jitter1 == 0 { - t.Error("Jitter has to be a hash of uint64") + if offsetSeed1 == 0 { + t.Error("Offset seed has to be a hash of uint64") } // Load the first config. cfg2 := getConfig("ha2") - if err := scrapeManager.setJitterSeed(cfg2.GlobalConfig.ExternalLabels); err != nil { + if err := scrapeManager.setOffsetSeed(cfg2.GlobalConfig.ExternalLabels); err != nil { t.Error(err) } - jitter2 := scrapeManager.jitterSeed + offsetSeed2 := scrapeManager.offsetSeed - if jitter1 == jitter2 { - t.Error("Jitter should not be the same on different set of external labels") + if offsetSeed1 == offsetSeed2 { + t.Error("Offset seed should not be the same on different set of external labels") } } diff --git a/scrape/scrape.go b/scrape/scrape.go index a97cbf539f..8c4cc51e74 100644 --- a/scrape/scrape.go +++ b/scrape/scrape.go @@ -279,7 +279,7 @@ const maxAheadTime = 10 * time.Minute // returning an empty label set is interpreted as "drop" type labelsMutator func(labels.Labels) labels.Labels -func newScrapePool(cfg *config.ScrapeConfig, app storage.Appendable, jitterSeed uint64, logger log.Logger, options *Options) (*scrapePool, error) { +func newScrapePool(cfg *config.ScrapeConfig, app storage.Appendable, offsetSeed uint64, logger log.Logger, options *Options) (*scrapePool, error) { targetScrapePools.Inc() if logger == nil { logger = log.NewNopLogger() @@ -325,7 +325,7 @@ func newScrapePool(cfg *config.ScrapeConfig, app storage.Appendable, jitterSeed func(l labels.Labels) labels.Labels { return mutateReportSampleLabels(l, opts.target) }, func(ctx context.Context) storage.Appender { return app.Appender(ctx) }, cache, - jitterSeed, + offsetSeed, opts.honorTimestamps, opts.sampleLimit, opts.bucketLimit, @@ -775,7 +775,7 @@ func appender(app storage.Appender, sampleLimit, bucketLimit int) storage.Append type scraper interface { scrape(ctx context.Context, w io.Writer) (string, error) Report(start time.Time, dur time.Duration, err error) - offset(interval time.Duration, jitterSeed uint64) time.Duration + offset(interval time.Duration, offsetSeed uint64) time.Duration } // targetScraper implements the scraper interface for a target. @@ -891,7 +891,7 @@ type scrapeLoop struct { cache *scrapeCache lastScrapeSize int buffers *pool.Pool - jitterSeed uint64 + offsetSeed uint64 honorTimestamps bool forcedErr error forcedErrMtx sync.Mutex @@ -1175,7 +1175,7 @@ func newScrapeLoop(ctx context.Context, reportSampleMutator labelsMutator, appender func(ctx context.Context) storage.Appender, cache *scrapeCache, - jitterSeed uint64, + offsetSeed uint64, honorTimestamps bool, sampleLimit int, bucketLimit int, @@ -1217,7 +1217,7 @@ func newScrapeLoop(ctx context.Context, sampleMutator: sampleMutator, reportSampleMutator: reportSampleMutator, stopped: make(chan struct{}), - jitterSeed: jitterSeed, + offsetSeed: offsetSeed, l: l, parentCtx: ctx, appenderCtx: appenderCtx, @@ -1238,7 +1238,7 @@ func newScrapeLoop(ctx context.Context, func (sl *scrapeLoop) run(errc chan<- error) { select { - case <-time.After(sl.scraper.offset(sl.interval, sl.jitterSeed)): + case <-time.After(sl.scraper.offset(sl.interval, sl.offsetSeed)): // Continue after a scraping offset. case <-sl.ctx.Done(): close(sl.stopped) diff --git a/scrape/target_test.go b/scrape/target_test.go index 12d3b5a4d7..4f0c840cd0 100644 --- a/scrape/target_test.go +++ b/scrape/target_test.go @@ -59,7 +59,7 @@ func TestTargetLabels(t *testing.T) { func TestTargetOffset(t *testing.T) { interval := 10 * time.Second - jitter := uint64(0) + offsetSeed := uint64(0) offsets := make([]time.Duration, 10000) @@ -68,7 +68,7 @@ func TestTargetOffset(t *testing.T) { target := newTestTarget("example.com:80", 0, labels.FromStrings( "label", fmt.Sprintf("%d", i), )) - offsets[i] = target.offset(interval, jitter) + offsets[i] = target.offset(interval, offsetSeed) } // Put the offsets into buckets and validate that they are all From ce236c61ab2e0cb6a9d5ef6a2e2ad720746861a0 Mon Sep 17 00:00:00 2001 From: Julien Pivotto Date: Thu, 25 May 2023 13:47:34 +0200 Subject: [PATCH 175/231] Mark 2.45 as LTS As the 2.37 LTS is going EOL in July 2023, let's mark 2.45 as LTS. I have synced with Jesus about this. He will bootstrap the release and after a few week I will do the maintenance for the lifetime of the LTS. Signed-off-by: Julien Pivotto --- RELEASE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RELEASE.md b/RELEASE.md index d7f24dabd5..f5c907fe99 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -49,7 +49,7 @@ Release cadence of first pre-releases being cut is 6 weeks. | v2.42 | 2023-01-25 | Kemal Akkoyun (GitHub: @kakkoyun) | | v2.43 | 2023-03-08 | Julien Pivotto (GitHub: @roidelapluie) | | v2.44 | 2023-04-19 | Bryan Boreham (GitHub: @bboreham) | -| v2.45 | 2023-05-31 | Jesus Vazquez (Github: @jesusvazquez) | +| v2.45 LTS | 2023-05-31 | Jesus Vazquez (Github: @jesusvazquez) | | v2.46 | 2023-07-12 | **searching for volunteer** | If you are interested in volunteering please create a pull request against the [prometheus/prometheus](https://github.com/prometheus/prometheus) repository and propose yourself for the release series of your choice. From 4c4454e4c9e12ad08e765335ba11199f296ad986 Mon Sep 17 00:00:00 2001 From: Justin Lei Date: Thu, 25 May 2023 13:12:32 -0700 Subject: [PATCH 176/231] Group args to append to memSeries in chunkOpts Signed-off-by: Justin Lei --- tsdb/head_append.go | 37 ++++++++++----- tsdb/head_test.go | 113 +++++++++++++++++++++++++++++++------------- tsdb/head_wal.go | 12 +++-- 3 files changed, 113 insertions(+), 49 deletions(-) diff --git a/tsdb/head_append.go b/tsdb/head_append.go index 44847dceb2..a77c8a4ebc 100644 --- a/tsdb/head_append.go +++ b/tsdb/head_append.go @@ -881,9 +881,13 @@ func (a *headAppender) Commit() (err error) { oooMmapMarkers map[chunks.HeadSeriesRef]chunks.ChunkDiskMapperRef oooRecords [][]byte oooCapMax = a.head.opts.OutOfOrderCapMax.Load() - chunkRange = a.head.chunkRange.Load() series *memSeries - enc record.Encoder + appendChunkOpts = chunkOpts{ + chunkDiskMapper: a.head.chunkDiskMapper, + chunkRange: a.head.chunkRange.Load(), + samplesPerChunk: a.head.opts.SamplesPerChunk, + } + enc record.Encoder ) defer func() { for i := range oooRecords { @@ -987,7 +991,7 @@ func (a *headAppender) Commit() (err error) { samplesAppended-- } default: - ok, chunkCreated = series.append(s.T, s.V, a.appendID, a.head.chunkDiskMapper, chunkRange, a.head.opts.SamplesPerChunk) + ok, chunkCreated = series.append(s.T, s.V, a.appendID, appendChunkOpts) if ok { if s.T < inOrderMint { inOrderMint = s.T @@ -1016,7 +1020,7 @@ func (a *headAppender) Commit() (err error) { for i, s := range a.histograms { series = a.histogramSeries[i] series.Lock() - ok, chunkCreated := series.appendHistogram(s.T, s.H, a.appendID, a.head.chunkDiskMapper, chunkRange, a.head.opts.SamplesPerChunk) + ok, chunkCreated := series.appendHistogram(s.T, s.H, a.appendID, appendChunkOpts) series.cleanupAppendIDsBelow(a.cleanupAppendIDsBelow) series.pendingCommit = false series.Unlock() @@ -1042,7 +1046,7 @@ func (a *headAppender) Commit() (err error) { for i, s := range a.floatHistograms { series = a.floatHistogramSeries[i] series.Lock() - ok, chunkCreated := series.appendFloatHistogram(s.T, s.FH, a.appendID, a.head.chunkDiskMapper, chunkRange, a.head.opts.SamplesPerChunk) + ok, chunkCreated := series.appendFloatHistogram(s.T, s.FH, a.appendID, appendChunkOpts) series.cleanupAppendIDsBelow(a.cleanupAppendIDsBelow) series.pendingCommit = false series.Unlock() @@ -1118,12 +1122,19 @@ func (s *memSeries) insert(t int64, v float64, chunkDiskMapper *chunks.ChunkDisk return ok, chunkCreated, mmapRef } +// chunkOpts are chunk-level options that are passed when appending to a memSeries. +type chunkOpts struct { + chunkDiskMapper *chunks.ChunkDiskMapper + chunkRange int64 + samplesPerChunk int +} + // append adds the sample (t, v) to the series. The caller also has to provide // the appendID for isolation. (The appendID can be zero, which results in no // isolation for this append.) // It is unsafe to call this concurrently with s.iterator(...) without holding the series lock. -func (s *memSeries) append(t int64, v float64, appendID uint64, chunkDiskMapper *chunks.ChunkDiskMapper, chunkRange int64, samplesPerChunk int) (sampleInOrder, chunkCreated bool) { - c, sampleInOrder, chunkCreated := s.appendPreprocessor(t, chunkenc.EncXOR, chunkDiskMapper, chunkRange, samplesPerChunk) +func (s *memSeries) append(t int64, v float64, appendID uint64, o chunkOpts) (sampleInOrder, chunkCreated bool) { + c, sampleInOrder, chunkCreated := s.appendPreprocessor(t, chunkenc.EncXOR, o.chunkDiskMapper, o.chunkRange, o.samplesPerChunk) if !sampleInOrder { return sampleInOrder, chunkCreated } @@ -1144,7 +1155,7 @@ func (s *memSeries) append(t int64, v float64, appendID uint64, chunkDiskMapper // appendHistogram adds the histogram. // It is unsafe to call this concurrently with s.iterator(...) without holding the series lock. -func (s *memSeries) appendHistogram(t int64, h *histogram.Histogram, appendID uint64, chunkDiskMapper *chunks.ChunkDiskMapper, chunkRange int64, samplesPerChunk int) (sampleInOrder, chunkCreated bool) { +func (s *memSeries) appendHistogram(t int64, h *histogram.Histogram, appendID uint64, o chunkOpts) (sampleInOrder, chunkCreated bool) { // Head controls the execution of recoding, so that we own the proper // chunk reference afterwards. We check for Appendable from appender before // appendPreprocessor because in case it ends up creating a new chunk, @@ -1157,7 +1168,7 @@ func (s *memSeries) appendHistogram(t int64, h *histogram.Histogram, appendID ui pMergedSpans, nMergedSpans []histogram.Span okToAppend, counterReset, gauge bool ) - c, sampleInOrder, chunkCreated := s.appendPreprocessor(t, chunkenc.EncHistogram, chunkDiskMapper, chunkRange, samplesPerChunk) + c, sampleInOrder, chunkCreated := s.appendPreprocessor(t, chunkenc.EncHistogram, o.chunkDiskMapper, o.chunkRange, o.samplesPerChunk) if !sampleInOrder { return sampleInOrder, chunkCreated } @@ -1193,7 +1204,7 @@ func (s *memSeries) appendHistogram(t int64, h *histogram.Histogram, appendID ui // - okToAppend and no inserts → Chunk is ready to support our histogram. switch { case !okToAppend || counterReset: - c = s.cutNewHeadChunk(t, chunkenc.EncHistogram, chunkDiskMapper, chunkRange) + c = s.cutNewHeadChunk(t, chunkenc.EncHistogram, o.chunkDiskMapper, o.chunkRange) chunkCreated = true case len(pForwardInserts) > 0 || len(nForwardInserts) > 0: // New buckets have appeared. We need to recode all @@ -1238,7 +1249,7 @@ func (s *memSeries) appendHistogram(t int64, h *histogram.Histogram, appendID ui // appendFloatHistogram adds the float histogram. // It is unsafe to call this concurrently with s.iterator(...) without holding the series lock. -func (s *memSeries) appendFloatHistogram(t int64, fh *histogram.FloatHistogram, appendID uint64, chunkDiskMapper *chunks.ChunkDiskMapper, chunkRange int64, samplesPerChunk int) (sampleInOrder, chunkCreated bool) { +func (s *memSeries) appendFloatHistogram(t int64, fh *histogram.FloatHistogram, appendID uint64, o chunkOpts) (sampleInOrder, chunkCreated bool) { // Head controls the execution of recoding, so that we own the proper // chunk reference afterwards. We check for Appendable from appender before // appendPreprocessor because in case it ends up creating a new chunk, @@ -1251,7 +1262,7 @@ func (s *memSeries) appendFloatHistogram(t int64, fh *histogram.FloatHistogram, pMergedSpans, nMergedSpans []histogram.Span okToAppend, counterReset, gauge bool ) - c, sampleInOrder, chunkCreated := s.appendPreprocessor(t, chunkenc.EncFloatHistogram, chunkDiskMapper, chunkRange, samplesPerChunk) + c, sampleInOrder, chunkCreated := s.appendPreprocessor(t, chunkenc.EncFloatHistogram, o.chunkDiskMapper, o.chunkRange, o.samplesPerChunk) if !sampleInOrder { return sampleInOrder, chunkCreated } @@ -1287,7 +1298,7 @@ func (s *memSeries) appendFloatHistogram(t int64, fh *histogram.FloatHistogram, // - okToAppend and no inserts → Chunk is ready to support our histogram. switch { case !okToAppend || counterReset: - c = s.cutNewHeadChunk(t, chunkenc.EncFloatHistogram, chunkDiskMapper, chunkRange) + c = s.cutNewHeadChunk(t, chunkenc.EncFloatHistogram, o.chunkDiskMapper, o.chunkRange) chunkCreated = true case len(pForwardInserts) > 0 || len(nForwardInserts) > 0: // New buckets have appeared. We need to recode all diff --git a/tsdb/head_test.go b/tsdb/head_test.go index 14468e0716..8eb218b5ac 100644 --- a/tsdb/head_test.go +++ b/tsdb/head_test.go @@ -283,10 +283,15 @@ func BenchmarkLoadWAL(b *testing.B) { if c.mmappedChunkT != 0 { chunkDiskMapper, err := chunks.NewChunkDiskMapper(nil, mmappedChunksDir(dir), chunkenc.NewPool(), chunks.DefaultWriteBufferSize, chunks.DefaultWriteQueueSize) require.NoError(b, err) + cOpts := chunkOpts{ + chunkDiskMapper: chunkDiskMapper, + chunkRange: c.mmappedChunkT, + samplesPerChunk: DefaultSamplesPerChunk, + } for k := 0; k < c.batches*c.seriesPerBatch; k++ { // Create one mmapped chunk per series, with one sample at the given time. s := newMemSeries(labels.Labels{}, chunks.HeadSeriesRef(k)*101, defaultIsolationDisabled) - s.append(c.mmappedChunkT, 42, 0, chunkDiskMapper, c.mmappedChunkT, DefaultSamplesPerChunk) + s.append(c.mmappedChunkT, 42, 0, cOpts) s.mmapCurrentHeadChunk(chunkDiskMapper) } require.NoError(b, chunkDiskMapper.Close()) @@ -799,7 +804,11 @@ func TestMemSeries_truncateChunks(t *testing.T) { defer func() { require.NoError(t, chunkDiskMapper.Close()) }() - const chunkRange = 2000 + cOpts := chunkOpts{ + chunkDiskMapper: chunkDiskMapper, + chunkRange: 2000, + samplesPerChunk: DefaultSamplesPerChunk, + } memChunkPool := sync.Pool{ New: func() interface{} { @@ -810,7 +819,7 @@ func TestMemSeries_truncateChunks(t *testing.T) { s := newMemSeries(labels.FromStrings("a", "b"), 1, defaultIsolationDisabled) for i := 0; i < 4000; i += 5 { - ok, _ := s.append(int64(i), float64(i), 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, _ := s.append(int64(i), float64(i), 0, cOpts) require.True(t, ok, "sample append failed") } @@ -1336,26 +1345,30 @@ func TestMemSeries_append(t *testing.T) { defer func() { require.NoError(t, chunkDiskMapper.Close()) }() - const chunkRange = 500 + cOpts := chunkOpts{ + chunkDiskMapper: chunkDiskMapper, + chunkRange: 500, + samplesPerChunk: DefaultSamplesPerChunk, + } s := newMemSeries(labels.Labels{}, 1, defaultIsolationDisabled) // Add first two samples at the very end of a chunk range and the next two // on and after it. // New chunk must correctly be cut at 1000. - ok, chunkCreated := s.append(998, 1, 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, chunkCreated := s.append(998, 1, 0, cOpts) require.True(t, ok, "append failed") require.True(t, chunkCreated, "first sample created chunk") - ok, chunkCreated = s.append(999, 2, 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, chunkCreated = s.append(999, 2, 0, cOpts) require.True(t, ok, "append failed") require.False(t, chunkCreated, "second sample should use same chunk") - ok, chunkCreated = s.append(1000, 3, 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, chunkCreated = s.append(1000, 3, 0, cOpts) require.True(t, ok, "append failed") require.True(t, chunkCreated, "expected new chunk on boundary") - ok, chunkCreated = s.append(1001, 4, 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, chunkCreated = s.append(1001, 4, 0, cOpts) require.True(t, ok, "append failed") require.False(t, chunkCreated, "second sample should use same chunk") @@ -1368,7 +1381,7 @@ func TestMemSeries_append(t *testing.T) { // Fill the range [1000,2000) with many samples. Intermediate chunks should be cut // at approximately 120 samples per chunk. for i := 1; i < 1000; i++ { - ok, _ := s.append(1001+int64(i), float64(i), 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, _ := s.append(1001+int64(i), float64(i), 0, cOpts) require.True(t, ok, "append failed") } @@ -1390,7 +1403,11 @@ func TestMemSeries_appendHistogram(t *testing.T) { defer func() { require.NoError(t, chunkDiskMapper.Close()) }() - chunkRange := int64(1000) + cOpts := chunkOpts{ + chunkDiskMapper: chunkDiskMapper, + chunkRange: int64(1000), + samplesPerChunk: DefaultSamplesPerChunk, + } s := newMemSeries(labels.Labels{}, 1, defaultIsolationDisabled) @@ -1404,19 +1421,19 @@ func TestMemSeries_appendHistogram(t *testing.T) { // Add first two samples at the very end of a chunk range and the next two // on and after it. // New chunk must correctly be cut at 1000. - ok, chunkCreated := s.appendHistogram(998, histograms[0], 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, chunkCreated := s.appendHistogram(998, histograms[0], 0, cOpts) require.True(t, ok, "append failed") require.True(t, chunkCreated, "first sample created chunk") - ok, chunkCreated = s.appendHistogram(999, histograms[1], 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, chunkCreated = s.appendHistogram(999, histograms[1], 0, cOpts) require.True(t, ok, "append failed") require.False(t, chunkCreated, "second sample should use same chunk") - ok, chunkCreated = s.appendHistogram(1000, histograms[2], 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, chunkCreated = s.appendHistogram(1000, histograms[2], 0, cOpts) require.True(t, ok, "append failed") require.True(t, chunkCreated, "expected new chunk on boundary") - ok, chunkCreated = s.appendHistogram(1001, histograms[3], 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, chunkCreated = s.appendHistogram(1001, histograms[3], 0, cOpts) require.True(t, ok, "append failed") require.False(t, chunkCreated, "second sample should use same chunk") @@ -1426,7 +1443,7 @@ func TestMemSeries_appendHistogram(t *testing.T) { require.Equal(t, int64(1000), s.headChunk.minTime, "wrong chunk range") require.Equal(t, int64(1001), s.headChunk.maxTime, "wrong chunk range") - ok, chunkCreated = s.appendHistogram(1002, histogramWithOneMoreBucket, 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, chunkCreated = s.appendHistogram(1002, histogramWithOneMoreBucket, 0, cOpts) require.True(t, ok, "append failed") require.False(t, chunkCreated, "third sample should trigger a re-encoded chunk") @@ -1446,7 +1463,11 @@ func TestMemSeries_append_atVariableRate(t *testing.T) { t.Cleanup(func() { require.NoError(t, chunkDiskMapper.Close()) }) - chunkRange := DefaultBlockDuration + cOpts := chunkOpts{ + chunkDiskMapper: chunkDiskMapper, + chunkRange: DefaultBlockDuration, + samplesPerChunk: samplesPerChunk, + } s := newMemSeries(labels.Labels{}, 1, defaultIsolationDisabled) @@ -1456,7 +1477,7 @@ func TestMemSeries_append_atVariableRate(t *testing.T) { var nextTs int64 var totalAppendedSamples int for i := 0; i < samplesPerChunk/4; i++ { - ok, _ := s.append(nextTs, float64(i), 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, _ := s.append(nextTs, float64(i), 0, cOpts) require.Truef(t, ok, "slow sample %d was not appended", i) nextTs += slowRate totalAppendedSamples++ @@ -1465,12 +1486,12 @@ func TestMemSeries_append_atVariableRate(t *testing.T) { // Suddenly, the rate increases and we receive a sample every millisecond. for i := 0; i < math.MaxUint16; i++ { - ok, _ := s.append(nextTs, float64(i), 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, _ := s.append(nextTs, float64(i), 0, cOpts) require.Truef(t, ok, "quick sample %d was not appended", i) nextTs++ totalAppendedSamples++ } - ok, chunkCreated := s.append(DefaultBlockDuration, float64(0), 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, chunkCreated := s.append(DefaultBlockDuration, float64(0), 0, cOpts) require.True(t, ok, "new chunk sample was not appended") require.True(t, chunkCreated, "sample at block duration timestamp should create a new chunk") @@ -1490,23 +1511,29 @@ func TestGCChunkAccess(t *testing.T) { require.NoError(t, h.Close()) }() + cOpts := chunkOpts{ + chunkDiskMapper: h.chunkDiskMapper, + chunkRange: chunkRange, + samplesPerChunk: DefaultSamplesPerChunk, + } + h.initTime(0) s, _, _ := h.getOrCreate(1, labels.FromStrings("a", "1")) // Appending 2 samples for the first chunk. - ok, chunkCreated := s.append(0, 0, 0, h.chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, chunkCreated := s.append(0, 0, 0, cOpts) require.True(t, ok, "series append failed") require.True(t, chunkCreated, "chunks was not created") - ok, chunkCreated = s.append(999, 999, 0, h.chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, chunkCreated = s.append(999, 999, 0, cOpts) require.True(t, ok, "series append failed") require.False(t, chunkCreated, "chunks was created") // A new chunks should be created here as it's beyond the chunk range. - ok, chunkCreated = s.append(1000, 1000, 0, h.chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, chunkCreated = s.append(1000, 1000, 0, cOpts) require.True(t, ok, "series append failed") require.True(t, chunkCreated, "chunks was not created") - ok, chunkCreated = s.append(1999, 1999, 0, h.chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, chunkCreated = s.append(1999, 1999, 0, cOpts) require.True(t, ok, "series append failed") require.False(t, chunkCreated, "chunks was created") @@ -1543,23 +1570,29 @@ func TestGCSeriesAccess(t *testing.T) { require.NoError(t, h.Close()) }() + cOpts := chunkOpts{ + chunkDiskMapper: h.chunkDiskMapper, + chunkRange: chunkRange, + samplesPerChunk: DefaultSamplesPerChunk, + } + h.initTime(0) s, _, _ := h.getOrCreate(1, labels.FromStrings("a", "1")) // Appending 2 samples for the first chunk. - ok, chunkCreated := s.append(0, 0, 0, h.chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, chunkCreated := s.append(0, 0, 0, cOpts) require.True(t, ok, "series append failed") require.True(t, chunkCreated, "chunks was not created") - ok, chunkCreated = s.append(999, 999, 0, h.chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, chunkCreated = s.append(999, 999, 0, cOpts) require.True(t, ok, "series append failed") require.False(t, chunkCreated, "chunks was created") // A new chunks should be created here as it's beyond the chunk range. - ok, chunkCreated = s.append(1000, 1000, 0, h.chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, chunkCreated = s.append(1000, 1000, 0, cOpts) require.True(t, ok, "series append failed") require.True(t, chunkCreated, "chunks was not created") - ok, chunkCreated = s.append(1999, 1999, 0, h.chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, chunkCreated = s.append(1999, 1999, 0, cOpts) require.True(t, ok, "series append failed") require.False(t, chunkCreated, "chunks was created") @@ -1791,14 +1824,20 @@ func TestHeadReadWriterRepair(t *testing.T) { require.Equal(t, 0.0, prom_testutil.ToFloat64(h.metrics.mmapChunkCorruptionTotal)) require.NoError(t, h.Init(math.MinInt64)) + cOpts := chunkOpts{ + chunkDiskMapper: h.chunkDiskMapper, + chunkRange: chunkRange, + samplesPerChunk: DefaultSamplesPerChunk, + } + s, created, _ := h.getOrCreate(1, labels.FromStrings("a", "1")) require.True(t, created, "series was not created") for i := 0; i < 7; i++ { - ok, chunkCreated := s.append(int64(i*chunkRange), float64(i*chunkRange), 0, h.chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, chunkCreated := s.append(int64(i*chunkRange), float64(i*chunkRange), 0, cOpts) require.True(t, ok, "series append failed") require.True(t, chunkCreated, "chunk was not created") - ok, chunkCreated = s.append(int64(i*chunkRange)+chunkRange-1, float64(i*chunkRange), 0, h.chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, chunkCreated = s.append(int64(i*chunkRange)+chunkRange-1, float64(i*chunkRange), 0, cOpts) require.True(t, ok, "series append failed") require.False(t, chunkCreated, "chunk was created") h.chunkDiskMapper.CutNewFile() @@ -2144,9 +2183,15 @@ func TestIsolationAppendIDZeroIsNoop(t *testing.T) { h.initTime(0) + cOpts := chunkOpts{ + chunkDiskMapper: h.chunkDiskMapper, + chunkRange: h.chunkRange.Load(), + samplesPerChunk: DefaultSamplesPerChunk, + } + s, _, _ := h.getOrCreate(1, labels.FromStrings("a", "1")) - ok, _ := s.append(0, 0, 0, h.chunkDiskMapper, h.chunkRange.Load(), DefaultSamplesPerChunk) + ok, _ := s.append(0, 0, 0, cOpts) require.True(t, ok, "Series append failed.") require.Equal(t, 0, s.txs.txIDCount, "Series should not have an appendID after append with appendID=0.") } @@ -2608,12 +2653,16 @@ func TestIteratorSeekIntoBuffer(t *testing.T) { defer func() { require.NoError(t, chunkDiskMapper.Close()) }() - const chunkRange = 500 + cOpts := chunkOpts{ + chunkDiskMapper: chunkDiskMapper, + chunkRange: 500, + samplesPerChunk: DefaultSamplesPerChunk, + } s := newMemSeries(labels.Labels{}, 1, defaultIsolationDisabled) for i := 0; i < 7; i++ { - ok, _ := s.append(int64(i), float64(i), 0, chunkDiskMapper, chunkRange, DefaultSamplesPerChunk) + ok, _ := s.append(int64(i), float64(i), 0, cOpts) require.True(t, ok, "sample append failed") } diff --git a/tsdb/head_wal.go b/tsdb/head_wal.go index 9741d1da04..71120c55e1 100644 --- a/tsdb/head_wal.go +++ b/tsdb/head_wal.go @@ -564,7 +564,11 @@ func (wp *walSubsetProcessor) processWALSamples(h *Head, mmappedChunks, oooMmapp minValidTime := h.minValidTime.Load() mint, maxt := int64(math.MaxInt64), int64(math.MinInt64) - chunkRange := h.chunkRange.Load() + appendChunkOpts := chunkOpts{ + chunkDiskMapper: h.chunkDiskMapper, + chunkRange: h.chunkRange.Load(), + samplesPerChunk: h.opts.SamplesPerChunk, + } for in := range wp.input { if in.existingSeries != nil { @@ -588,7 +592,7 @@ func (wp *walSubsetProcessor) processWALSamples(h *Head, mmappedChunks, oooMmapp if s.T <= ms.mmMaxTime { continue } - if _, chunkCreated := ms.append(s.T, s.V, 0, h.chunkDiskMapper, chunkRange, h.opts.SamplesPerChunk); chunkCreated { + if _, chunkCreated := ms.append(s.T, s.V, 0, appendChunkOpts); chunkCreated { h.metrics.chunksCreated.Inc() h.metrics.chunks.Inc() } @@ -618,9 +622,9 @@ func (wp *walSubsetProcessor) processWALSamples(h *Head, mmappedChunks, oooMmapp } var chunkCreated bool if s.h != nil { - _, chunkCreated = ms.appendHistogram(s.t, s.h, 0, h.chunkDiskMapper, chunkRange, h.opts.SamplesPerChunk) + _, chunkCreated = ms.appendHistogram(s.t, s.h, 0, appendChunkOpts) } else { - _, chunkCreated = ms.appendFloatHistogram(s.t, s.fh, 0, h.chunkDiskMapper, chunkRange, h.opts.SamplesPerChunk) + _, chunkCreated = ms.appendFloatHistogram(s.t, s.fh, 0, appendChunkOpts) } if chunkCreated { h.metrics.chunksCreated.Inc() From e73d8b208408cf5566541a68c610776935f8789d Mon Sep 17 00:00:00 2001 From: Justin Lei Date: Thu, 25 May 2023 13:35:09 -0700 Subject: [PATCH 177/231] Also pass chunkOpts into appendPreprocessor Signed-off-by: Justin Lei --- tsdb/head_append.go | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/tsdb/head_append.go b/tsdb/head_append.go index a77c8a4ebc..3d828d0667 100644 --- a/tsdb/head_append.go +++ b/tsdb/head_append.go @@ -1134,7 +1134,7 @@ type chunkOpts struct { // isolation for this append.) // It is unsafe to call this concurrently with s.iterator(...) without holding the series lock. func (s *memSeries) append(t int64, v float64, appendID uint64, o chunkOpts) (sampleInOrder, chunkCreated bool) { - c, sampleInOrder, chunkCreated := s.appendPreprocessor(t, chunkenc.EncXOR, o.chunkDiskMapper, o.chunkRange, o.samplesPerChunk) + c, sampleInOrder, chunkCreated := s.appendPreprocessor(t, chunkenc.EncXOR, o) if !sampleInOrder { return sampleInOrder, chunkCreated } @@ -1168,7 +1168,7 @@ func (s *memSeries) appendHistogram(t int64, h *histogram.Histogram, appendID ui pMergedSpans, nMergedSpans []histogram.Span okToAppend, counterReset, gauge bool ) - c, sampleInOrder, chunkCreated := s.appendPreprocessor(t, chunkenc.EncHistogram, o.chunkDiskMapper, o.chunkRange, o.samplesPerChunk) + c, sampleInOrder, chunkCreated := s.appendPreprocessor(t, chunkenc.EncHistogram, o) if !sampleInOrder { return sampleInOrder, chunkCreated } @@ -1262,7 +1262,7 @@ func (s *memSeries) appendFloatHistogram(t int64, fh *histogram.FloatHistogram, pMergedSpans, nMergedSpans []histogram.Span okToAppend, counterReset, gauge bool ) - c, sampleInOrder, chunkCreated := s.appendPreprocessor(t, chunkenc.EncFloatHistogram, o.chunkDiskMapper, o.chunkRange, o.samplesPerChunk) + c, sampleInOrder, chunkCreated := s.appendPreprocessor(t, chunkenc.EncFloatHistogram, o) if !sampleInOrder { return sampleInOrder, chunkCreated } @@ -1344,9 +1344,7 @@ func (s *memSeries) appendFloatHistogram(t int64, fh *histogram.FloatHistogram, // appendPreprocessor takes care of cutting new chunks and m-mapping old chunks. // It is unsafe to call this concurrently with s.iterator(...) without holding the series lock. // This should be called only when appending data. -func (s *memSeries) appendPreprocessor( - t int64, e chunkenc.Encoding, chunkDiskMapper *chunks.ChunkDiskMapper, chunkRange int64, samplesPerChunk int, -) (c *memChunk, sampleInOrder, chunkCreated bool) { +func (s *memSeries) appendPreprocessor(t int64, e chunkenc.Encoding, o chunkOpts) (c *memChunk, sampleInOrder, chunkCreated bool) { c = s.head() if c == nil { @@ -1355,7 +1353,7 @@ func (s *memSeries) appendPreprocessor( return c, false, false } // There is no head chunk in this series yet, create the first chunk for the sample. - c = s.cutNewHeadChunk(t, e, chunkDiskMapper, chunkRange) + c = s.cutNewHeadChunk(t, e, o.chunkDiskMapper, o.chunkRange) chunkCreated = true } @@ -1367,7 +1365,7 @@ func (s *memSeries) appendPreprocessor( if c.chunk.Encoding() != e { // The chunk encoding expected by this append is different than the head chunk's // encoding. So we cut a new chunk with the expected encoding. - c = s.cutNewHeadChunk(t, e, chunkDiskMapper, chunkRange) + c = s.cutNewHeadChunk(t, e, o.chunkDiskMapper, o.chunkRange) chunkCreated = true } @@ -1376,14 +1374,14 @@ func (s *memSeries) appendPreprocessor( // It could be the new chunk created after reading the chunk snapshot, // hence we fix the minTime of the chunk here. c.minTime = t - s.nextAt = rangeForTimestamp(c.minTime, chunkRange) + s.nextAt = rangeForTimestamp(c.minTime, o.chunkRange) } // If we reach 25% of a chunk's desired sample count, predict an end time // for this chunk that will try to make samples equally distributed within // the remaining chunks in the current chunk range. // At latest it must happen at the timestamp set when the chunk was cut. - if numSamples == samplesPerChunk/4 { + if numSamples == o.samplesPerChunk/4 { s.nextAt = computeChunkEndTime(c.minTime, c.maxTime, s.nextAt) } // If numSamples > samplesPerChunk*2 then our previous prediction was invalid, @@ -1391,8 +1389,8 @@ func (s *memSeries) appendPreprocessor( // Since we assume that the rate is higher, we're being conservative and cutting at 2*samplesPerChunk // as we expect more chunks to come. // Note that next chunk will have its nextAt recalculated for the new rate. - if t >= s.nextAt || numSamples >= samplesPerChunk*2 { - c = s.cutNewHeadChunk(t, e, chunkDiskMapper, chunkRange) + if t >= s.nextAt || numSamples >= o.samplesPerChunk*2 { + c = s.cutNewHeadChunk(t, e, o.chunkDiskMapper, o.chunkRange) chunkCreated = true } From a308ea773d124de39958e0b50282115c341dc247 Mon Sep 17 00:00:00 2001 From: marcoderama Date: Fri, 26 May 2023 16:39:55 -0700 Subject: [PATCH 178/231] Update functions.md Fix small typo Signed-off-by: marcoderama --- docs/querying/functions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/querying/functions.md b/docs/querying/functions.md index c8831c0792..e1a0b4a769 100644 --- a/docs/querying/functions.md +++ b/docs/querying/functions.md @@ -359,7 +359,7 @@ increase(http_requests_total{job="api-server"}[5m]) ``` `increase` acts on native histograms by calculating a new histogram where each -compononent (sum and count of observations, buckets) is the increase between +component (sum and count of observations, buckets) is the increase between the respective component in the first and last native histogram in `v`. However, each element in `v` that contains a mix of float and native histogram samples within the range, will be missing from the result vector. From 4c27fa0c1d718894a1a584f75f18a881950c93a3 Mon Sep 17 00:00:00 2001 From: Julien Pivotto Date: Sun, 28 May 2023 15:00:48 +0200 Subject: [PATCH 179/231] Update OpenTelemetry dependencies Signed-off-by: Julien Pivotto --- go.mod | 42 ++++++++++++++++-------------- go.sum | 82 +++++++++++++++++++++++++++++++--------------------------- 2 files changed, 66 insertions(+), 58 deletions(-) diff --git a/go.mod b/go.mod index c902854388..998c173cda 100644 --- a/go.mod +++ b/go.mod @@ -49,27 +49,27 @@ require ( github.com/prometheus/exporter-toolkit v0.9.1 github.com/scaleway/scaleway-sdk-go v1.0.0-beta.15 github.com/shurcooL/httpfs v0.0.0-20190707220628-8d4bc4ba7749 - github.com/stretchr/testify v1.8.2 + github.com/stretchr/testify v1.8.3 github.com/vultr/govultr/v2 v2.17.2 - go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.40.0 - go.opentelemetry.io/otel v1.14.0 - go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.14.0 - go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.14.0 - go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.14.0 - go.opentelemetry.io/otel/sdk v1.14.0 - go.opentelemetry.io/otel/trace v1.14.0 + go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.42.0 + go.opentelemetry.io/otel v1.16.0 + go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.16.0 + go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.16.0 + go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.16.0 + go.opentelemetry.io/otel/sdk v1.16.0 + go.opentelemetry.io/otel/trace v1.16.0 go.uber.org/atomic v1.10.0 go.uber.org/automaxprocs v1.5.2 go.uber.org/goleak v1.2.1 - golang.org/x/net v0.9.0 + golang.org/x/net v0.10.0 golang.org/x/oauth2 v0.7.0 golang.org/x/sync v0.1.0 - golang.org/x/sys v0.7.0 + golang.org/x/sys v0.8.0 golang.org/x/time v0.3.0 golang.org/x/tools v0.8.0 google.golang.org/api v0.114.0 - google.golang.org/genproto v0.0.0-20230306155012-7f2fa6fef1f4 - google.golang.org/grpc v1.53.0 + google.golang.org/genproto/googleapis/api v0.0.0-20230526203410-71b5a4ffd15e + google.golang.org/grpc v1.55.0 google.golang.org/protobuf v1.30.0 gopkg.in/yaml.v2 v2.4.0 gopkg.in/yaml.v3 v3.0.1 @@ -87,10 +87,12 @@ require ( github.com/hashicorp/go-multierror v1.1.1 // indirect github.com/rogpeppe/go-internal v1.10.0 // indirect github.com/xhit/go-str2duration/v2 v2.1.0 // indirect + google.golang.org/genproto v0.0.0-20230526203410-71b5a4ffd15e // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20230526203410-71b5a4ffd15e // indirect ) require ( - cloud.google.com/go/compute v1.18.0 // indirect + cloud.google.com/go/compute v1.19.0 // indirect github.com/Azure/go-autorest v14.2.0+incompatible // indirect github.com/Azure/go-autorest/autorest/date v0.3.0 // indirect github.com/Azure/go-autorest/autorest/to v0.4.0 // indirect @@ -101,8 +103,8 @@ require ( github.com/armon/go-metrics v0.4.1 // indirect github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect github.com/beorn7/perks v1.0.1 // indirect - github.com/cenkalti/backoff/v4 v4.2.0 // indirect - github.com/cncf/xds/go v0.0.0-20230112175826-46e39c7b9b43 // indirect + github.com/cenkalti/backoff/v4 v4.2.1 // indirect + github.com/cncf/xds/go v0.0.0-20230310173818-32f1caf87195 // indirect github.com/davecgh/go-spew v1.1.1 // indirect github.com/docker/distribution v2.8.1+incompatible // indirect github.com/docker/go-connections v0.4.0 // indirect @@ -113,7 +115,7 @@ require ( github.com/felixge/httpsnoop v1.0.3 // indirect github.com/ghodss/yaml v1.0.0 // indirect github.com/go-kit/kit v0.12.0 // indirect - github.com/go-logr/logr v1.2.3 // indirect + github.com/go-logr/logr v1.2.4 // indirect github.com/go-logr/stdr v1.2.2 // indirect github.com/go-openapi/analysis v0.21.4 // indirect github.com/go-openapi/errors v0.20.3 // indirect @@ -125,7 +127,7 @@ require ( github.com/go-openapi/validate v0.22.1 // indirect github.com/go-resty/resty/v2 v2.7.0 // indirect github.com/golang-jwt/jwt/v4 v4.5.0 // indirect - github.com/golang/glog v1.0.0 // indirect + github.com/golang/glog v1.1.0 // indirect github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect github.com/golang/protobuf v1.5.3 // indirect github.com/google/gnostic v0.6.9 // indirect @@ -168,13 +170,13 @@ require ( github.com/spf13/pflag v1.0.5 // indirect go.mongodb.org/mongo-driver v1.11.3 // indirect go.opencensus.io v0.24.0 // indirect - go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.14.0 // indirect - go.opentelemetry.io/otel/metric v0.37.0 // indirect + go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.16.0 // indirect + go.opentelemetry.io/otel/metric v1.16.0 // indirect go.opentelemetry.io/proto/otlp v0.19.0 // indirect golang.org/x/crypto v0.7.0 // indirect golang.org/x/exp v0.0.0-20230321023759-10a507213a29 golang.org/x/mod v0.10.0 // indirect - golang.org/x/term v0.7.0 // indirect + golang.org/x/term v0.8.0 // indirect golang.org/x/text v0.9.0 // indirect google.golang.org/appengine v1.6.7 // indirect gopkg.in/inf.v0 v0.9.1 // indirect diff --git a/go.sum b/go.sum index 9ecc5c7275..545c85a9e1 100644 --- a/go.sum +++ b/go.sum @@ -19,8 +19,8 @@ cloud.google.com/go/bigquery v1.4.0/go.mod h1:S8dzgnTigyfTmLBfrtrhyYhwRxG72rYxvf cloud.google.com/go/bigquery v1.5.0/go.mod h1:snEHRnqQbz117VIFhE8bmtwIDY80NLUZUMb4Nv6dBIg= cloud.google.com/go/bigquery v1.7.0/go.mod h1://okPTzCYNXSlb24MZs83e2Do+h+VXtc4gLoIoXIAPc= cloud.google.com/go/bigquery v1.8.0/go.mod h1:J5hqkt3O0uAFnINi6JXValWIb1v0goeZM77hZzJN/fQ= -cloud.google.com/go/compute v1.18.0 h1:FEigFqoDbys2cvFkZ9Fjq4gnHBP55anJ0yQyau2f9oY= -cloud.google.com/go/compute v1.18.0/go.mod h1:1X7yHxec2Ga+Ss6jPyjxRxpu2uu7PLgsOVXvgU0yacs= +cloud.google.com/go/compute v1.19.0 h1:+9zda3WGgW1ZSTlVppLCYFIr48Pa35q1uG2N1itbCEQ= +cloud.google.com/go/compute v1.19.0/go.mod h1:rikpw2y+UMidAe9tISo04EHNOIf42RLYF/q8Bs93scU= cloud.google.com/go/compute/metadata v0.2.3 h1:mg4jlk7mCAj6xXp9UJ4fjI9VUI5rubuGBW5aJ7UnBMY= cloud.google.com/go/compute/metadata v0.2.3/go.mod h1:VAV5nSsACxMJvgaAuX6Pk2AawlZn8kiOGuCv6gTkwuA= cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE= @@ -109,8 +109,8 @@ github.com/bgentry/speakeasy v0.1.0/go.mod h1:+zsyZBPWlz7T6j88CTgSN5bM796AkVf0kB github.com/buger/jsonparser v1.1.1/go.mod h1:6RYKKt7H4d4+iWqouImQ9R2FZql3VbhNgx27UK13J/0= github.com/casbin/casbin/v2 v2.1.2/go.mod h1:YcPU1XXisHhLzuxH9coDNf2FbKpjGlbCg3n9yuLkIJQ= github.com/cenkalti/backoff v2.2.1+incompatible/go.mod h1:90ReRw6GdpyfrHakVjL/QHaoyV4aDUVVkXQJJJ3NXXM= -github.com/cenkalti/backoff/v4 v4.2.0 h1:HN5dHm3WBOgndBH6E8V0q2jIYIR3s9yglV8k/+MN3u4= -github.com/cenkalti/backoff/v4 v4.2.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE= +github.com/cenkalti/backoff/v4 v4.2.1 h1:y4OZtCnogmCPw98Zjyt5a6+QwPLGkiQsYW5oUqylYbM= +github.com/cenkalti/backoff/v4 v4.2.1/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE= github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc= github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= @@ -130,8 +130,8 @@ github.com/cncf/xds/go v0.0.0-20210312221358-fbca930ec8ed/go.mod h1:eXthEFrGJvWH github.com/cncf/xds/go v0.0.0-20210805033703-aa0b78936158/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= github.com/cncf/xds/go v0.0.0-20210922020428-25de7278fc84/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= github.com/cncf/xds/go v0.0.0-20211011173535-cb28da3451f1/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= -github.com/cncf/xds/go v0.0.0-20230112175826-46e39c7b9b43 h1:XP+uhjN0yBCN/tPkr8Z0BNDc5rZam9RG6UWyf2FrSQ0= -github.com/cncf/xds/go v0.0.0-20230112175826-46e39c7b9b43/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= +github.com/cncf/xds/go v0.0.0-20230310173818-32f1caf87195 h1:58f1tJ1ra+zFINPlwLWvQsR9CzAKt2e+EWV2yX9oXQ4= +github.com/cncf/xds/go v0.0.0-20230310173818-32f1caf87195/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= github.com/cockroachdb/datadriven v0.0.0-20190809214429-80d97fb3cbaa/go.mod h1:zn76sxSg3SzpJ0PPJaLDCu+Bu0Lg3sKTORVIj19EIF8= github.com/codahale/hdrhistogram v0.0.0-20161010025455-3a0bb77429bd/go.mod h1:sE/e/2PUdi/liOCUjSTXgM1o87ZssimdTWN964YiIeI= github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= @@ -216,8 +216,8 @@ github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG github.com/go-logfmt/logfmt v0.6.0 h1:wGYYu3uicYdqXVgoYbvnkrPVXkuLM1p1ifugDMEdRi4= github.com/go-logfmt/logfmt v0.6.0/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs= github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= -github.com/go-logr/logr v1.2.3 h1:2DntVwHkVopvECVRSlL5PSo9eG+cAkDCuckLubN+rq0= -github.com/go-logr/logr v1.2.3/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= +github.com/go-logr/logr v1.2.4 h1:g01GSCwiDw2xSZfjJ2/T9M+S6pFdcNtFYsp+Y43HYDQ= +github.com/go-logr/logr v1.2.4/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= github.com/go-openapi/analysis v0.21.2/go.mod h1:HZwRk4RRisyG8vx2Oe6aqeSQcoxRp47Xkp3+K6q+LdY= @@ -297,8 +297,9 @@ github.com/golang-jwt/jwt/v4 v4.2.0/go.mod h1:/xlHOz8bRuivTWchD4jCa+NbatV+wEUSzw github.com/golang-jwt/jwt/v4 v4.5.0 h1:7cYmW1XlMY7h7ii7UhUyChSgS5wUJEnm9uZVTGqOWzg= github.com/golang-jwt/jwt/v4 v4.5.0/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= -github.com/golang/glog v1.0.0 h1:nfP3RFugxnNRyKgeWd4oI1nYvXpxrx8ck8ZrcizshdQ= github.com/golang/glog v1.0.0/go.mod h1:EWib/APOK0SL3dFbYqvxE3UYd8E6s1ouQ7iEp/0LWV4= +github.com/golang/glog v1.1.0 h1:/d3pCKDPWNnvIWe0vVUpNP32qc8U3PDVxySP/y360qE= +github.com/golang/glog v1.1.0/go.mod h1:pfYeQZ3JWZoXTV5sFc986z3HTpwQs9At6P4ImfuP3NQ= github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= @@ -737,8 +738,9 @@ github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/ github.com/stretchr/testify v1.7.2/go.mod h1:R6va5+xMeoiuVRoj+gSkQ7d3FALtqAAGI1FQKckRals= github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= -github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ8= github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= +github.com/stretchr/testify v1.8.3 h1:RP3t2pwF7cMEbC1dqtB6poj3niw/9gnV4Cjg5oW5gtY= +github.com/stretchr/testify v1.8.3/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= github.com/tidwall/pretty v1.0.0 h1:HsD+QiTn7sK6flMKIvNmpqz1qrpP3Ps6jOKIKMooyg4= github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk= github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= @@ -780,24 +782,24 @@ go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= go.opencensus.io v0.22.4/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= go.opencensus.io v0.24.0 h1:y73uSU6J157QMP2kn2r30vwW1A2W2WFwSCGnAVxeaD0= go.opencensus.io v0.24.0/go.mod h1:vNK8G9p7aAivkbmorf4v+7Hgx+Zs0yY+0fOtgBfjQKo= -go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.40.0 h1:lE9EJyw3/JhrjWH/hEy9FptnalDQgj7vpbgC2KCCCxE= -go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.40.0/go.mod h1:pcQ3MM3SWvrA71U4GDqv9UFDJ3HQsW7y5ZO3tDTlUdI= -go.opentelemetry.io/otel v1.14.0 h1:/79Huy8wbf5DnIPhemGB+zEPVwnN6fuQybr/SRXa6hM= -go.opentelemetry.io/otel v1.14.0/go.mod h1:o4buv+dJzx8rohcUeRmWUZhqupFvzWis188WlggnNeU= -go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.14.0 h1:/fXHZHGvro6MVqV34fJzDhi7sHGpX3Ej/Qjmfn003ho= -go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.14.0/go.mod h1:UFG7EBMRdXyFstOwH028U0sVf+AvukSGhF0g8+dmNG8= -go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.14.0 h1:TKf2uAs2ueguzLaxOCBXNpHxfO/aC7PAdDsSH0IbeRQ= -go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.14.0/go.mod h1:HrbCVv40OOLTABmOn1ZWty6CHXkU8DK/Urc43tHug70= -go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.14.0 h1:ap+y8RXX3Mu9apKVtOkM6WSFESLM8K3wNQyOU8sWHcc= -go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.14.0/go.mod h1:5w41DY6S9gZrbjuq6Y+753e96WfPha5IcsOSZTtullM= -go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.14.0 h1:3jAYbRHQAqzLjd9I4tzxwJ8Pk/N6AqBcF6m1ZHrxG94= -go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.14.0/go.mod h1:+N7zNjIJv4K+DeX67XXET0P+eIciESgaFDBqh+ZJFS4= -go.opentelemetry.io/otel/metric v0.37.0 h1:pHDQuLQOZwYD+Km0eb657A25NaRzy0a+eLyKfDXedEs= -go.opentelemetry.io/otel/metric v0.37.0/go.mod h1:DmdaHfGt54iV6UKxsV9slj2bBRJcKC1B1uvDLIioc1s= -go.opentelemetry.io/otel/sdk v1.14.0 h1:PDCppFRDq8A1jL9v6KMI6dYesaq+DFcDZvjsoGvxGzY= -go.opentelemetry.io/otel/sdk v1.14.0/go.mod h1:bwIC5TjrNG6QDCHNWvW4HLHtUQ4I+VQDsnjhvyZCALM= -go.opentelemetry.io/otel/trace v1.14.0 h1:wp2Mmvj41tDsyAJXiWDWpfNsOiIyd38fy85pyKcFq/M= -go.opentelemetry.io/otel/trace v1.14.0/go.mod h1:8avnQLK+CG77yNLUae4ea2JDQ6iT+gozhnZjy/rw9G8= +go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.42.0 h1:pginetY7+onl4qN1vl0xW/V/v6OBZ0vVdH+esuJgvmM= +go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.42.0/go.mod h1:XiYsayHc36K3EByOO6nbAXnAWbrUxdjUROCEeeROOH8= +go.opentelemetry.io/otel v1.16.0 h1:Z7GVAX/UkAXPKsy94IU+i6thsQS4nb7LviLpnaNeW8s= +go.opentelemetry.io/otel v1.16.0/go.mod h1:vl0h9NUa1D5s1nv3A5vZOYWn8av4K8Ml6JDeHrT/bx4= +go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.16.0 h1:t4ZwRPU+emrcvM2e9DHd0Fsf0JTPVcbfa/BhTDF03d0= +go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.16.0/go.mod h1:vLarbg68dH2Wa77g71zmKQqlQ8+8Rq3GRG31uc0WcWI= +go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.16.0 h1:cbsD4cUcviQGXdw8+bo5x2wazq10SKz8hEbtCRPcU78= +go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.16.0/go.mod h1:JgXSGah17croqhJfhByOLVY719k1emAXC8MVhCIJlRs= +go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.16.0 h1:TVQp/bboR4mhZSav+MdgXB8FaRho1RC8UwVn3T0vjVc= +go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.16.0/go.mod h1:I33vtIe0sR96wfrUcilIzLoA3mLHhRmz9S9Te0S3gDo= +go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.16.0 h1:iqjq9LAB8aK++sKVcELezzn655JnBNdsDhghU4G/So8= +go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.16.0/go.mod h1:hGXzO5bhhSHZnKvrDaXB82Y9DRFour0Nz/KrBh7reWw= +go.opentelemetry.io/otel/metric v1.16.0 h1:RbrpwVG1Hfv85LgnZ7+txXioPDoh6EdbZHo26Q3hqOo= +go.opentelemetry.io/otel/metric v1.16.0/go.mod h1:QE47cpOmkwipPiefDwo2wDzwJrlfxxNYodqc4xnGCo4= +go.opentelemetry.io/otel/sdk v1.16.0 h1:Z1Ok1YsijYL0CSJpHt4cS3wDDh7p572grzNrBMiMWgE= +go.opentelemetry.io/otel/sdk v1.16.0/go.mod h1:tMsIuKXuuIWPBAOrH+eHtvhTL+SntFtXF9QD68aP6p4= +go.opentelemetry.io/otel/trace v1.16.0 h1:8JRpaObFoW0pxuVPapkgH8UhHQj+bJW8jJsCZEu5MQs= +go.opentelemetry.io/otel/trace v1.16.0/go.mod h1:Yt9vYq1SdNz3xdjZZK7wcXv1qv2pwLkqr2QVwea0ef0= go.opentelemetry.io/proto/otlp v0.7.0/go.mod h1:PqfVotwruBrMGOCsRd/89rSnXhoiJIqeYNgFYFoEGnI= go.opentelemetry.io/proto/otlp v0.19.0 h1:IVN6GR+mhC4s5yfcTbmzHYODqvWAp3ZedA2SJPI1Nnw= go.opentelemetry.io/proto/otlp v0.19.0/go.mod h1:H7XAot3MsfNsj7EXtrA2q5xSNQ10UqI405h3+duxN4U= @@ -914,8 +916,8 @@ golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qx golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.1.0/go.mod h1:Cx3nUiGt4eDBEyega/BKRp+/AlGL8hYe7U9odMt2Cco= golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= -golang.org/x/net v0.9.0 h1:aWJ/m6xSmxWBx+V0XRHTlrYrPG56jKsLdTFmsSsCzOM= -golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns= +golang.org/x/net v0.10.0 h1:X2//UzNDwYmtCLn7To6G58Wr6f5ahEAQgKNzv9Y951M= +golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= @@ -1006,14 +1008,14 @@ golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.7.0 h1:3jlCCIQZPdOYu1h8BkNvLz8Kgwtae2cagcG/VamtZRU= -golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.8.0 h1:EBmGv8NaZBZTWvrbjNoL6HVt+IVy3QDQpJs7VRIw3tU= +golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.1.0/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= -golang.org/x/term v0.7.0 h1:BEvjmm5fURWqcfbSKTdpkDXYBrUS1c0m8agp14W48vQ= -golang.org/x/term v0.7.0/go.mod h1:P32HKFT3hSsZrRxla30E9HqToFYAQPCMs/zFMBUFqPY= +golang.org/x/term v0.8.0 h1:n5xxQn2i3PC0yLAbjTpNT85q/Kgzcr2gIoX9OrJUols= +golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo= golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= @@ -1148,8 +1150,12 @@ google.golang.org/genproto v0.0.0-20200618031413-b414f8b61790/go.mod h1:jDfRM7Fc google.golang.org/genproto v0.0.0-20200825200019-8632dd797987/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= google.golang.org/genproto v0.0.0-20211118181313-81c1377c94b1/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= google.golang.org/genproto v0.0.0-20220107163113-42d7afdf6368/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= -google.golang.org/genproto v0.0.0-20230306155012-7f2fa6fef1f4 h1:DdoeryqhaXp1LtT/emMP1BRJPHHKFi5akj/nbx/zNTA= -google.golang.org/genproto v0.0.0-20230306155012-7f2fa6fef1f4/go.mod h1:NWraEVixdDnqcqQ30jipen1STv2r/n24Wb7twVTGR4s= +google.golang.org/genproto v0.0.0-20230526203410-71b5a4ffd15e h1:Ao9GzfUMPH3zjVfzXG5rlWlk+Q8MXWKwWpwVQE1MXfw= +google.golang.org/genproto v0.0.0-20230526203410-71b5a4ffd15e/go.mod h1:zqTuNwFlFRsw5zIts5VnzLQxSRqh+CGOTVMlYbY0Eyk= +google.golang.org/genproto/googleapis/api v0.0.0-20230526203410-71b5a4ffd15e h1:AZX1ra8YbFMSb7+1pI8S9v4rrgRR7jU1FmuFSSjTVcQ= +google.golang.org/genproto/googleapis/api v0.0.0-20230526203410-71b5a4ffd15e/go.mod h1:vHYtlOoi6TsQ3Uk2yxR7NI5z8uoV+3pZtR4jmHIkRig= +google.golang.org/genproto/googleapis/rpc v0.0.0-20230526203410-71b5a4ffd15e h1:NumxXLPfHSndr3wBBdeKiVHjGVFzi9RX2HwwQke94iY= +google.golang.org/genproto/googleapis/rpc v0.0.0-20230526203410-71b5a4ffd15e/go.mod h1:66JfowdXAEgad5O9NnYcsNPLCPZJD++2L9X0PCMODrA= google.golang.org/grpc v1.17.0/go.mod h1:6QZJwpn2B+Zp71q/5VxRsJ6NXXVCE5NRUHRo+f3cWCs= google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= google.golang.org/grpc v1.20.0/go.mod h1:chYK+tFQF0nDUGJgXMSgLCQk3phJEuONr2DCgLDdAQM= @@ -1171,8 +1177,8 @@ google.golang.org/grpc v1.33.2/go.mod h1:JMHMWHQWaTccqQQlmk3MJZS+GWXOdAesneDmEnv google.golang.org/grpc v1.36.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU= google.golang.org/grpc v1.40.0/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34= google.golang.org/grpc v1.42.0/go.mod h1:k+4IHHFw41K8+bbowsex27ge2rCb65oeWqe4jJ590SU= -google.golang.org/grpc v1.53.0 h1:LAv2ds7cmFV/XTS3XG1NneeENYrXGmorPxsBbptIjNc= -google.golang.org/grpc v1.53.0/go.mod h1:OnIrk0ipVdj4N5d9IUoFUx72/VlD7+jUsHwZgwSMQpw= +google.golang.org/grpc v1.55.0 h1:3Oj82/tFSCeUrRTg/5E/7d/W5A1tj6Ky1ABAuZuv5ag= +google.golang.org/grpc v1.55.0/go.mod h1:iYEXKGkEBhg1PjZQvoYEVPTDkHo1/bjTnfwTeGONTY8= google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= From 4341b98eb2f4dc3d41d5afa9726feef7ea35e3c3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fran=C3=A7ois=20Gouteroux?= Date: Sun, 28 May 2023 19:55:00 +0200 Subject: [PATCH 180/231] fix: apply suggested changes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: François Gouteroux --- cmd/promtool/main.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/cmd/promtool/main.go b/cmd/promtool/main.go index b3d743e441..c55e5be1ef 100644 --- a/cmd/promtool/main.go +++ b/cmd/promtool/main.go @@ -131,7 +131,7 @@ func main() { checkRulesCmd := checkCmd.Command("rules", "Check if the rule files are valid or not.") ruleFiles := checkRulesCmd.Arg( "rule-files", - "The rule files to check, default is read from standard input (STDIN).", + "The rule files to check, default is read from standard input.", ).ExistingFiles() checkRulesLint := checkRulesCmd.Flag( "lint", @@ -690,7 +690,7 @@ func CheckRules(ls lintConfig, files ...string) int { failed := false hasErrors := false - // add empty string to avoid matching filename + // Add empty string to avoid matching filename. if len(files) == 0 { files = append(files, "") } @@ -723,7 +723,7 @@ func checkRules(filename string, lintSettings lintConfig) (int, []error) { var rgs *rulefmt.RuleGroups var errs []error - // if filename is an empty string it is a stdin + // Empty string is stdin input. if filename == "" { data, err := io.ReadAll(os.Stdin) if err != nil { From 9cf1b4a2e635f6a959118fd5a19e3cba7ed54308 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fran=C3=A7ois=20Gouteroux?= Date: Sun, 28 May 2023 22:31:57 +0200 Subject: [PATCH 181/231] fix: update doc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: François Gouteroux --- docs/command-line/promtool.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/command-line/promtool.md b/docs/command-line/promtool.md index 59c46dd79b..6784fb99fd 100644 --- a/docs/command-line/promtool.md +++ b/docs/command-line/promtool.md @@ -181,7 +181,7 @@ Check if the rule files are valid or not. | Argument | Description | | --- | --- | -| rule-files | The rule files to check, default is read from standard input (STDIN). | +| rule-files | The rule files to check, default is read from standard input. | From 6e7ac76981a53f6727da04f3a360494dc6696fb0 Mon Sep 17 00:00:00 2001 From: cui fliter Date: Mon, 29 May 2023 16:26:11 +0800 Subject: [PATCH 182/231] fix problematic link (#12405) Signed-off-by: cui fliter --- web/ui/module/codemirror-promql/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/web/ui/module/codemirror-promql/README.md b/web/ui/module/codemirror-promql/README.md index 441a81315b..627e4fe15a 100644 --- a/web/ui/module/codemirror-promql/README.md +++ b/web/ui/module/codemirror-promql/README.md @@ -1,7 +1,7 @@ CodeMirror-promql ================= -This project provides a mode for [CodeMirror Next](https://codemirror.net/6) that handles syntax highlighting, linting +This project provides a mode for [CodeMirror](https://codemirror.net/6/) that handles syntax highlighting, linting and autocompletion for PromQL ([Prometheus Query Language](https://prometheus.io/docs/introduction/overview/)). ![preview](https://user-images.githubusercontent.com/4548045/95660829-d5e4b680-0b2a-11eb-9ecb-41dca6396273.gif) @@ -15,7 +15,7 @@ npm install --save @prometheus-io/codemirror-promql ``` **Note:** You will have to manually install different packages that are part -of [CodeMirror Next](https://codemirror.net/6), as they are a peer dependency to this package. Here are the different +of [CodeMirror](https://codemirror.net/6/), as they are a peer dependency to this package. Here are the different packages you need to install: * **@codemirror/autocomplete** From 044e004a81157c681ca1c9512d1101bb05c6ff88 Mon Sep 17 00:00:00 2001 From: SuperQ Date: Tue, 30 May 2023 09:08:00 +0200 Subject: [PATCH 183/231] Update exporter-toolkit Adds web config option `client_allowed_sans`. This enables Prometheus to limit the Subject Alternate Name (SAN) allowed to connect. Signed-off-by: SuperQ --- docs/configuration/https.md | 7 +++++++ go.mod | 4 ++-- go.sum | 8 ++++---- 3 files changed, 13 insertions(+), 6 deletions(-) diff --git a/docs/configuration/https.md b/docs/configuration/https.md index d31aca7b4b..bc83e07a38 100644 --- a/docs/configuration/https.md +++ b/docs/configuration/https.md @@ -44,6 +44,13 @@ tls_server_config: # CA certificate for client certificate authentication to the server. [ client_ca_file: ] + # Verify that the client certificate has a Subject Alternate Name (SAN) + # which is an exact match to an entry in this list, else terminate the + # connection. SAN match can be one or multiple of the following: DNS, + # IP, e-mail, or URI address from https://pkg.go.dev/crypto/x509#Certificate. + [ client_allowed_sans: + [ - ] ] + # Minimum TLS version that is acceptable. [ min_version: | default = "TLS12" ] diff --git a/go.mod b/go.mod index 998c173cda..15daecc2cf 100644 --- a/go.mod +++ b/go.mod @@ -46,7 +46,7 @@ require ( github.com/prometheus/common v0.42.0 github.com/prometheus/common/assets v0.2.0 github.com/prometheus/common/sigv4 v0.1.0 - github.com/prometheus/exporter-toolkit v0.9.1 + github.com/prometheus/exporter-toolkit v0.10.0 github.com/scaleway/scaleway-sdk-go v1.0.0-beta.15 github.com/shurcooL/httpfs v0.0.0-20190707220628-8d4bc4ba7749 github.com/stretchr/testify v1.8.3 @@ -173,7 +173,7 @@ require ( go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.16.0 // indirect go.opentelemetry.io/otel/metric v1.16.0 // indirect go.opentelemetry.io/proto/otlp v0.19.0 // indirect - golang.org/x/crypto v0.7.0 // indirect + golang.org/x/crypto v0.8.0 // indirect golang.org/x/exp v0.0.0-20230321023759-10a507213a29 golang.org/x/mod v0.10.0 // indirect golang.org/x/term v0.8.0 // indirect diff --git a/go.sum b/go.sum index 545c85a9e1..18d98dfaeb 100644 --- a/go.sum +++ b/go.sum @@ -671,8 +671,8 @@ github.com/prometheus/common/assets v0.2.0 h1:0P5OrzoHrYBOSM1OigWL3mY8ZvV2N4zIE/ github.com/prometheus/common/assets v0.2.0/go.mod h1:D17UVUE12bHbim7HzwUvtqm6gwBEaDQ0F+hIGbFbccI= github.com/prometheus/common/sigv4 v0.1.0 h1:qoVebwtwwEhS85Czm2dSROY5fTo2PAPEVdDeppTwGX4= github.com/prometheus/common/sigv4 v0.1.0/go.mod h1:2Jkxxk9yYvCkE5G1sQT7GuEXm57JrvHu9k5YwTjsNtI= -github.com/prometheus/exporter-toolkit v0.9.1 h1:cNkC01riqiOS+kh3zdnNwRsbe/Blh0WwK3ij5rPJ9Sw= -github.com/prometheus/exporter-toolkit v0.9.1/go.mod h1:iFlTmFISCix0vyuyBmm0UqOUCTao9+RsAsKJP3YM9ec= +github.com/prometheus/exporter-toolkit v0.10.0 h1:yOAzZTi4M22ZzVxD+fhy1URTuNRj/36uQJJ5S8IPza8= +github.com/prometheus/exporter-toolkit v0.10.0/go.mod h1:+sVFzuvV5JDyw+Ih6p3zFxZNVnKQa3x5qPmDSiPu4ZY= github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk= github.com/prometheus/procfs v0.0.0-20190117184657-bf6a532e95b1/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk= github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA= @@ -832,8 +832,8 @@ golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0 golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.0.0-20220829220503-c86fa9a7ed90/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58= -golang.org/x/crypto v0.7.0 h1:AvwMYaRytfdeVt3u6mLaxYtErKYjxA2OXjJ1HHq6t3A= -golang.org/x/crypto v0.7.0/go.mod h1:pYwdfH91IfpZVANVyUOhSIPZaFoJGxTFbZhFTx+dXZU= +golang.org/x/crypto v0.8.0 h1:pd9TJtTueMTVQXzk8E2XESSMQDj/U7OUu0PqJqPXQjQ= +golang.org/x/crypto v0.8.0/go.mod h1:mRqEX+O9/h5TFCrQhkgjo2yKi0yYA+9ecGkdQoHrywE= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= From a16b876a05faef651f6fd4f3dd71a32a996ba367 Mon Sep 17 00:00:00 2001 From: Arianna Vespri <36129782+vesari@users.noreply.github.com> Date: Tue, 30 May 2023 10:22:23 +0200 Subject: [PATCH 184/231] Add limits to global config (#12126) * Add limits to global config Signed-off-by: Arianna Vespri * Move changes into Validate func Signed-off-by: Arianna Vespri * Make comments consistent wrt 0 meaning no limit Signed-off-by: Arianna Vespri * Document global limits Signed-off-by: Arianna Vespri --------- Signed-off-by: Arianna Vespri --- config/config.go | 61 ++++- config/config_test.go | 399 +++++++++++++++++++++------- config/testdata/conf.good.yml | 11 + docs/configuration/configuration.md | 33 +++ 4 files changed, 394 insertions(+), 110 deletions(-) diff --git a/config/config.go b/config/config.go index d0ba03ab29..9f81bbfd57 100644 --- a/config/config.go +++ b/config/config.go @@ -267,7 +267,7 @@ func (c *Config) GetScrapeConfigs() ([]*ScrapeConfig, error) { for i, scfg := range c.ScrapeConfigs { // We do these checks for library users that would not call Validate in // Unmarshal. - if err := scfg.Validate(c.GlobalConfig.ScrapeInterval, c.GlobalConfig.ScrapeTimeout); err != nil { + if err := scfg.Validate(c.GlobalConfig); err != nil { return nil, err } @@ -294,7 +294,7 @@ func (c *Config) GetScrapeConfigs() ([]*ScrapeConfig, error) { return nil, fileErr(filename, err) } for _, scfg := range cfg.ScrapeConfigs { - if err := scfg.Validate(c.GlobalConfig.ScrapeInterval, c.GlobalConfig.ScrapeTimeout); err != nil { + if err := scfg.Validate(c.GlobalConfig); err != nil { return nil, fileErr(filename, err) } @@ -343,7 +343,7 @@ func (c *Config) UnmarshalYAML(unmarshal func(interface{}) error) error { // Do global overrides and validate unique names. jobNames := map[string]struct{}{} for _, scfg := range c.ScrapeConfigs { - if err := scfg.Validate(c.GlobalConfig.ScrapeInterval, c.GlobalConfig.ScrapeTimeout); err != nil { + if err := scfg.Validate(c.GlobalConfig); err != nil { return err } @@ -390,6 +390,24 @@ type GlobalConfig struct { QueryLogFile string `yaml:"query_log_file,omitempty"` // The labels to add to any timeseries that this Prometheus instance scrapes. ExternalLabels labels.Labels `yaml:"external_labels,omitempty"` + // An uncompressed response body larger than this many bytes will cause the + // scrape to fail. 0 means no limit. + BodySizeLimit units.Base2Bytes `yaml:"body_size_limit,omitempty"` + // More than this many samples post metric-relabeling will cause the scrape to + // fail. 0 means no limit. + SampleLimit uint `yaml:"sample_limit,omitempty"` + // More than this many targets after the target relabeling will cause the + // scrapes to fail. 0 means no limit. + TargetLimit uint `yaml:"target_limit,omitempty"` + // More than this many labels post metric-relabeling will cause the scrape to + // fail. 0 means no limit. + LabelLimit uint `yaml:"label_limit,omitempty"` + // More than this label name length post metric-relabeling will cause the + // scrape to fail. 0 means no limit. + LabelNameLengthLimit uint `yaml:"label_name_length_limit,omitempty"` + // More than this label value length post metric-relabeling will cause the + // scrape to fail. 0 means no limit. + LabelValueLengthLimit uint `yaml:"label_value_length_limit,omitempty"` } // SetDirectory joins any relative file paths with dir. @@ -478,19 +496,19 @@ type ScrapeConfig struct { // scrape to fail. 0 means no limit. BodySizeLimit units.Base2Bytes `yaml:"body_size_limit,omitempty"` // More than this many samples post metric-relabeling will cause the scrape to - // fail. + // fail. 0 means no limit. SampleLimit uint `yaml:"sample_limit,omitempty"` // More than this many targets after the target relabeling will cause the - // scrapes to fail. + // scrapes to fail. 0 means no limit. TargetLimit uint `yaml:"target_limit,omitempty"` // More than this many labels post metric-relabeling will cause the scrape to - // fail. + // fail. 0 means no limit. LabelLimit uint `yaml:"label_limit,omitempty"` // More than this label name length post metric-relabeling will cause the - // scrape to fail. + // scrape to fail. 0 means no limit. LabelNameLengthLimit uint `yaml:"label_name_length_limit,omitempty"` // More than this label value length post metric-relabeling will cause the - // scrape to fail. + // scrape to fail. 0 means no limit. LabelValueLengthLimit uint `yaml:"label_value_length_limit,omitempty"` // More than this many buckets in a native histogram will cause the scrape to // fail. @@ -552,25 +570,44 @@ func (c *ScrapeConfig) UnmarshalYAML(unmarshal func(interface{}) error) error { return nil } -func (c *ScrapeConfig) Validate(defaultInterval, defaultTimeout model.Duration) error { +func (c *ScrapeConfig) Validate(globalConfig GlobalConfig) error { if c == nil { return errors.New("empty or null scrape config section") } // First set the correct scrape interval, then check that the timeout // (inferred or explicit) is not greater than that. if c.ScrapeInterval == 0 { - c.ScrapeInterval = defaultInterval + c.ScrapeInterval = globalConfig.ScrapeInterval } if c.ScrapeTimeout > c.ScrapeInterval { return fmt.Errorf("scrape timeout greater than scrape interval for scrape config with job name %q", c.JobName) } if c.ScrapeTimeout == 0 { - if defaultTimeout > c.ScrapeInterval { + if globalConfig.ScrapeTimeout > c.ScrapeInterval { c.ScrapeTimeout = c.ScrapeInterval } else { - c.ScrapeTimeout = defaultTimeout + c.ScrapeTimeout = globalConfig.ScrapeTimeout } } + if c.BodySizeLimit == 0 { + c.BodySizeLimit = globalConfig.BodySizeLimit + } + if c.SampleLimit == 0 { + c.SampleLimit = globalConfig.SampleLimit + } + if c.TargetLimit == 0 { + c.TargetLimit = globalConfig.TargetLimit + } + if c.LabelLimit == 0 { + c.LabelLimit = globalConfig.LabelLimit + } + if c.LabelNameLengthLimit == 0 { + c.LabelNameLengthLimit = globalConfig.LabelNameLengthLimit + } + if c.LabelValueLengthLimit == 0 { + c.LabelValueLengthLimit = globalConfig.LabelValueLengthLimit + } + return nil } diff --git a/config/config_test.go b/config/config_test.go index bde09dfece..d243d687c4 100644 --- a/config/config_test.go +++ b/config/config_test.go @@ -68,6 +68,15 @@ func mustParseURL(u string) *config.URL { return &config.URL{URL: parsed} } +const ( + globBodySizeLimit = 15 * units.MiB + globSampleLimit = 1500 + globTargetLimit = 30 + globLabelLimit = 30 + globLabelNameLengthLimit = 200 + globLabelValueLengthLimit = 200 +) + var expectedConf = &Config{ GlobalConfig: GlobalConfig{ ScrapeInterval: model.Duration(15 * time.Second), @@ -76,6 +85,13 @@ var expectedConf = &Config{ QueryLogFile: "", ExternalLabels: labels.FromStrings("foo", "bar", "monitor", "codelab"), + + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, }, RuleFiles: []string{ @@ -165,10 +181,16 @@ var expectedConf = &Config{ { JobName: "prometheus", - HonorLabels: true, - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorLabels: true, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -261,11 +283,15 @@ var expectedConf = &Config{ { JobName: "service-x", - HonorTimestamps: true, - ScrapeInterval: model.Duration(50 * time.Second), - ScrapeTimeout: model.Duration(5 * time.Second), - BodySizeLimit: 10 * units.MiB, - SampleLimit: 1000, + HonorTimestamps: true, + ScrapeInterval: model.Duration(50 * time.Second), + ScrapeTimeout: model.Duration(5 * time.Second), + BodySizeLimit: 10 * units.MiB, + SampleLimit: 1000, + TargetLimit: 35, + LabelLimit: 35, + LabelNameLengthLimit: 210, + LabelValueLengthLimit: 210, HTTPClientConfig: config.HTTPClientConfig{ BasicAuth: &config.BasicAuth{ @@ -352,9 +378,15 @@ var expectedConf = &Config{ { JobName: "service-y", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -399,9 +431,15 @@ var expectedConf = &Config{ { JobName: "service-z", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: model.Duration(10 * time.Second), + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: model.Duration(10 * time.Second), + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: "/metrics", Scheme: "http", @@ -424,9 +462,15 @@ var expectedConf = &Config{ { JobName: "service-kubernetes", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -455,9 +499,15 @@ var expectedConf = &Config{ { JobName: "service-kubernetes-namespaces", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -486,9 +536,15 @@ var expectedConf = &Config{ { JobName: "service-kuma", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -506,9 +562,15 @@ var expectedConf = &Config{ { JobName: "service-marathon", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -535,9 +597,15 @@ var expectedConf = &Config{ { JobName: "service-nomad", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -561,9 +629,15 @@ var expectedConf = &Config{ { JobName: "service-ec2", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -594,9 +668,15 @@ var expectedConf = &Config{ { JobName: "service-lightsail", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -617,9 +697,15 @@ var expectedConf = &Config{ { JobName: "service-azure", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -643,9 +729,15 @@ var expectedConf = &Config{ { JobName: "service-nerve", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -662,9 +754,15 @@ var expectedConf = &Config{ { JobName: "0123service-xxx", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -684,9 +782,15 @@ var expectedConf = &Config{ { JobName: "badfederation", - HonorTimestamps: false, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: false, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: "/federate", Scheme: DefaultScrapeConfig.Scheme, @@ -706,9 +810,15 @@ var expectedConf = &Config{ { JobName: "測試", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -728,9 +838,15 @@ var expectedConf = &Config{ { JobName: "httpsd", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -747,9 +863,15 @@ var expectedConf = &Config{ { JobName: "service-triton", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -774,9 +896,15 @@ var expectedConf = &Config{ { JobName: "digitalocean-droplets", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -800,9 +928,15 @@ var expectedConf = &Config{ { JobName: "docker", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -822,9 +956,15 @@ var expectedConf = &Config{ { JobName: "dockerswarm", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -844,9 +984,15 @@ var expectedConf = &Config{ { JobName: "service-openstack", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -870,9 +1016,15 @@ var expectedConf = &Config{ { JobName: "service-puppetdb", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -898,10 +1050,16 @@ var expectedConf = &Config{ }, }, { - JobName: "hetzner", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + JobName: "hetzner", + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -947,9 +1105,15 @@ var expectedConf = &Config{ { JobName: "service-eureka", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -966,9 +1130,16 @@ var expectedConf = &Config{ { JobName: "ovhcloud", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, + HTTPClientConfig: config.DefaultHTTPClientConfig, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -995,9 +1166,16 @@ var expectedConf = &Config{ { JobName: "scaleway", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, + HTTPClientConfig: config.DefaultHTTPClientConfig, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -1030,9 +1208,15 @@ var expectedConf = &Config{ { JobName: "linode-instances", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -1057,9 +1241,16 @@ var expectedConf = &Config{ { JobName: "uyuni", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, + HTTPClientConfig: config.DefaultHTTPClientConfig, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -1076,10 +1267,16 @@ var expectedConf = &Config{ }, }, { - JobName: "ionos", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + JobName: "ionos", + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, @@ -1101,9 +1298,15 @@ var expectedConf = &Config{ { JobName: "vultr", - HonorTimestamps: true, - ScrapeInterval: model.Duration(15 * time.Second), - ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + HonorTimestamps: true, + ScrapeInterval: model.Duration(15 * time.Second), + ScrapeTimeout: DefaultGlobalConfig.ScrapeTimeout, + BodySizeLimit: globBodySizeLimit, + SampleLimit: globSampleLimit, + TargetLimit: globTargetLimit, + LabelLimit: globLabelLimit, + LabelNameLengthLimit: globLabelNameLengthLimit, + LabelValueLengthLimit: globLabelValueLengthLimit, MetricsPath: DefaultScrapeConfig.MetricsPath, Scheme: DefaultScrapeConfig.Scheme, diff --git a/config/testdata/conf.good.yml b/config/testdata/conf.good.yml index 388b9de32d..19cfe1eb5d 100644 --- a/config/testdata/conf.good.yml +++ b/config/testdata/conf.good.yml @@ -2,6 +2,12 @@ global: scrape_interval: 15s evaluation_interval: 30s + body_size_limit: 15MB + sample_limit: 1500 + target_limit: 30 + label_limit: 30 + label_name_length_limit: 200 + label_value_length_limit: 200 # scrape_timeout is set to the global default (10s). external_labels: @@ -111,6 +117,11 @@ scrape_configs: body_size_limit: 10MB sample_limit: 1000 + target_limit: 35 + label_limit: 35 + label_name_length_limit: 210 + label_value_length_limit: 210 + metrics_path: /my_path scheme: https diff --git a/docs/configuration/configuration.md b/docs/configuration/configuration.md index b0b587e02a..ff1449e34a 100644 --- a/docs/configuration/configuration.md +++ b/docs/configuration/configuration.md @@ -73,6 +73,39 @@ global: # Reloading the configuration will reopen the file. [ query_log_file: ] + # An uncompressed response body larger than this many bytes will cause the + # scrape to fail. 0 means no limit. Example: 100MB. + # This is an experimental feature, this behaviour could + # change or be removed in the future. + [ body_size_limit: | default = 0 ] + + # Per-scrape limit on number of scraped samples that will be accepted. + # If more than this number of samples are present after metric relabeling + # the entire scrape will be treated as failed. 0 means no limit. + [ sample_limit: | default = 0 ] + + # Per-scrape limit on number of labels that will be accepted for a sample. If + # more than this number of labels are present post metric-relabeling, the + # entire scrape will be treated as failed. 0 means no limit. + [ label_limit: | default = 0 ] + + # Per-scrape limit on length of labels name that will be accepted for a sample. + # If a label name is longer than this number post metric-relabeling, the entire + # scrape will be treated as failed. 0 means no limit. + [ label_name_length_limit: | default = 0 ] + + # Per-scrape limit on length of labels value that will be accepted for a sample. + # If a label value is longer than this number post metric-relabeling, the + # entire scrape will be treated as failed. 0 means no limit. + [ label_value_length_limit: | default = 0 ] + + # Per-scrape config limit on number of unique targets that will be + # accepted. If more than this number of targets are present after target + # relabeling, Prometheus will mark the targets as failed without scraping them. + # 0 means no limit. This is an experimental feature, this behaviour could + # change in the future. + [ target_limit: | default = 0 ] + # Rule files specifies a list of globs. Rules and alerts are read from # all matching files. rule_files: From 73078bf73870be23142d3c6703d52b5da3279d17 Mon Sep 17 00:00:00 2001 From: Alan Protasio Date: Tue, 30 May 2023 04:49:22 -0700 Subject: [PATCH 185/231] Opmizing Group Regex (#12375) Signed-off-by: Alan Protasio --- tsdb/querier.go | 9 ++++++++- tsdb/querier_bench_test.go | 4 ++++ tsdb/querier_test.go | 21 +++++++++++++++++++++ 3 files changed, 33 insertions(+), 1 deletion(-) diff --git a/tsdb/querier.go b/tsdb/querier.go index 9baf3f2429..72b6b51414 100644 --- a/tsdb/querier.go +++ b/tsdb/querier.go @@ -190,7 +190,14 @@ func findSetMatches(pattern string) []string { } escaped := false sets := []*strings.Builder{{}} - for i := 4; i < len(pattern)-2; i++ { + init := 4 + end := len(pattern) - 2 + // If the regex is wrapped in a group we can remove the first and last parentheses + if pattern[init] == '(' && pattern[end-1] == ')' { + init++ + end-- + } + for i := init; i < end; i++ { if escaped { switch { case isRegexMetaCharacter(pattern[i]): diff --git a/tsdb/querier_bench_test.go b/tsdb/querier_bench_test.go index 19619e35bb..c6deaeb44c 100644 --- a/tsdb/querier_bench_test.go +++ b/tsdb/querier_bench_test.go @@ -113,6 +113,7 @@ func benchmarkPostingsForMatchers(b *testing.B, ir IndexReader) { jXplus := labels.MustNewMatcher(labels.MatchRegexp, "j", "X.+") iCharSet := labels.MustNewMatcher(labels.MatchRegexp, "i", "1[0-9]") iAlternate := labels.MustNewMatcher(labels.MatchRegexp, "i", "(1|2|3|4|5|6|20|55)") + iNotAlternate := labels.MustNewMatcher(labels.MatchNotRegexp, "i", "(1|2|3|4|5|6|20|55)") iXYZ := labels.MustNewMatcher(labels.MatchRegexp, "i", "X|Y|Z") iNotXYZ := labels.MustNewMatcher(labels.MatchNotRegexp, "i", "X|Y|Z") cases := []struct { @@ -132,6 +133,7 @@ func benchmarkPostingsForMatchers(b *testing.B, ir IndexReader) { {`j=~"XXX|YYY"`, []*labels.Matcher{jXXXYYY}}, {`j=~"X.+"`, []*labels.Matcher{jXplus}}, {`i=~"(1|2|3|4|5|6|20|55)"`, []*labels.Matcher{iAlternate}}, + {`i!~"(1|2|3|4|5|6|20|55)"`, []*labels.Matcher{iNotAlternate}}, {`i=~"X|Y|Z"`, []*labels.Matcher{iXYZ}}, {`i!~"X|Y|Z"`, []*labels.Matcher{iNotXYZ}}, {`i=~".*"`, []*labels.Matcher{iStar}}, @@ -161,6 +163,8 @@ func benchmarkPostingsForMatchers(b *testing.B, ir IndexReader) { for _, c := range cases { b.Run(c.name, func(b *testing.B) { + b.ReportAllocs() + b.ResetTimer() for i := 0; i < b.N; i++ { _, err := PostingsForMatchers(ir, c.matchers...) require.NoError(b, err) diff --git a/tsdb/querier_test.go b/tsdb/querier_test.go index 5bf721a620..e9dd3b75f4 100644 --- a/tsdb/querier_test.go +++ b/tsdb/querier_test.go @@ -2051,6 +2051,12 @@ func TestPostingsForMatchers(t *testing.T) { labels.FromStrings("n", "2"), }, }, + { + matchers: []*labels.Matcher{labels.MustNewMatcher(labels.MatchNotRegexp, "n", "(1|2.5)")}, + exp: []labels.Labels{ + labels.FromStrings("n", "2"), + }, + }, { matchers: []*labels.Matcher{labels.MustNewMatcher(labels.MatchEqual, "n", "1"), labels.MustNewMatcher(labels.MatchNotRegexp, "i", "^a$")}, exp: []labels.Labels{ @@ -2112,6 +2118,13 @@ func TestPostingsForMatchers(t *testing.T) { labels.FromStrings("n", "1", "i", "b"), }, }, + { + matchers: []*labels.Matcher{labels.MustNewMatcher(labels.MatchRegexp, "i", "(a|b)")}, + exp: []labels.Labels{ + labels.FromStrings("n", "1", "i", "a"), + labels.FromStrings("n", "1", "i", "b"), + }, + }, { matchers: []*labels.Matcher{labels.MustNewMatcher(labels.MatchRegexp, "n", "x1|2")}, exp: []labels.Labels{ @@ -2134,6 +2147,14 @@ func TestPostingsForMatchers(t *testing.T) { labels.FromStrings("n", "2.5"), }, }, + { + matchers: []*labels.Matcher{labels.MustNewMatcher(labels.MatchRegexp, "i", "(c||d)")}, + exp: []labels.Labels{ + labels.FromStrings("n", "1"), + labels.FromStrings("n", "2"), + labels.FromStrings("n", "2.5"), + }, + }, } ir, err := h.Index() From dfae954dc1137568f33564e8cffda321f2867925 Mon Sep 17 00:00:00 2001 From: Alan Protasio Date: Tue, 30 May 2023 08:42:38 -0700 Subject: [PATCH 186/231] Improving Performance on the API Gzip Handler (#12363) Using github.com/klauspost/compress package to replace the current Gzip Handler on the API. We see significant improvements using this handler over the current one as shown in the benchmark added. Also: * move selection of compression from `newCompressedResponseWriter` to `*CompressionHandler.ServeHTTP`. * renaming `compressedResponseWriter` since it now only does one kind of compression. Signed-off-by: Alan Protasio --- go.mod | 1 + go.sum | 1 + util/httputil/compression.go | 74 ++++++++----------- util/httputil/compression_test.go | 115 +++++++++++++++++++++++++++--- 4 files changed, 139 insertions(+), 52 deletions(-) diff --git a/go.mod b/go.mod index 15daecc2cf..9b826ad334 100644 --- a/go.mod +++ b/go.mod @@ -32,6 +32,7 @@ require ( github.com/hetznercloud/hcloud-go v1.45.1 github.com/ionos-cloud/sdk-go/v6 v6.1.6 github.com/json-iterator/go v1.1.12 + github.com/klauspost/compress v1.13.6 github.com/kolo/xmlrpc v0.0.0-20220921171641-a4b6fa1dd06b github.com/linode/linodego v1.16.1 github.com/miekg/dns v1.1.53 diff --git a/go.sum b/go.sum index 18d98dfaeb..3a472c99a0 100644 --- a/go.sum +++ b/go.sum @@ -499,6 +499,7 @@ github.com/karrick/godirwalk v1.10.3/go.mod h1:RoGL9dQei4vP9ilrpETWE8CLOZ1kiN0Lh github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= +github.com/klauspost/compress v1.13.6 h1:P76CopJELS0TiO2mebmnzgWaajssP/EszplttgQxcgc= github.com/klauspost/compress v1.13.6/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk= github.com/kolo/xmlrpc v0.0.0-20220921171641-a4b6fa1dd06b h1:udzkj9S/zlT5X367kqJis0QP7YMxobob6zhzq6Yre00= github.com/kolo/xmlrpc v0.0.0-20220921171641-a4b6fa1dd06b/go.mod h1:pcaDhQK0/NJZEvtCO0qQPPropqV0sJOJ6YW7X+9kRwM= diff --git a/util/httputil/compression.go b/util/httputil/compression.go index b96c088cbb..5e9276958d 100644 --- a/util/httputil/compression.go +++ b/util/httputil/compression.go @@ -14,11 +14,11 @@ package httputil import ( - "compress/gzip" - "compress/zlib" - "io" "net/http" "strings" + + "github.com/klauspost/compress/gzhttp" + "github.com/klauspost/compress/zlib" ) const ( @@ -28,53 +28,27 @@ const ( deflateEncoding = "deflate" ) -// Wrapper around http.Handler which adds suitable response compression based -// on the client's Accept-Encoding headers. -type compressedResponseWriter struct { +// Wrapper around http.ResponseWriter which adds deflate compression +type deflatedResponseWriter struct { http.ResponseWriter - writer io.Writer + writer *zlib.Writer } // Writes HTTP response content data. -func (c *compressedResponseWriter) Write(p []byte) (int, error) { +func (c *deflatedResponseWriter) Write(p []byte) (int, error) { return c.writer.Write(p) } -// Closes the compressedResponseWriter and ensures to flush all data before. -func (c *compressedResponseWriter) Close() { - if zlibWriter, ok := c.writer.(*zlib.Writer); ok { - zlibWriter.Flush() - } - if gzipWriter, ok := c.writer.(*gzip.Writer); ok { - gzipWriter.Flush() - } - if closer, ok := c.writer.(io.Closer); ok { - defer closer.Close() - } +// Close Closes the deflatedResponseWriter and ensures to flush all data before. +func (c *deflatedResponseWriter) Close() { + c.writer.Close() } -// Constructs a new compressedResponseWriter based on client request headers. -func newCompressedResponseWriter(writer http.ResponseWriter, req *http.Request) *compressedResponseWriter { - encodings := strings.Split(req.Header.Get(acceptEncodingHeader), ",") - for _, encoding := range encodings { - switch strings.TrimSpace(encoding) { - case gzipEncoding: - writer.Header().Set(contentEncodingHeader, gzipEncoding) - return &compressedResponseWriter{ - ResponseWriter: writer, - writer: gzip.NewWriter(writer), - } - case deflateEncoding: - writer.Header().Set(contentEncodingHeader, deflateEncoding) - return &compressedResponseWriter{ - ResponseWriter: writer, - writer: zlib.NewWriter(writer), - } - } - } - return &compressedResponseWriter{ +// Constructs a new deflatedResponseWriter to compress the original writer using 'deflate' compression. +func newDeflateResponseWriter(writer http.ResponseWriter) *deflatedResponseWriter { + return &deflatedResponseWriter{ ResponseWriter: writer, - writer: writer, + writer: zlib.NewWriter(writer), } } @@ -86,7 +60,21 @@ type CompressionHandler struct { // ServeHTTP adds compression to the original http.Handler's ServeHTTP() method. func (c CompressionHandler) ServeHTTP(writer http.ResponseWriter, req *http.Request) { - compWriter := newCompressedResponseWriter(writer, req) - c.Handler.ServeHTTP(compWriter, req) - compWriter.Close() + encodings := strings.Split(req.Header.Get(acceptEncodingHeader), ",") + for _, encoding := range encodings { + switch strings.TrimSpace(encoding) { + case gzipEncoding: + gzhttp.GzipHandler(c.Handler).ServeHTTP(writer, req) + return + case deflateEncoding: + compWriter := newDeflateResponseWriter(writer) + writer.Header().Set(contentEncodingHeader, deflateEncoding) + c.Handler.ServeHTTP(compWriter, req) + compWriter.Close() + return + default: + c.Handler.ServeHTTP(writer, req) + return + } + } } diff --git a/util/httputil/compression_test.go b/util/httputil/compression_test.go index 8512797613..b7148fc1cc 100644 --- a/util/httputil/compression_test.go +++ b/util/httputil/compression_test.go @@ -17,23 +17,30 @@ import ( "bytes" "compress/gzip" "compress/zlib" + "encoding/json" + "fmt" "io" "net/http" "net/http/httptest" + "strings" "testing" "github.com/stretchr/testify/require" + + "github.com/prometheus/prometheus/model/labels" ) var ( - mux *http.ServeMux - server *httptest.Server + mux *http.ServeMux + server *httptest.Server + respBody = strings.Repeat("Hello World!", 500) ) func setup() func() { mux = http.NewServeMux() server = httptest.NewServer(mux) return func() { + server.CloseClientConnections() server.Close() } } @@ -41,7 +48,7 @@ func setup() func() { func getCompressionHandlerFunc() CompressionHandler { hf := func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) - w.Write([]byte("Hello World!")) + w.Write([]byte(respBody)) } return CompressionHandler{ Handler: http.HandlerFunc(hf), @@ -67,9 +74,8 @@ func TestCompressionHandler_PlainText(t *testing.T) { contents, err := io.ReadAll(resp.Body) require.NoError(t, err, "unexpected error while creating the response body reader") - expected := "Hello World!" actual := string(contents) - require.Equal(t, expected, actual, "expected response with content") + require.Equal(t, respBody, actual, "expected response with content") } func TestCompressionHandler_Gzip(t *testing.T) { @@ -103,8 +109,7 @@ func TestCompressionHandler_Gzip(t *testing.T) { require.NoError(t, err, "unexpected error while reading the response body") actual := buf.String() - expected := "Hello World!" - require.Equal(t, expected, actual, "unexpected response content") + require.Equal(t, respBody, actual, "unexpected response content") } func TestCompressionHandler_Deflate(t *testing.T) { @@ -138,6 +143,98 @@ func TestCompressionHandler_Deflate(t *testing.T) { require.NoError(t, err, "unexpected error while reading the response body") actual := buf.String() - expected := "Hello World!" - require.Equal(t, expected, actual, "expected response with content") + require.Equal(t, respBody, actual, "expected response with content") +} + +func Benchmark_compression(b *testing.B) { + client := &http.Client{ + Transport: &http.Transport{ + DisableCompression: true, + }, + } + + cases := map[string]struct { + enc string + numberOfLabels int + }{ + "gzip-10-labels": { + enc: gzipEncoding, + numberOfLabels: 10, + }, + "gzip-100-labels": { + enc: gzipEncoding, + numberOfLabels: 100, + }, + "gzip-1K-labels": { + enc: gzipEncoding, + numberOfLabels: 1000, + }, + "gzip-10K-labels": { + enc: gzipEncoding, + numberOfLabels: 10000, + }, + "gzip-100K-labels": { + enc: gzipEncoding, + numberOfLabels: 100000, + }, + "gzip-1M-labels": { + enc: gzipEncoding, + numberOfLabels: 1000000, + }, + } + + for name, tc := range cases { + b.Run(name, func(b *testing.B) { + tearDown := setup() + defer tearDown() + labels := labels.ScratchBuilder{} + + for i := 0; i < tc.numberOfLabels; i++ { + labels.Add(fmt.Sprintf("Name%v", i), fmt.Sprintf("Value%v", i)) + } + + respBody, err := json.Marshal(labels.Labels()) + require.NoError(b, err) + + hf := func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusOK) + w.Write(respBody) + } + h := CompressionHandler{ + Handler: http.HandlerFunc(hf), + } + + mux.Handle("/foo_endpoint", h) + + req, _ := http.NewRequest("GET", server.URL+"/foo_endpoint", nil) + req.Header.Set(acceptEncodingHeader, tc.enc) + + b.ReportAllocs() + b.ResetTimer() + + // Reusing the array to read the body and avoid allocation on the test + encRespBody := make([]byte, len(respBody)) + + for i := 0; i < b.N; i++ { + resp, err := client.Do(req) + + require.NoError(b, err) + + require.NoError(b, err, "client get failed with unexpected error") + responseBodySize := 0 + for { + n, err := resp.Body.Read(encRespBody) + responseBodySize += n + if err == io.EOF { + break + } + } + + b.ReportMetric(float64(responseBodySize), "ContentLength") + resp.Body.Close() + } + + client.CloseIdleConnections() + }) + } } From ca6580828aeef078370116b86ff3fbe9759b75ed Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fran=C3=A7ois=20Gouteroux?= Date: Wed, 31 May 2023 15:17:44 +0200 Subject: [PATCH 187/231] feat: support histogram and summary metric types MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: François Gouteroux --- util/fmtutil/format.go | 171 +++++++++++++++++++++++++----------- util/fmtutil/format_test.go | 140 ++++++++++++++++++++++++++++- 2 files changed, 256 insertions(+), 55 deletions(-) diff --git a/util/fmtutil/format.go b/util/fmtutil/format.go index b5bb9469ce..291308dc23 100644 --- a/util/fmtutil/format.go +++ b/util/fmtutil/format.go @@ -14,6 +14,7 @@ package fmtutil import ( + "errors" "fmt" "io" "sort" @@ -26,6 +27,12 @@ import ( "github.com/prometheus/prometheus/prompb" ) +const ( + sumStr = "_sum" + countStr = "_count" + bucketStr = "_bucket" +) + var MetricMetadataTypeValue = map[string]int32{ "UNKNOWN": 0, "COUNTER": 1, @@ -60,71 +67,129 @@ func FormatMetrics(mf map[string]*dto.MetricFamily, extraLabels map[string]strin wr.Metadata = append(wr.Metadata, metadata) for _, metric := range mf[metricName].Metric { - var timeserie prompb.TimeSeries - - // build labels map - labels := make(map[string]string, len(metric.Label)+len(extraLabels)) - labels[model.MetricNameLabel] = metricName - - // add extra labels - for key, value := range extraLabels { - labels[key] = value + labels := makeLabelsMap(metric, metricName, extraLabels) + if err := makeTimeseries(wr, labels, metric); err != nil { + return wr, err } - - // add metric labels - for _, label := range metric.Label { - labelname := label.GetName() - if labelname == model.JobLabel { - labelname = fmt.Sprintf("%s%s", model.ExportedLabelPrefix, labelname) - } - labels[labelname] = label.GetValue() - } - - // build labels name list - sortedLabelNames := make([]string, 0, len(labels)) - for label := range labels { - sortedLabelNames = append(sortedLabelNames, label) - } - // sort labels name in lexicographical order - sort.Strings(sortedLabelNames) - - for _, label := range sortedLabelNames { - timeserie.Labels = append(timeserie.Labels, prompb.Label{ - Name: label, - Value: labels[label], - }) - } - - timestamp := metric.GetTimestampMs() - if timestamp == 0 { - timestamp = time.Now().UnixNano() / int64(time.Millisecond) - } - - timeserie.Samples = []prompb.Sample{ - { - Timestamp: timestamp, - Value: getMetricsValue(metric), - }, - } - - wr.Timeseries = append(wr.Timeseries, timeserie) } } return wr, nil } -// getMetricsValue return the value of a timeserie without the need to give value type -func getMetricsValue(m *dto.Metric) float64 { +func makeTimeserie(wr *prompb.WriteRequest, labels map[string]string, timestamp int64, value float64) { + var timeserie prompb.TimeSeries + timeserie.Labels = makeLabels(labels) + timeserie.Samples = []prompb.Sample{ + { + Timestamp: timestamp, + Value: value, + }, + } + wr.Timeseries = append(wr.Timeseries, timeserie) +} + +func makeTimeseries(wr *prompb.WriteRequest, labels map[string]string, m *dto.Metric) error { + var err error + + timestamp := m.GetTimestampMs() + if timestamp == 0 { + timestamp = time.Now().UnixNano() / int64(time.Millisecond) + } + switch { case m.Gauge != nil: - return m.GetGauge().GetValue() + makeTimeserie(wr, labels, timestamp, m.GetGauge().GetValue()) case m.Counter != nil: - return m.GetCounter().GetValue() + makeTimeserie(wr, labels, timestamp, m.GetCounter().GetValue()) + case m.Summary != nil: + metricName := labels[model.MetricNameLabel] + // Preserve metric name order with first quantile labels timeseries then sum suffix timeserie and finally count suffix timeserie + // Add Summary quantile timeseries + quantileLabels := make(map[string]string, len(labels)+1) + for key, value := range labels { + quantileLabels[key] = value + } + + for _, q := range m.GetSummary().Quantile { + quantileLabels[model.QuantileLabel] = fmt.Sprint(q.GetQuantile()) + makeTimeserie(wr, quantileLabels, timestamp, q.GetValue()) + } + // Overwrite label model.MetricNameLabel for count and sum metrics + // Add Summary sum timeserie + labels[model.MetricNameLabel] = metricName + sumStr + makeTimeserie(wr, labels, timestamp, m.GetSummary().GetSampleSum()) + // Add Summary count timeserie + labels[model.MetricNameLabel] = metricName + countStr + makeTimeserie(wr, labels, timestamp, float64(m.GetSummary().GetSampleCount())) + + case m.Histogram != nil: + metricName := labels[model.MetricNameLabel] + // Preserve metric name order with first bucket suffix timeseries then sum suffix timeserie and finally count suffix timeserie + // Add Histogram bucket timeseries + bucketLabels := make(map[string]string, len(labels)+1) + for key, value := range labels { + bucketLabels[key] = value + } + for _, b := range m.GetHistogram().Bucket { + bucketLabels[model.MetricNameLabel] = metricName + bucketStr + bucketLabels[model.BucketLabel] = fmt.Sprint(b.GetUpperBound()) + makeTimeserie(wr, bucketLabels, timestamp, float64(b.GetCumulativeCount())) + } + // Overwrite label model.MetricNameLabel for count and sum metrics + // Add Histogram sum timeserie + labels[model.MetricNameLabel] = metricName + sumStr + makeTimeserie(wr, labels, timestamp, m.GetHistogram().GetSampleSum()) + // Add Histogram count timeserie + labels[model.MetricNameLabel] = metricName + countStr + makeTimeserie(wr, labels, timestamp, float64(m.GetHistogram().GetSampleCount())) + case m.Untyped != nil: - return m.GetUntyped().GetValue() + makeTimeserie(wr, labels, timestamp, m.GetUntyped().GetValue()) default: - return 0. + err = errors.New("unsupported metric type") } + return err +} + +func makeLabels(labelsMap map[string]string) []prompb.Label { + // build labels name list + sortedLabelNames := make([]string, 0, len(labelsMap)) + for label := range labelsMap { + sortedLabelNames = append(sortedLabelNames, label) + } + // sort labels name in lexicographical order + sort.Strings(sortedLabelNames) + + var labels []prompb.Label + for _, label := range sortedLabelNames { + labels = append(labels, prompb.Label{ + Name: label, + Value: labelsMap[label], + }) + } + return labels +} + +func makeLabelsMap(m *dto.Metric, metricName string, extraLabels map[string]string) map[string]string { + // build labels map + labels := make(map[string]string, len(m.Label)+len(extraLabels)) + labels[model.MetricNameLabel] = metricName + + // add extra labels + for key, value := range extraLabels { + labels[key] = value + } + + // add metric labels + for _, label := range m.Label { + labelname := label.GetName() + if labelname == model.JobLabel { + labelname = fmt.Sprintf("%s%s", model.ExportedLabelPrefix, labelname) + } + labels[labelname] = label.GetValue() + } + + return labels } // ParseMetricsTextReader consumes an io.Reader and returns the MetricFamily. diff --git a/util/fmtutil/format_test.go b/util/fmtutil/format_test.go index 9deed2de90..e2ac301353 100644 --- a/util/fmtutil/format_test.go +++ b/util/fmtutil/format_test.go @@ -24,13 +24,130 @@ import ( var writeRequestFixture = &prompb.WriteRequest{ Metadata: []prompb.MetricMetadata{ + { + MetricFamilyName: "http_request_duration_seconds", + Type: 3, + Help: "A histogram of the request duration.", + }, + { + MetricFamilyName: "http_requests_total", + Type: 1, + Help: "The total number of HTTP requests.", + }, + { + MetricFamilyName: "rpc_duration_seconds", + Type: 5, + Help: "A summary of the RPC duration in seconds.", + }, { MetricFamilyName: "test_metric1", Type: 2, - Help: "this is a test metric", + Help: "This is a test metric.", }, }, Timeseries: []prompb.TimeSeries{ + { + Labels: []prompb.Label{ + {Name: "__name__", Value: "http_request_duration_seconds_bucket"}, + {Name: "job", Value: "promtool"}, + {Name: "le", Value: "0.1"}, + }, + Samples: []prompb.Sample{{Value: 33444, Timestamp: 1}}, + }, + { + Labels: []prompb.Label{ + {Name: "__name__", Value: "http_request_duration_seconds_bucket"}, + {Name: "job", Value: "promtool"}, + {Name: "le", Value: "0.5"}, + }, + Samples: []prompb.Sample{{Value: 129389, Timestamp: 1}}, + }, + { + Labels: []prompb.Label{ + {Name: "__name__", Value: "http_request_duration_seconds_bucket"}, + {Name: "job", Value: "promtool"}, + {Name: "le", Value: "1"}, + }, + Samples: []prompb.Sample{{Value: 133988, Timestamp: 1}}, + }, + { + Labels: []prompb.Label{ + {Name: "__name__", Value: "http_request_duration_seconds_bucket"}, + {Name: "job", Value: "promtool"}, + {Name: "le", Value: "+Inf"}, + }, + Samples: []prompb.Sample{{Value: 144320, Timestamp: 1}}, + }, + { + Labels: []prompb.Label{ + {Name: "__name__", Value: "http_request_duration_seconds_sum"}, + {Name: "job", Value: "promtool"}, + }, + Samples: []prompb.Sample{{Value: 53423, Timestamp: 1}}, + }, + { + Labels: []prompb.Label{ + {Name: "__name__", Value: "http_request_duration_seconds_count"}, + {Name: "job", Value: "promtool"}, + }, + Samples: []prompb.Sample{{Value: 144320, Timestamp: 1}}, + }, + { + Labels: []prompb.Label{ + {Name: "__name__", Value: "http_requests_total"}, + {Name: "code", Value: "200"}, + {Name: "job", Value: "promtool"}, + {Name: "method", Value: "post"}, + }, + Samples: []prompb.Sample{{Value: 1027, Timestamp: 1395066363000}}, + }, + { + Labels: []prompb.Label{ + {Name: "__name__", Value: "http_requests_total"}, + {Name: "code", Value: "400"}, + {Name: "job", Value: "promtool"}, + {Name: "method", Value: "post"}, + }, + Samples: []prompb.Sample{{Value: 3, Timestamp: 1395066363000}}, + }, + { + Labels: []prompb.Label{ + {Name: "__name__", Value: "rpc_duration_seconds"}, + {Name: "job", Value: "promtool"}, + {Name: "quantile", Value: "0.01"}, + }, + Samples: []prompb.Sample{{Value: 3102, Timestamp: 1}}, + }, + { + Labels: []prompb.Label{ + {Name: "__name__", Value: "rpc_duration_seconds"}, + {Name: "job", Value: "promtool"}, + {Name: "quantile", Value: "0.5"}, + }, + Samples: []prompb.Sample{{Value: 4773, Timestamp: 1}}, + }, + { + Labels: []prompb.Label{ + {Name: "__name__", Value: "rpc_duration_seconds"}, + {Name: "job", Value: "promtool"}, + {Name: "quantile", Value: "0.99"}, + }, + Samples: []prompb.Sample{{Value: 76656, Timestamp: 1}}, + }, + { + Labels: []prompb.Label{ + {Name: "__name__", Value: "rpc_duration_seconds_sum"}, + {Name: "job", Value: "promtool"}, + }, + Samples: []prompb.Sample{{Value: 1.7560473e+07, Timestamp: 1}}, + }, + { + Labels: []prompb.Label{ + {Name: "__name__", Value: "rpc_duration_seconds_count"}, + {Name: "job", Value: "promtool"}, + }, + Samples: []prompb.Sample{{Value: 2693, Timestamp: 1}}, + }, { Labels: []prompb.Label{ {Name: "__name__", Value: "test_metric1"}, @@ -58,7 +175,26 @@ var writeRequestFixture = &prompb.WriteRequest{ func TestParseMetricsTextAndFormat(t *testing.T) { input := bytes.NewReader([]byte(` - # HELP test_metric1 this is a test metric + # HELP http_request_duration_seconds A histogram of the request duration. + # TYPE http_request_duration_seconds histogram + http_request_duration_seconds_bucket{le="0.1"} 33444 1 + http_request_duration_seconds_bucket{le="0.5"} 129389 1 + http_request_duration_seconds_bucket{le="1"} 133988 1 + http_request_duration_seconds_bucket{le="+Inf"} 144320 1 + http_request_duration_seconds_sum 53423 1 + http_request_duration_seconds_count 144320 1 + # HELP http_requests_total The total number of HTTP requests. + # TYPE http_requests_total counter + http_requests_total{method="post",code="200"} 1027 1395066363000 + http_requests_total{method="post",code="400"} 3 1395066363000 + # HELP rpc_duration_seconds A summary of the RPC duration in seconds. + # TYPE rpc_duration_seconds summary + rpc_duration_seconds{quantile="0.01"} 3102 1 + rpc_duration_seconds{quantile="0.5"} 4773 1 + rpc_duration_seconds{quantile="0.99"} 76656 1 + rpc_duration_seconds_sum 1.7560473e+07 1 + rpc_duration_seconds_count 2693 1 + # HELP test_metric1 This is a test metric. # TYPE test_metric1 gauge test_metric1{b="c",baz="qux",d="e",foo="bar"} 1 1 test_metric1{b="c",baz="qux",d="e",foo="bar"} 2 1 From 6ae4a46845295e88c353de0dd42caa2b0fe1b46f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fran=C3=A7ois=20Gouteroux?= Date: Thu, 1 Jun 2023 10:28:55 +0200 Subject: [PATCH 188/231] feat: enhance stdin check and add tests parsing error MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: François Gouteroux --- cmd/promtool/main.go | 2 +- cmd/promtool/metrics.go | 90 +++++++++++++++++------------------ docs/command-line/promtool.md | 2 +- util/fmtutil/format_test.go | 27 ++++++++++- 4 files changed, 71 insertions(+), 50 deletions(-) diff --git a/cmd/promtool/main.go b/cmd/promtool/main.go index c76790e13b..bcfcc24223 100644 --- a/cmd/promtool/main.go +++ b/cmd/promtool/main.go @@ -185,7 +185,7 @@ func main() { pushMetricsCmd.Arg("remote-write-url", "Prometheus remote write url to push metrics.").Required().URLVar(&remoteWriteURL) metricFiles := pushMetricsCmd.Arg( "metric-files", - "The metric files to push, default is read from standard input (STDIN).", + "The metric files to push, default is read from standard input.", ).ExistingFiles() pushMetricsLabels := pushMetricsCmd.Flag("label", "Label to attach to metrics. Can be specified multiple times.").Default("job=promtool").StringMap() pushMetricsTimeout := pushMetricsCmd.Flag("timeout", "The time to wait for pushing metrics.").Default("30s").Duration() diff --git a/cmd/promtool/metrics.go b/cmd/promtool/metrics.go index 8abe32cf41..4a6fafd407 100644 --- a/cmd/promtool/metrics.go +++ b/cmd/promtool/metrics.go @@ -33,8 +33,6 @@ import ( // Push metrics to a prometheus remote write (for testing purpose only). func PushMetrics(url *url.URL, roundTripper http.RoundTripper, headers map[string]string, timeout time.Duration, labels map[string]string, files ...string) int { - failed := false - addressURL, err := url.Parse(url.String()) if err != nil { fmt.Fprintln(os.Stderr, err) @@ -63,63 +61,37 @@ func PushMetrics(url *url.URL, roundTripper http.RoundTripper, headers map[strin headers: headers, } - // add empty string to avoid matching filename + var data []byte + var failed bool + if len(files) == 0 { - files = append(files, "") + data, err = io.ReadAll(os.Stdin) + if err != nil { + fmt.Fprintln(os.Stderr, " FAILED:", err) + return failureExitCode + } + fmt.Printf("Parsing standard input\n") + if parseAndPushMetrics(client, data, labels) { + fmt.Printf(" SUCCESS: metrics pushed to remote write.\n") + return successExitCode + } + return failureExitCode } for _, file := range files { - var data []byte - var err error - - // if file is an empty string it is a stdin - if file == "" { - data, err = io.ReadAll(os.Stdin) - if err != nil { - fmt.Fprintln(os.Stderr, " FAILED:", err) - failed = true - break - } - - fmt.Printf("Parsing input from stdin\n") - } else { - data, err = os.ReadFile(file) - if err != nil { - fmt.Fprintln(os.Stderr, " FAILED:", err) - failed = true - continue - } - - fmt.Printf("Parsing input from metric file %s\n", file) - } - metricsData, err := fmtutil.ParseMetricsTextAndFormat(bytes.NewReader(data), labels) + data, err = os.ReadFile(file) if err != nil { fmt.Fprintln(os.Stderr, " FAILED:", err) failed = true continue } - raw, err := metricsData.Marshal() - if err != nil { - fmt.Fprintln(os.Stderr, " FAILED:", err) - failed = true + fmt.Printf("Parsing metrics file %s\n", file) + if parseAndPushMetrics(client, data, labels) { + fmt.Printf(" SUCCESS: metrics file %s pushed to remote write.\n", file) continue } - - // Encode the request body into snappy encoding. - compressed := snappy.Encode(nil, raw) - err = client.Store(context.Background(), compressed) - if err != nil { - fmt.Fprintln(os.Stderr, " FAILED:", err) - failed = true - continue - } - - if file == "" { - fmt.Printf(" SUCCESS: metric pushed to remote write.\n") - } else { - fmt.Printf(" SUCCESS: metric file %s pushed to remote write.\n", file) - } + failed = true } if failed { @@ -129,6 +101,30 @@ func PushMetrics(url *url.URL, roundTripper http.RoundTripper, headers map[strin return successExitCode } +func parseAndPushMetrics(client *remote.Client, data []byte, labels map[string]string) bool { + metricsData, err := fmtutil.ParseMetricsTextAndFormat(bytes.NewReader(data), labels) + if err != nil { + fmt.Fprintln(os.Stderr, " FAILED:", err) + return false + } + + raw, err := metricsData.Marshal() + if err != nil { + fmt.Fprintln(os.Stderr, " FAILED:", err) + return false + } + + // Encode the request body into snappy encoding. + compressed := snappy.Encode(nil, raw) + err = client.Store(context.Background(), compressed) + if err != nil { + fmt.Fprintln(os.Stderr, " FAILED:", err) + return false + } + + return true +} + type setHeadersTransport struct { http.RoundTripper headers map[string]string diff --git a/docs/command-line/promtool.md b/docs/command-line/promtool.md index c78900b991..c36caaf616 100644 --- a/docs/command-line/promtool.md +++ b/docs/command-line/promtool.md @@ -410,7 +410,7 @@ Push metrics to a prometheus remote write (for testing purpose only). | Argument | Description | Required | | --- | --- | --- | | remote-write-url | Prometheus remote write url to push metrics. | Yes | -| metric-files | The metric files to push, default is read from standard input (STDIN). | | +| metric-files | The metric files to push, default is read from standard input. | | diff --git a/util/fmtutil/format_test.go b/util/fmtutil/format_test.go index e2ac301353..5c1ab5bde0 100644 --- a/util/fmtutil/format_test.go +++ b/util/fmtutil/format_test.go @@ -173,7 +173,7 @@ var writeRequestFixture = &prompb.WriteRequest{ }, } -func TestParseMetricsTextAndFormat(t *testing.T) { +func TestParseAndPushMetricsTextAndFormat(t *testing.T) { input := bytes.NewReader([]byte(` # HELP http_request_duration_seconds A histogram of the request duration. # TYPE http_request_duration_seconds histogram @@ -206,3 +206,28 @@ func TestParseMetricsTextAndFormat(t *testing.T) { require.Equal(t, writeRequestFixture, expected) } + +func TestParseMetricsTextAndFormatErrorParsingFloatValue(t *testing.T) { + input := bytes.NewReader([]byte(` + # HELP http_requests_total The total number of HTTP requests. + # TYPE http_requests_total counter + http_requests_total{method="post",code="200"} 1027Error 1395066363000 + http_requests_total{method="post",code="400"} 3 1395066363000 + `)) + labels := map[string]string{"job": "promtool"} + + _, err := ParseMetricsTextAndFormat(input, labels) + require.Equal(t, err.Error(), "text format parsing error in line 4: expected float as value, got \"1027Error\"") +} + +func TestParseMetricsTextAndFormatErrorParsingMetricType(t *testing.T) { + input := bytes.NewReader([]byte(` + # HELP node_info node info summary. + # TYPE node_info info + node_info{test="summary"} 1 1395066363000 + `)) + labels := map[string]string{"job": "promtool"} + + _, err := ParseMetricsTextAndFormat(input, labels) + require.Equal(t, err.Error(), "text format parsing error in line 3: unknown metric type \"info\"") +} From a8772a41782758e8153c494dcd207e770d8421a4 Mon Sep 17 00:00:00 2001 From: Nidhey Nitin Indurkar <46122307+nidhey27@users.noreply.github.com> Date: Thu, 1 Jun 2023 17:13:09 +0530 Subject: [PATCH 189/231] Feat: Get block by id directly on promtool analyze & get latest block if ID not provided (#12031) * feat: analyze latest block or block by ID in CLI (promtool) Signed-off-by: nidhey27 * address remarks Signed-off-by: nidhey60@gmail.com * address latest review comments Signed-off-by: nidhey60@gmail.com --------- Signed-off-by: nidhey27 Signed-off-by: nidhey60@gmail.com --- cmd/promtool/tsdb.go | 28 +++++++++-------------- tsdb/db.go | 54 ++++++++++++++++++++++++++++++++++++++++++++ tsdb/db_test.go | 19 +++++++++++++++- 3 files changed, 83 insertions(+), 18 deletions(-) diff --git a/cmd/promtool/tsdb.go b/cmd/promtool/tsdb.go index 84aa43a9c4..b7fad5fe09 100644 --- a/cmd/promtool/tsdb.go +++ b/cmd/promtool/tsdb.go @@ -398,26 +398,20 @@ func openBlock(path, blockID string) (*tsdb.DBReadOnly, tsdb.BlockReader, error) if err != nil { return nil, nil, err } - blocks, err := db.Blocks() + + if blockID == "" { + blockID, err = db.LastBlockID() + if err != nil { + return nil, nil, err + } + } + + b, err := db.Block(blockID) if err != nil { return nil, nil, err } - var block tsdb.BlockReader - switch { - case blockID != "": - for _, b := range blocks { - if b.Meta().ULID.String() == blockID { - block = b - break - } - } - case len(blocks) > 0: - block = blocks[len(blocks)-1] - } - if block == nil { - return nil, nil, fmt.Errorf("block %s not found", blockID) - } - return db, block, nil + + return db, b, nil } func analyzeBlock(path, blockID string, limit int, runExtended bool) error { diff --git a/tsdb/db.go b/tsdb/db.go index 12974150be..32dae57a52 100644 --- a/tsdb/db.go +++ b/tsdb/db.go @@ -607,6 +607,60 @@ func (db *DBReadOnly) Blocks() ([]BlockReader, error) { return blockReaders, nil } +// LastBlockID returns the BlockID of latest block. +func (db *DBReadOnly) LastBlockID() (string, error) { + entries, err := os.ReadDir(db.dir) + if err != nil { + return "", err + } + + max := uint64(0) + + lastBlockID := "" + + for _, e := range entries { + // Check if dir is a block dir or not. + dirName := e.Name() + ulidObj, err := ulid.ParseStrict(dirName) + if err != nil { + continue // Not a block dir. + } + timestamp := ulidObj.Time() + if timestamp > max { + max = timestamp + lastBlockID = dirName + } + } + + if lastBlockID == "" { + return "", errors.New("no blocks found") + } + + return lastBlockID, nil +} + +// Block returns a block reader by given block id. +func (db *DBReadOnly) Block(blockID string) (BlockReader, error) { + select { + case <-db.closed: + return nil, ErrClosed + default: + } + + _, err := os.Stat(filepath.Join(db.dir, blockID)) + if os.IsNotExist(err) { + return nil, errors.Errorf("invalid block ID %s", blockID) + } + + block, err := OpenBlock(db.logger, filepath.Join(db.dir, blockID), nil) + if err != nil { + return nil, err + } + db.closers = append(db.closers, block) + + return block, nil +} + // Close all block readers. func (db *DBReadOnly) Close() error { select { diff --git a/tsdb/db_test.go b/tsdb/db_test.go index 0b980f6ec6..427a3b7afa 100644 --- a/tsdb/db_test.go +++ b/tsdb/db_test.go @@ -2384,6 +2384,7 @@ func TestDBReadOnly(t *testing.T) { dbDir string logger = log.NewLogfmtLogger(log.NewSyncWriter(os.Stderr)) expBlocks []*Block + expBlock *Block expSeries map[string][]tsdbutil.Sample expChunks map[string][][]tsdbutil.Sample expDBHash []byte @@ -2427,6 +2428,7 @@ func TestDBReadOnly(t *testing.T) { require.NoError(t, app.Commit()) expBlocks = dbWritable.Blocks() + expBlock = expBlocks[0] expDbSize, err := fileutil.DirSize(dbWritable.Dir()) require.NoError(t, err) require.Greater(t, expDbSize, dbSizeBeforeAppend, "db size didn't increase after an append") @@ -2455,7 +2457,22 @@ func TestDBReadOnly(t *testing.T) { require.Equal(t, expBlock.Meta(), blocks[i].Meta(), "block meta mismatch") } }) - + t.Run("block", func(t *testing.T) { + blockID := expBlock.meta.ULID.String() + block, err := dbReadOnly.Block(blockID) + require.NoError(t, err) + require.Equal(t, expBlock.Meta(), block.Meta(), "block meta mismatch") + }) + t.Run("invalid block ID", func(t *testing.T) { + blockID := "01GTDVZZF52NSWB5SXQF0P2PGF" + _, err := dbReadOnly.Block(blockID) + require.Error(t, err) + }) + t.Run("last block ID", func(t *testing.T) { + blockID, err := dbReadOnly.LastBlockID() + require.NoError(t, err) + require.Equal(t, expBlocks[2].Meta().ULID.String(), blockID) + }) t.Run("querier", func(t *testing.T) { // Open a read only db and ensure that the API returns the same result as the normal DB. q, err := dbReadOnly.Querier(context.TODO(), math.MinInt64, math.MaxInt64) From f676d4a756f6833ba2694b2c6600259ff3c62533 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fran=C3=A7ois=20Gouteroux?= Date: Thu, 1 Jun 2023 17:39:04 +0200 Subject: [PATCH 190/231] feat refactoring checkrules func MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: François Gouteroux --- cmd/promtool/main.go | 102 +++++++++++++++++++++++-------------------- 1 file changed, 55 insertions(+), 47 deletions(-) diff --git a/cmd/promtool/main.go b/cmd/promtool/main.go index c55e5be1ef..e29d6f1602 100644 --- a/cmd/promtool/main.go +++ b/cmd/promtool/main.go @@ -448,20 +448,12 @@ func CheckConfig(agentMode, checkSyntaxOnly bool, lintSettings lintConfig, files } fmt.Println() - for _, rf := range ruleFiles { - if n, errs := checkRules(rf, lintSettings); len(errs) > 0 { - fmt.Fprintln(os.Stderr, " FAILED:") - for _, err := range errs { - fmt.Fprintln(os.Stderr, " ", err) - } - failed = true - for _, err := range errs { - hasErrors = hasErrors || !errors.Is(err, lintError) - } - } else { - fmt.Printf(" SUCCESS: %d rules found\n", n) - } - fmt.Println() + rulesFailed, rulesHasErrors := checkRules(ruleFiles, lintSettings) + if rulesFailed { + failed = rulesFailed + } + if rulesHasErrors { + hasErrors = rulesHasErrors } } if failed && hasErrors { @@ -689,14 +681,57 @@ func checkSDFile(filename string) ([]*targetgroup.Group, error) { func CheckRules(ls lintConfig, files ...string) int { failed := false hasErrors := false - - // Add empty string to avoid matching filename. if len(files) == 0 { - files = append(files, "") + fmt.Println("Checking standard input") + data, err := io.ReadAll(os.Stdin) + if err != nil { + fmt.Fprintln(os.Stderr, " FAILED:", err) + return failureExitCode + } + rgs, errs := rulefmt.Parse(data) + for _, e := range errs { + fmt.Fprintln(os.Stderr, e.Error()) + return failureExitCode + } + if n, errs := checkRuleGroups(rgs, ls); errs != nil { + fmt.Fprintln(os.Stderr, " FAILED:") + for _, e := range errs { + fmt.Fprintln(os.Stderr, e.Error()) + } + failed = true + for _, err := range errs { + hasErrors = hasErrors || !errors.Is(err, lintError) + } + } else { + fmt.Printf(" SUCCESS: %d rules found\n", n) + } + fmt.Println() + } else { + failed, hasErrors = checkRules(files, ls) } + if failed && hasErrors { + return failureExitCode + } + if failed && ls.fatal { + return lintErrExitCode + } + + return successExitCode +} + +// checkRules validates rule files. +func checkRules(files []string, ls lintConfig) (bool, bool) { + failed := false + hasErrors := false for _, f := range files { - if n, errs := checkRules(f, ls); errs != nil { + fmt.Println("Checking", f) + rgs, errs := rulefmt.ParseFile(f) + if errs != nil { + failed = true + continue + } + if n, errs := checkRuleGroups(rgs, ls); errs != nil { fmt.Fprintln(os.Stderr, " FAILED:") for _, e := range errs { fmt.Fprintln(os.Stderr, e.Error()) @@ -710,37 +745,10 @@ func CheckRules(ls lintConfig, files ...string) int { } fmt.Println() } - if failed && hasErrors { - return failureExitCode - } - if failed && ls.fatal { - return lintErrExitCode - } - return successExitCode + return failed, hasErrors } -func checkRules(filename string, lintSettings lintConfig) (int, []error) { - var rgs *rulefmt.RuleGroups - var errs []error - - // Empty string is stdin input. - if filename == "" { - data, err := io.ReadAll(os.Stdin) - if err != nil { - errs = append(errs, err) - return failureExitCode, errs - } - fmt.Println("Checking stdin") - rgs, errs = rulefmt.Parse(data) - } else { - fmt.Println("Checking", filename) - rgs, errs = rulefmt.ParseFile(filename) - } - - if errs != nil { - return successExitCode, errs - } - +func checkRuleGroups(rgs *rulefmt.RuleGroups, lintSettings lintConfig) (int, []error) { numRules := 0 for _, rg := range rgs.Groups { numRules += len(rg.Rules) From 1f3821379c23ab5855df88ad33f5bb5ecfd68a03 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Thu, 1 Jun 2023 17:17:04 +0000 Subject: [PATCH 191/231] promql: refactor: extract fn to wait on concurrency limit Signed-off-by: Bryan Boreham --- promql/engine.go | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/promql/engine.go b/promql/engine.go index 8a64fdf394..8730466f75 100644 --- a/promql/engine.go +++ b/promql/engine.go @@ -589,18 +589,11 @@ func (ng *Engine) exec(ctx context.Context, q *query) (v parser.Value, ws storag execSpanTimer, ctx := q.stats.GetSpanTimer(ctx, stats.ExecTotalTime) defer execSpanTimer.Finish() - queueSpanTimer, _ := q.stats.GetSpanTimer(ctx, stats.ExecQueueTime, ng.metrics.queryQueueTime) - // Log query in active log. The active log guarantees that we don't run over - // MaxConcurrent queries. - if ng.activeQueryTracker != nil { - queryIndex, err := ng.activeQueryTracker.Insert(ctx, q.q) - if err != nil { - queueSpanTimer.Finish() - return nil, nil, contextErr(err, "query queue") - } - defer ng.activeQueryTracker.Delete(queryIndex) + if finish, err := ng.queueActive(ctx, q); err != nil { + return nil, nil, err + } else { + defer finish() } - queueSpanTimer.Finish() // Cancel when execution is done or an error was raised. defer q.cancel() @@ -623,6 +616,18 @@ func (ng *Engine) exec(ctx context.Context, q *query) (v parser.Value, ws storag panic(fmt.Errorf("promql.Engine.exec: unhandled statement of type %T", q.Statement())) } +// Log query in active log. The active log guarantees that we don't run over +// MaxConcurrent queries. +func (ng *Engine) queueActive(ctx context.Context, q *query) (func(), error) { + if ng.activeQueryTracker == nil { + return func() {}, nil + } + queueSpanTimer, _ := q.stats.GetSpanTimer(ctx, stats.ExecQueueTime, ng.metrics.queryQueueTime) + queryIndex, err := ng.activeQueryTracker.Insert(ctx, q.q) + queueSpanTimer.Finish() + return func() { ng.activeQueryTracker.Delete(queryIndex) }, err +} + func timeMilliseconds(t time.Time) int64 { return t.UnixNano() / int64(time.Millisecond/time.Nanosecond) } From 71fc4f1516b8264606daca7bf7c0e011b7f91f2e Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Thu, 1 Jun 2023 17:54:17 +0000 Subject: [PATCH 192/231] promql: refactor: create query object before parsing Signed-off-by: Bryan Boreham --- promql/engine.go | 26 +++++++++++--------------- 1 file changed, 11 insertions(+), 15 deletions(-) diff --git a/promql/engine.go b/promql/engine.go index 8730466f75..81fb3c9168 100644 --- a/promql/engine.go +++ b/promql/engine.go @@ -409,15 +409,15 @@ func (ng *Engine) SetQueryLogger(l QueryLogger) { // NewInstantQuery returns an evaluation query for the given expression at the given time. func (ng *Engine) NewInstantQuery(_ context.Context, q storage.Queryable, opts *QueryOpts, qs string, ts time.Time) (Query, error) { + pExpr, qry := ng.newQuery(q, qs, opts, ts, ts, 0) expr, err := parser.ParseExpr(qs) if err != nil { return nil, err } - qry, err := ng.newQuery(q, opts, expr, ts, ts, 0) - if err != nil { + if err := ng.validateOpts(expr); err != nil { return nil, err } - qry.q = qs + *pExpr = PreprocessExpr(expr, ts, ts) return qry, nil } @@ -425,27 +425,23 @@ func (ng *Engine) NewInstantQuery(_ context.Context, q storage.Queryable, opts * // NewRangeQuery returns an evaluation query for the given time range and with // the resolution set by the interval. func (ng *Engine) NewRangeQuery(_ context.Context, q storage.Queryable, opts *QueryOpts, qs string, start, end time.Time, interval time.Duration) (Query, error) { + pExpr, qry := ng.newQuery(q, qs, opts, start, end, interval) expr, err := parser.ParseExpr(qs) if err != nil { return nil, err } + if err := ng.validateOpts(expr); err != nil { + return nil, err + } if expr.Type() != parser.ValueTypeVector && expr.Type() != parser.ValueTypeScalar { return nil, fmt.Errorf("invalid expression type %q for range query, must be Scalar or instant Vector", parser.DocumentedType(expr.Type())) } - qry, err := ng.newQuery(q, opts, expr, start, end, interval) - if err != nil { - return nil, err - } - qry.q = qs + *pExpr = PreprocessExpr(expr, start, end) return qry, nil } -func (ng *Engine) newQuery(q storage.Queryable, opts *QueryOpts, expr parser.Expr, start, end time.Time, interval time.Duration) (*query, error) { - if err := ng.validateOpts(expr); err != nil { - return nil, err - } - +func (ng *Engine) newQuery(q storage.Queryable, qs string, opts *QueryOpts, start, end time.Time, interval time.Duration) (*parser.Expr, *query) { // Default to empty QueryOpts if not provided. if opts == nil { opts = &QueryOpts{} @@ -457,20 +453,20 @@ func (ng *Engine) newQuery(q storage.Queryable, opts *QueryOpts, expr parser.Exp } es := &parser.EvalStmt{ - Expr: PreprocessExpr(expr, start, end), Start: start, End: end, Interval: interval, LookbackDelta: lookbackDelta, } qry := &query{ + q: qs, stmt: es, ng: ng, stats: stats.NewQueryTimers(), sampleStats: stats.NewQuerySamples(ng.enablePerStepStats && opts.EnablePerStepStats), queryable: q, } - return qry, nil + return &es.Expr, qry } var ( From bb0d8320ddc3b8f8d4888524a7f631f904a5ec6f Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Thu, 1 Jun 2023 18:16:05 +0000 Subject: [PATCH 193/231] promql: include parsing in active-query tracking So that the max-concurrency limit is applied. Signed-off-by: Bryan Boreham --- promql/engine.go | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/promql/engine.go b/promql/engine.go index 81fb3c9168..085a7d23c4 100644 --- a/promql/engine.go +++ b/promql/engine.go @@ -408,8 +408,13 @@ func (ng *Engine) SetQueryLogger(l QueryLogger) { } // NewInstantQuery returns an evaluation query for the given expression at the given time. -func (ng *Engine) NewInstantQuery(_ context.Context, q storage.Queryable, opts *QueryOpts, qs string, ts time.Time) (Query, error) { +func (ng *Engine) NewInstantQuery(ctx context.Context, q storage.Queryable, opts *QueryOpts, qs string, ts time.Time) (Query, error) { pExpr, qry := ng.newQuery(q, qs, opts, ts, ts, 0) + if finish, err := ng.queueActive(ctx, qry); err != nil { + return nil, err + } else { + defer finish() + } expr, err := parser.ParseExpr(qs) if err != nil { return nil, err @@ -424,8 +429,13 @@ func (ng *Engine) NewInstantQuery(_ context.Context, q storage.Queryable, opts * // NewRangeQuery returns an evaluation query for the given time range and with // the resolution set by the interval. -func (ng *Engine) NewRangeQuery(_ context.Context, q storage.Queryable, opts *QueryOpts, qs string, start, end time.Time, interval time.Duration) (Query, error) { +func (ng *Engine) NewRangeQuery(ctx context.Context, q storage.Queryable, opts *QueryOpts, qs string, start, end time.Time, interval time.Duration) (Query, error) { pExpr, qry := ng.newQuery(q, qs, opts, start, end, interval) + if finish, err := ng.queueActive(ctx, qry); err != nil { + return nil, err + } else { + defer finish() + } expr, err := parser.ParseExpr(qs) if err != nil { return nil, err From 67d2ef004d84427626fadb6e62b420b80712a7f9 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Thu, 1 Jun 2023 18:36:34 +0000 Subject: [PATCH 194/231] Placate lint I think the version using scoping was better, but I'm out of energy to fight the linter. Signed-off-by: Bryan Boreham --- promql/engine.go | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/promql/engine.go b/promql/engine.go index 085a7d23c4..f29db3a647 100644 --- a/promql/engine.go +++ b/promql/engine.go @@ -410,11 +410,11 @@ func (ng *Engine) SetQueryLogger(l QueryLogger) { // NewInstantQuery returns an evaluation query for the given expression at the given time. func (ng *Engine) NewInstantQuery(ctx context.Context, q storage.Queryable, opts *QueryOpts, qs string, ts time.Time) (Query, error) { pExpr, qry := ng.newQuery(q, qs, opts, ts, ts, 0) - if finish, err := ng.queueActive(ctx, qry); err != nil { + finishQueue, err := ng.queueActive(ctx, qry) + if err != nil { return nil, err - } else { - defer finish() } + defer finishQueue() expr, err := parser.ParseExpr(qs) if err != nil { return nil, err @@ -431,11 +431,11 @@ func (ng *Engine) NewInstantQuery(ctx context.Context, q storage.Queryable, opts // the resolution set by the interval. func (ng *Engine) NewRangeQuery(ctx context.Context, q storage.Queryable, opts *QueryOpts, qs string, start, end time.Time, interval time.Duration) (Query, error) { pExpr, qry := ng.newQuery(q, qs, opts, start, end, interval) - if finish, err := ng.queueActive(ctx, qry); err != nil { + finishQueue, err := ng.queueActive(ctx, qry) + if err != nil { return nil, err - } else { - defer finish() } + defer finishQueue() expr, err := parser.ParseExpr(qs) if err != nil { return nil, err @@ -595,11 +595,11 @@ func (ng *Engine) exec(ctx context.Context, q *query) (v parser.Value, ws storag execSpanTimer, ctx := q.stats.GetSpanTimer(ctx, stats.ExecTotalTime) defer execSpanTimer.Finish() - if finish, err := ng.queueActive(ctx, q); err != nil { + finishQueue, err := ng.queueActive(ctx, q) + if err != nil { return nil, nil, err - } else { - defer finish() } + defer finishQueue() // Cancel when execution is done or an error was raised. defer q.cancel() From b1675e23af84e6b156da0c17b5b12736775896b1 Mon Sep 17 00:00:00 2001 From: rakshith210 <56937390+rakshith210@users.noreply.github.com> Date: Thu, 1 Jun 2023 14:20:10 -0700 Subject: [PATCH 195/231] Add Azure AD package for remote write (#11944) * Add Azure AD package for remote write * Made AzurePublic default and updated configuration.md * Updated config structure and removed getToken at initialization * Changed passing context from request Signed-off-by: Rakshith Padmanabha Signed-off-by: rakshith210 --- config/config.go | 12 +- config/config_test.go | 2 +- docs/configuration/configuration.md | 14 +- go.mod | 9 +- go.sum | 12 + storage/remote/azuread/README.md | 8 + storage/remote/azuread/azuread.go | 247 +++++++++++++++++ storage/remote/azuread/azuread_test.go | 252 ++++++++++++++++++ .../testdata/azuread_bad_clientidmissing.yaml | 1 + .../testdata/azuread_bad_invalidclientid.yaml | 3 + .../remote/azuread/testdata/azuread_good.yaml | 3 + .../testdata/azuread_good_cloudmissing.yaml | 2 + storage/remote/client.go | 9 + storage/remote/write.go | 1 + 14 files changed, 568 insertions(+), 7 deletions(-) create mode 100644 storage/remote/azuread/README.md create mode 100644 storage/remote/azuread/azuread.go create mode 100644 storage/remote/azuread/azuread_test.go create mode 100644 storage/remote/azuread/testdata/azuread_bad_clientidmissing.yaml create mode 100644 storage/remote/azuread/testdata/azuread_bad_invalidclientid.yaml create mode 100644 storage/remote/azuread/testdata/azuread_good.yaml create mode 100644 storage/remote/azuread/testdata/azuread_good_cloudmissing.yaml diff --git a/config/config.go b/config/config.go index 9f81bbfd57..d32fcc33c9 100644 --- a/config/config.go +++ b/config/config.go @@ -34,6 +34,7 @@ import ( "github.com/prometheus/prometheus/discovery" "github.com/prometheus/prometheus/model/labels" "github.com/prometheus/prometheus/model/relabel" + "github.com/prometheus/prometheus/storage/remote/azuread" ) var ( @@ -907,6 +908,7 @@ type RemoteWriteConfig struct { QueueConfig QueueConfig `yaml:"queue_config,omitempty"` MetadataConfig MetadataConfig `yaml:"metadata_config,omitempty"` SigV4Config *sigv4.SigV4Config `yaml:"sigv4,omitempty"` + AzureADConfig *azuread.AzureADConfig `yaml:"azuread,omitempty"` } // SetDirectory joins any relative file paths with dir. @@ -943,8 +945,12 @@ func (c *RemoteWriteConfig) UnmarshalYAML(unmarshal func(interface{}) error) err httpClientConfigAuthEnabled := c.HTTPClientConfig.BasicAuth != nil || c.HTTPClientConfig.Authorization != nil || c.HTTPClientConfig.OAuth2 != nil - if httpClientConfigAuthEnabled && c.SigV4Config != nil { - return fmt.Errorf("at most one of basic_auth, authorization, oauth2, & sigv4 must be configured") + if httpClientConfigAuthEnabled && (c.SigV4Config != nil || c.AzureADConfig != nil) { + return fmt.Errorf("at most one of basic_auth, authorization, oauth2, sigv4, & azuread must be configured") + } + + if c.SigV4Config != nil && c.AzureADConfig != nil { + return fmt.Errorf("at most one of basic_auth, authorization, oauth2, sigv4, & azuread must be configured") } return nil @@ -965,7 +971,7 @@ func validateHeadersForTracing(headers map[string]string) error { func validateHeaders(headers map[string]string) error { for header := range headers { if strings.ToLower(header) == "authorization" { - return errors.New("authorization header must be changed via the basic_auth, authorization, oauth2, or sigv4 parameter") + return errors.New("authorization header must be changed via the basic_auth, authorization, oauth2, sigv4, or azuread parameter") } if _, ok := reservedHeaders[strings.ToLower(header)]; ok { return fmt.Errorf("%s is a reserved header. It must not be changed", header) diff --git a/config/config_test.go b/config/config_test.go index d243d687c4..d3288cc90d 100644 --- a/config/config_test.go +++ b/config/config_test.go @@ -1727,7 +1727,7 @@ var expectedErrors = []struct { }, { filename: "remote_write_authorization_header.bad.yml", - errMsg: `authorization header must be changed via the basic_auth, authorization, oauth2, or sigv4 parameter`, + errMsg: `authorization header must be changed via the basic_auth, authorization, oauth2, sigv4, or azuread parameter`, }, { filename: "remote_write_url_missing.bad.yml", diff --git a/docs/configuration/configuration.md b/docs/configuration/configuration.md index ff1449e34a..3a9ace2b6c 100644 --- a/docs/configuration/configuration.md +++ b/docs/configuration/configuration.md @@ -3466,7 +3466,7 @@ authorization: [ credentials_file: ] # Optionally configures AWS's Signature Verification 4 signing process to -# sign requests. Cannot be set at the same time as basic_auth, authorization, or oauth2. +# sign requests. Cannot be set at the same time as basic_auth, authorization, oauth2, or azuread. # To use the default credentials from the AWS SDK, use `sigv4: {}`. sigv4: # The AWS region. If blank, the region from the default credentials chain @@ -3485,10 +3485,20 @@ sigv4: [ role_arn: ] # Optional OAuth 2.0 configuration. -# Cannot be used at the same time as basic_auth, authorization, or sigv4. +# Cannot be used at the same time as basic_auth, authorization, sigv4, or azuread. oauth2: [ ] +# Optional AzureAD configuration. +# Cannot be used at the same time as basic_auth, authorization, oauth2, or sigv4. +azuread: + # The Azure Cloud. Options are 'AzurePublic', 'AzureChina', or 'AzureGovernment'. + [ cloud: | default = AzurePublic ] + + # Azure User-assigned Managed identity. + [ managed_identity: + [ client_id: ] + # Configures the remote write request's TLS settings. tls_config: [ ] diff --git a/go.mod b/go.mod index 9b826ad334..71f6a2b8db 100644 --- a/go.mod +++ b/go.mod @@ -4,6 +4,8 @@ go 1.19 require ( github.com/Azure/azure-sdk-for-go v65.0.0+incompatible + github.com/Azure/azure-sdk-for-go/sdk/azcore v1.3.1 + github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.1 github.com/Azure/go-autorest/autorest v0.11.28 github.com/Azure/go-autorest/autorest/adal v0.9.23 github.com/alecthomas/kingpin/v2 v2.3.2 @@ -83,10 +85,15 @@ require ( require ( cloud.google.com/go/compute/metadata v0.2.3 // indirect + github.com/Azure/azure-sdk-for-go/sdk/internal v1.1.2 // indirect + github.com/AzureAD/microsoft-authentication-library-for-go v0.8.1 // indirect github.com/coreos/go-systemd/v22 v22.5.0 // indirect github.com/hashicorp/errwrap v1.1.0 // indirect github.com/hashicorp/go-multierror v1.1.1 // indirect + github.com/kylelemons/godebug v1.1.0 // indirect + github.com/pkg/browser v0.0.0-20210115035449-ce105d075bb4 // indirect github.com/rogpeppe/go-internal v1.10.0 // indirect + github.com/stretchr/objx v0.5.0 // indirect github.com/xhit/go-str2duration/v2 v2.1.0 // indirect google.golang.org/genproto v0.0.0-20230526203410-71b5a4ffd15e // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20230526203410-71b5a4ffd15e // indirect @@ -135,7 +142,7 @@ require ( github.com/google/go-cmp v0.5.9 // indirect github.com/google/go-querystring v1.1.0 // indirect github.com/google/gofuzz v1.2.0 // indirect - github.com/google/uuid v1.3.0 // indirect + github.com/google/uuid v1.3.0 github.com/googleapis/enterprise-certificate-proxy v0.2.3 // indirect github.com/googleapis/gax-go/v2 v2.7.1 // indirect github.com/gorilla/websocket v1.5.0 // indirect diff --git a/go.sum b/go.sum index 3a472c99a0..859ce81ca2 100644 --- a/go.sum +++ b/go.sum @@ -38,6 +38,12 @@ cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9 dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU= github.com/Azure/azure-sdk-for-go v65.0.0+incompatible h1:HzKLt3kIwMm4KeJYTdx9EbjRYTySD/t8i1Ee/W5EGXw= github.com/Azure/azure-sdk-for-go v65.0.0+incompatible/go.mod h1:9XXNKU+eRnpl9moKnB4QOLf1HestfXbmab5FXxiDBjc= +github.com/Azure/azure-sdk-for-go/sdk/azcore v1.3.1 h1:gVXuXcWd1i4C2Ruxe321aU+IKGaStvGB/S90PUPB/W8= +github.com/Azure/azure-sdk-for-go/sdk/azcore v1.3.1/go.mod h1:DffdKW9RFqa5VgmsjUOsS7UE7eiA5iAvYUs63bhKQ0M= +github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.1 h1:T8quHYlUGyb/oqtSTwqlCr1ilJHrDv+ZtpSfo+hm1BU= +github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.1/go.mod h1:gLa1CL2RNE4s7M3yopJ/p0iq5DdY6Yv5ZUt9MTRZOQM= +github.com/Azure/azure-sdk-for-go/sdk/internal v1.1.2 h1:+5VZ72z0Qan5Bog5C+ZkgSqUbeVUd9wgtHOrIKuc5b8= +github.com/Azure/azure-sdk-for-go/sdk/internal v1.1.2/go.mod h1:eWRD7oawr1Mu1sLCawqVc0CUiF43ia3qQMxLscsKQ9w= github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 h1:UQHMgLO+TxOElx5B5HZ4hJQsoJ/PvUvKRhJHDQXO8P8= github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E= github.com/Azure/go-autorest v14.2.0+incompatible h1:V5VMDjClD3GiElqLWO7mz2MxNAK/vTfRHdAubSIPRgs= @@ -60,6 +66,8 @@ github.com/Azure/go-autorest/logger v0.2.1 h1:IG7i4p/mDa2Ce4TRyAO8IHnVhAVF3RFU+Z github.com/Azure/go-autorest/logger v0.2.1/go.mod h1:T9E3cAhj2VqvPOtCYAvby9aBXkZmbF5NWuPV8+WeEW8= github.com/Azure/go-autorest/tracing v0.6.0 h1:TYi4+3m5t6K48TGI9AUdb+IzbnSxvnvUMfuitfgcfuo= github.com/Azure/go-autorest/tracing v0.6.0/go.mod h1:+vhtPC754Xsa23ID7GlGsrdKBpUA79WCAKPPZVC2DeU= +github.com/AzureAD/microsoft-authentication-library-for-go v0.8.1 h1:oPdPEZFSbl7oSPEAIPMPBMUmiL+mqgzBJwM/9qYcwNg= +github.com/AzureAD/microsoft-authentication-library-for-go v0.8.1/go.mod h1:4qFor3D/HDsvBME35Xy9rwW9DecL+M2sNw1ybjPtwA0= github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo= github.com/DataDog/datadog-go v3.2.0+incompatible/go.mod h1:LButxg5PwREeZtORoXG3tL4fMGNddJ+vMq1mwgfaqoQ= @@ -515,6 +523,8 @@ github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc= +github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw= github.com/lightstep/lightstep-tracer-common/golang/gogo v0.0.0-20190605223551-bc2310a04743/go.mod h1:qklhhLq1aX+mtWk9cPHPzaBjWImj5ULL6C7HFJtXQMM= github.com/lightstep/lightstep-tracer-go v0.18.1/go.mod h1:jlF1pusYV4pidLvZ+XD0UBX0ZE6WURAspgAczcDHrL4= github.com/linode/linodego v1.16.1 h1:5otq57M4PdHycPERRfSFZ0s1yz1ETVWGjCp3hh7+F9w= @@ -630,6 +640,8 @@ github.com/pelletier/go-toml v1.7.0/go.mod h1:vwGMzjaWMwyfHwgIBhI2YUM4fB6nL6lVAv github.com/performancecopilot/speed v3.0.0+incompatible/go.mod h1:/CLtqpZ5gBg1M9iaPbIdPPGyKcA8hKdoy6hAWba7Yac= github.com/pierrec/lz4 v1.0.2-0.20190131084431-473cd7ce01a1/go.mod h1:3/3N9NVKO0jef7pBehbT1qWhCMrIgbYNnFAZCqQ5LRc= github.com/pierrec/lz4 v2.0.5+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi+IEE17M5jbnwPHcY= +github.com/pkg/browser v0.0.0-20210115035449-ce105d075bb4 h1:Qj1ukM4GlMWXNdMBuXcXfz/Kw9s1qm0CLY32QxuSImI= +github.com/pkg/browser v0.0.0-20210115035449-ce105d075bb4/go.mod h1:N6UoU20jOqggOuDwUaBQpluzLNDqif3kq9z2wpdYEfQ= github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= diff --git a/storage/remote/azuread/README.md b/storage/remote/azuread/README.md new file mode 100644 index 0000000000..b3b4457a6e --- /dev/null +++ b/storage/remote/azuread/README.md @@ -0,0 +1,8 @@ +azuread package +========================================= + +azuread provides an http.RoundTripper that attaches an Azure AD accessToken +to remote write requests. + +This module is considered internal to Prometheus, without any stability +guarantees for external usage. diff --git a/storage/remote/azuread/azuread.go b/storage/remote/azuread/azuread.go new file mode 100644 index 0000000000..94b6144da1 --- /dev/null +++ b/storage/remote/azuread/azuread.go @@ -0,0 +1,247 @@ +// Copyright 2023 The Prometheus Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package azuread + +import ( + "context" + "errors" + "fmt" + "net/http" + "strings" + "sync" + "time" + + "github.com/Azure/azure-sdk-for-go/sdk/azcore" + "github.com/Azure/azure-sdk-for-go/sdk/azcore/policy" + "github.com/Azure/azure-sdk-for-go/sdk/azidentity" + "github.com/google/uuid" +) + +const ( + // Clouds. + AzureChina = "AzureChina" + AzureGovernment = "AzureGovernment" + AzurePublic = "AzurePublic" + + // Audiences. + IngestionChinaAudience = "https://monitor.azure.cn//.default" + IngestionGovernmentAudience = "https://monitor.azure.us//.default" + IngestionPublicAudience = "https://monitor.azure.com//.default" +) + +// ManagedIdentityConfig is used to store managed identity config values +type ManagedIdentityConfig struct { + // ClientID is the clientId of the managed identity that is being used to authenticate. + ClientID string `yaml:"client_id,omitempty"` +} + +// AzureADConfig is used to store the config values. +type AzureADConfig struct { // nolint:revive + // ManagedIdentity is the managed identity that is being used to authenticate. + ManagedIdentity *ManagedIdentityConfig `yaml:"managed_identity,omitempty"` + + // Cloud is the Azure cloud in which the service is running. Example: AzurePublic/AzureGovernment/AzureChina. + Cloud string `yaml:"cloud,omitempty"` +} + +// azureADRoundTripper is used to store the roundtripper and the tokenprovider. +type azureADRoundTripper struct { + next http.RoundTripper + tokenProvider *tokenProvider +} + +// tokenProvider is used to store and retrieve Azure AD accessToken. +type tokenProvider struct { + // token is member used to store the current valid accessToken. + token string + // mu guards access to token. + mu sync.Mutex + // refreshTime is used to store the refresh time of the current valid accessToken. + refreshTime time.Time + // credentialClient is the Azure AD credential client that is being used to retrieve accessToken. + credentialClient azcore.TokenCredential + options *policy.TokenRequestOptions +} + +// Validate validates config values provided. +func (c *AzureADConfig) Validate() error { + if c.Cloud == "" { + c.Cloud = AzurePublic + } + + if c.Cloud != AzureChina && c.Cloud != AzureGovernment && c.Cloud != AzurePublic { + return fmt.Errorf("must provide a cloud in the Azure AD config") + } + + if c.ManagedIdentity == nil { + return fmt.Errorf("must provide an Azure Managed Identity in the Azure AD config") + } + + if c.ManagedIdentity.ClientID == "" { + return fmt.Errorf("must provide an Azure Managed Identity client_id in the Azure AD config") + } + + _, err := uuid.Parse(c.ManagedIdentity.ClientID) + if err != nil { + return fmt.Errorf("the provided Azure Managed Identity client_id provided is invalid") + } + return nil +} + +// UnmarshalYAML unmarshal the Azure AD config yaml. +func (c *AzureADConfig) UnmarshalYAML(unmarshal func(interface{}) error) error { + type plain AzureADConfig + *c = AzureADConfig{} + if err := unmarshal((*plain)(c)); err != nil { + return err + } + return c.Validate() +} + +// NewAzureADRoundTripper creates round tripper adding Azure AD authorization to calls. +func NewAzureADRoundTripper(cfg *AzureADConfig, next http.RoundTripper) (http.RoundTripper, error) { + if next == nil { + next = http.DefaultTransport + } + + cred, err := newTokenCredential(cfg) + if err != nil { + return nil, err + } + + tokenProvider, err := newTokenProvider(cfg, cred) + if err != nil { + return nil, err + } + + rt := &azureADRoundTripper{ + next: next, + tokenProvider: tokenProvider, + } + return rt, nil +} + +// RoundTrip sets Authorization header for requests. +func (rt *azureADRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) { + accessToken, err := rt.tokenProvider.getAccessToken(req.Context()) + if err != nil { + return nil, err + } + bearerAccessToken := "Bearer " + accessToken + req.Header.Set("Authorization", bearerAccessToken) + + return rt.next.RoundTrip(req) +} + +// newTokenCredential returns a TokenCredential of different kinds like Azure Managed Identity and Azure AD application. +func newTokenCredential(cfg *AzureADConfig) (azcore.TokenCredential, error) { + cred, err := newManagedIdentityTokenCredential(cfg.ManagedIdentity.ClientID) + if err != nil { + return nil, err + } + + return cred, nil +} + +// newManagedIdentityTokenCredential returns new Managed Identity token credential. +func newManagedIdentityTokenCredential(managedIdentityClientID string) (azcore.TokenCredential, error) { + clientID := azidentity.ClientID(managedIdentityClientID) + opts := &azidentity.ManagedIdentityCredentialOptions{ID: clientID} + return azidentity.NewManagedIdentityCredential(opts) +} + +// newTokenProvider helps to fetch accessToken for different types of credential. This also takes care of +// refreshing the accessToken before expiry. This accessToken is attached to the Authorization header while making requests. +func newTokenProvider(cfg *AzureADConfig, cred azcore.TokenCredential) (*tokenProvider, error) { + audience, err := getAudience(cfg.Cloud) + if err != nil { + return nil, err + } + + tokenProvider := &tokenProvider{ + credentialClient: cred, + options: &policy.TokenRequestOptions{Scopes: []string{audience}}, + } + + return tokenProvider, nil +} + +// getAccessToken returns the current valid accessToken. +func (tokenProvider *tokenProvider) getAccessToken(ctx context.Context) (string, error) { + tokenProvider.mu.Lock() + defer tokenProvider.mu.Unlock() + if tokenProvider.valid() { + return tokenProvider.token, nil + } + err := tokenProvider.getToken(ctx) + if err != nil { + return "", errors.New("Failed to get access token: " + err.Error()) + } + return tokenProvider.token, nil +} + +// valid checks if the token in the token provider is valid and not expired. +func (tokenProvider *tokenProvider) valid() bool { + if len(tokenProvider.token) == 0 { + return false + } + if tokenProvider.refreshTime.After(time.Now().UTC()) { + return true + } + return false +} + +// getToken retrieves a new accessToken and stores the newly retrieved token in the tokenProvider. +func (tokenProvider *tokenProvider) getToken(ctx context.Context) error { + accessToken, err := tokenProvider.credentialClient.GetToken(ctx, *tokenProvider.options) + if err != nil { + return err + } + if len(accessToken.Token) == 0 { + return errors.New("access token is empty") + } + + tokenProvider.token = accessToken.Token + err = tokenProvider.updateRefreshTime(accessToken) + if err != nil { + return err + } + return nil +} + +// updateRefreshTime handles logic to set refreshTime. The refreshTime is set at half the duration of the actual token expiry. +func (tokenProvider *tokenProvider) updateRefreshTime(accessToken azcore.AccessToken) error { + tokenExpiryTimestamp := accessToken.ExpiresOn.UTC() + deltaExpirytime := time.Now().Add(time.Until(tokenExpiryTimestamp) / 2) + if deltaExpirytime.After(time.Now().UTC()) { + tokenProvider.refreshTime = deltaExpirytime + } else { + return errors.New("access token expiry is less than the current time") + } + return nil +} + +// getAudience returns audiences for different clouds. +func getAudience(cloud string) (string, error) { + switch strings.ToLower(cloud) { + case strings.ToLower(AzureChina): + return IngestionChinaAudience, nil + case strings.ToLower(AzureGovernment): + return IngestionGovernmentAudience, nil + case strings.ToLower(AzurePublic): + return IngestionPublicAudience, nil + default: + return "", errors.New("Cloud is not specified or is incorrect: " + cloud) + } +} diff --git a/storage/remote/azuread/azuread_test.go b/storage/remote/azuread/azuread_test.go new file mode 100644 index 0000000000..ebbd98958c --- /dev/null +++ b/storage/remote/azuread/azuread_test.go @@ -0,0 +1,252 @@ +// Copyright 2023 The Prometheus Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package azuread + +import ( + "context" + "net/http" + "os" + "strings" + "testing" + "time" + + "github.com/Azure/azure-sdk-for-go/sdk/azcore" + "github.com/Azure/azure-sdk-for-go/sdk/azcore/policy" + "github.com/google/uuid" + "github.com/prometheus/client_golang/prometheus/promhttp" + "github.com/stretchr/testify/mock" + "github.com/stretchr/testify/suite" + "gopkg.in/yaml.v2" +) + +const ( + dummyAudience = "dummyAudience" + dummyClientID = "00000000-0000-0000-0000-000000000000" + testTokenString = "testTokenString" +) + +var testTokenExpiry = time.Now().Add(10 * time.Second) + +type AzureAdTestSuite struct { + suite.Suite + mockCredential *mockCredential +} + +type TokenProviderTestSuite struct { + suite.Suite + mockCredential *mockCredential +} + +// mockCredential mocks azidentity TokenCredential interface. +type mockCredential struct { + mock.Mock +} + +func (ad *AzureAdTestSuite) BeforeTest(_, _ string) { + ad.mockCredential = new(mockCredential) +} + +func TestAzureAd(t *testing.T) { + suite.Run(t, new(AzureAdTestSuite)) +} + +func (ad *AzureAdTestSuite) TestAzureAdRoundTripper() { + var gotReq *http.Request + + testToken := &azcore.AccessToken{ + Token: testTokenString, + ExpiresOn: testTokenExpiry, + } + + managedIdentityConfig := &ManagedIdentityConfig{ + ClientID: dummyClientID, + } + + azureAdConfig := &AzureADConfig{ + Cloud: "AzurePublic", + ManagedIdentity: managedIdentityConfig, + } + + ad.mockCredential.On("GetToken", mock.Anything, mock.Anything).Return(*testToken, nil) + + tokenProvider, err := newTokenProvider(azureAdConfig, ad.mockCredential) + ad.Assert().NoError(err) + + rt := &azureADRoundTripper{ + next: promhttp.RoundTripperFunc(func(req *http.Request) (*http.Response, error) { + gotReq = req + return &http.Response{StatusCode: http.StatusOK}, nil + }), + tokenProvider: tokenProvider, + } + + cli := &http.Client{Transport: rt} + + req, err := http.NewRequest(http.MethodPost, "https://example.com", strings.NewReader("Hello, world!")) + ad.Assert().NoError(err) + + _, err = cli.Do(req) + ad.Assert().NoError(err) + ad.Assert().NotNil(gotReq) + + origReq := gotReq + ad.Assert().NotEmpty(origReq.Header.Get("Authorization")) + ad.Assert().Equal("Bearer "+testTokenString, origReq.Header.Get("Authorization")) +} + +func loadAzureAdConfig(filename string) (*AzureADConfig, error) { + content, err := os.ReadFile(filename) + if err != nil { + return nil, err + } + cfg := AzureADConfig{} + if err = yaml.UnmarshalStrict(content, &cfg); err != nil { + return nil, err + } + return &cfg, nil +} + +func testGoodConfig(t *testing.T, filename string) { + _, err := loadAzureAdConfig(filename) + if err != nil { + t.Fatalf("Unexpected error parsing %s: %s", filename, err) + } +} + +func TestGoodAzureAdConfig(t *testing.T) { + filename := "testdata/azuread_good.yaml" + testGoodConfig(t, filename) +} + +func TestGoodCloudMissingAzureAdConfig(t *testing.T) { + filename := "testdata/azuread_good_cloudmissing.yaml" + testGoodConfig(t, filename) +} + +func TestBadClientIdMissingAzureAdConfig(t *testing.T) { + filename := "testdata/azuread_bad_clientidmissing.yaml" + _, err := loadAzureAdConfig(filename) + if err == nil { + t.Fatalf("Did not receive expected error unmarshaling bad azuread config") + } + if !strings.Contains(err.Error(), "must provide an Azure Managed Identity in the Azure AD config") { + t.Errorf("Received unexpected error from unmarshal of %s: %s", filename, err.Error()) + } +} + +func TestBadInvalidClientIdAzureAdConfig(t *testing.T) { + filename := "testdata/azuread_bad_invalidclientid.yaml" + _, err := loadAzureAdConfig(filename) + if err == nil { + t.Fatalf("Did not receive expected error unmarshaling bad azuread config") + } + if !strings.Contains(err.Error(), "the provided Azure Managed Identity client_id provided is invalid") { + t.Errorf("Received unexpected error from unmarshal of %s: %s", filename, err.Error()) + } +} + +func (m *mockCredential) GetToken(ctx context.Context, options policy.TokenRequestOptions) (azcore.AccessToken, error) { + args := m.MethodCalled("GetToken", ctx, options) + if args.Get(0) == nil { + return azcore.AccessToken{}, args.Error(1) + } + + return args.Get(0).(azcore.AccessToken), nil +} + +func (s *TokenProviderTestSuite) BeforeTest(_, _ string) { + s.mockCredential = new(mockCredential) +} + +func TestTokenProvider(t *testing.T) { + suite.Run(t, new(TokenProviderTestSuite)) +} + +func (s *TokenProviderTestSuite) TestNewTokenProvider_NilAudience_Fail() { + managedIdentityConfig := &ManagedIdentityConfig{ + ClientID: dummyClientID, + } + + azureAdConfig := &AzureADConfig{ + Cloud: "PublicAzure", + ManagedIdentity: managedIdentityConfig, + } + + actualTokenProvider, actualErr := newTokenProvider(azureAdConfig, s.mockCredential) + + s.Assert().Nil(actualTokenProvider) + s.Assert().NotNil(actualErr) + s.Assert().Equal("Cloud is not specified or is incorrect: "+azureAdConfig.Cloud, actualErr.Error()) +} + +func (s *TokenProviderTestSuite) TestNewTokenProvider_Success() { + managedIdentityConfig := &ManagedIdentityConfig{ + ClientID: dummyClientID, + } + + azureAdConfig := &AzureADConfig{ + Cloud: "AzurePublic", + ManagedIdentity: managedIdentityConfig, + } + s.mockCredential.On("GetToken", mock.Anything, mock.Anything).Return(getToken(), nil) + + actualTokenProvider, actualErr := newTokenProvider(azureAdConfig, s.mockCredential) + + s.Assert().NotNil(actualTokenProvider) + s.Assert().Nil(actualErr) + s.Assert().NotNil(actualTokenProvider.getAccessToken(context.Background())) +} + +func (s *TokenProviderTestSuite) TestPeriodicTokenRefresh_Success() { + // setup + managedIdentityConfig := &ManagedIdentityConfig{ + ClientID: dummyClientID, + } + + azureAdConfig := &AzureADConfig{ + Cloud: "AzurePublic", + ManagedIdentity: managedIdentityConfig, + } + testToken := &azcore.AccessToken{ + Token: testTokenString, + ExpiresOn: testTokenExpiry, + } + + s.mockCredential.On("GetToken", mock.Anything, mock.Anything).Return(*testToken, nil).Once(). + On("GetToken", mock.Anything, mock.Anything).Return(getToken(), nil) + + actualTokenProvider, actualErr := newTokenProvider(azureAdConfig, s.mockCredential) + + s.Assert().NotNil(actualTokenProvider) + s.Assert().Nil(actualErr) + s.Assert().NotNil(actualTokenProvider.getAccessToken(context.Background())) + + // Token set to refresh at half of the expiry time. The test tokens are set to expiry in 10s. + // Hence, the 6 seconds wait to check if the token is refreshed. + time.Sleep(6 * time.Second) + + s.Assert().NotNil(actualTokenProvider.getAccessToken(context.Background())) + + s.mockCredential.AssertNumberOfCalls(s.T(), "GetToken", 2) + accessToken, err := actualTokenProvider.getAccessToken(context.Background()) + s.Assert().Nil(err) + s.Assert().NotEqual(accessToken, testTokenString) +} + +func getToken() azcore.AccessToken { + return azcore.AccessToken{ + Token: uuid.New().String(), + ExpiresOn: time.Now().Add(10 * time.Second), + } +} diff --git a/storage/remote/azuread/testdata/azuread_bad_clientidmissing.yaml b/storage/remote/azuread/testdata/azuread_bad_clientidmissing.yaml new file mode 100644 index 0000000000..68b119cd42 --- /dev/null +++ b/storage/remote/azuread/testdata/azuread_bad_clientidmissing.yaml @@ -0,0 +1 @@ +cloud: AzurePublic diff --git a/storage/remote/azuread/testdata/azuread_bad_invalidclientid.yaml b/storage/remote/azuread/testdata/azuread_bad_invalidclientid.yaml new file mode 100644 index 0000000000..1f72fbb71f --- /dev/null +++ b/storage/remote/azuread/testdata/azuread_bad_invalidclientid.yaml @@ -0,0 +1,3 @@ +cloud: AzurePublic +managed_identity: + client_id: foo-foobar-bar-foo-00000000 diff --git a/storage/remote/azuread/testdata/azuread_good.yaml b/storage/remote/azuread/testdata/azuread_good.yaml new file mode 100644 index 0000000000..de39f0a060 --- /dev/null +++ b/storage/remote/azuread/testdata/azuread_good.yaml @@ -0,0 +1,3 @@ +cloud: AzurePublic +managed_identity: + client_id: 00000000-0000-0000-0000-000000000000 diff --git a/storage/remote/azuread/testdata/azuread_good_cloudmissing.yaml b/storage/remote/azuread/testdata/azuread_good_cloudmissing.yaml new file mode 100644 index 0000000000..bef6318743 --- /dev/null +++ b/storage/remote/azuread/testdata/azuread_good_cloudmissing.yaml @@ -0,0 +1,2 @@ +managed_identity: + client_id: 00000000-0000-0000-0000-000000000000 diff --git a/storage/remote/client.go b/storage/remote/client.go index 1625c9918d..33774203c5 100644 --- a/storage/remote/client.go +++ b/storage/remote/client.go @@ -36,6 +36,7 @@ import ( "go.opentelemetry.io/otel/trace" "github.com/prometheus/prometheus/prompb" + "github.com/prometheus/prometheus/storage/remote/azuread" ) const maxErrMsgLen = 1024 @@ -97,6 +98,7 @@ type ClientConfig struct { Timeout model.Duration HTTPClientConfig config_util.HTTPClientConfig SigV4Config *sigv4.SigV4Config + AzureADConfig *azuread.AzureADConfig Headers map[string]string RetryOnRateLimit bool } @@ -146,6 +148,13 @@ func NewWriteClient(name string, conf *ClientConfig) (WriteClient, error) { } } + if conf.AzureADConfig != nil { + t, err = azuread.NewAzureADRoundTripper(conf.AzureADConfig, httpClient.Transport) + if err != nil { + return nil, err + } + } + if len(conf.Headers) > 0 { t = newInjectHeadersRoundTripper(conf.Headers, t) } diff --git a/storage/remote/write.go b/storage/remote/write.go index 6a33c0adf9..4b0a249014 100644 --- a/storage/remote/write.go +++ b/storage/remote/write.go @@ -158,6 +158,7 @@ func (rws *WriteStorage) ApplyConfig(conf *config.Config) error { Timeout: rwConf.RemoteTimeout, HTTPClientConfig: rwConf.HTTPClientConfig, SigV4Config: rwConf.SigV4Config, + AzureADConfig: rwConf.AzureADConfig, Headers: rwConf.Headers, RetryOnRateLimit: rwConf.QueueConfig.RetryOnRateLimit, }) From 084f38df12fe7c5693f7b0e7164cdb1570a525b7 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 1 Jun 2023 23:57:03 +0000 Subject: [PATCH 196/231] build(deps): bump bufbuild/buf-setup-action from 1.17.0 to 1.20.0 Bumps [bufbuild/buf-setup-action](https://github.com/bufbuild/buf-setup-action) from 1.17.0 to 1.20.0. - [Release notes](https://github.com/bufbuild/buf-setup-action/releases) - [Commits](https://github.com/bufbuild/buf-setup-action/compare/v1.17.0...v1.20.0) --- updated-dependencies: - dependency-name: bufbuild/buf-setup-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- .github/workflows/buf-lint.yml | 2 +- .github/workflows/buf.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/buf-lint.yml b/.github/workflows/buf-lint.yml index 79430ee56a..a72837b79e 100644 --- a/.github/workflows/buf-lint.yml +++ b/.github/workflows/buf-lint.yml @@ -10,7 +10,7 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - - uses: bufbuild/buf-setup-action@v1.17.0 + - uses: bufbuild/buf-setup-action@v1.20.0 with: github_token: ${{ secrets.GITHUB_TOKEN }} - uses: bufbuild/buf-lint-action@v1 diff --git a/.github/workflows/buf.yml b/.github/workflows/buf.yml index 06e53172e8..b63f073fa5 100644 --- a/.github/workflows/buf.yml +++ b/.github/workflows/buf.yml @@ -9,7 +9,7 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - - uses: bufbuild/buf-setup-action@v1.17.0 + - uses: bufbuild/buf-setup-action@v1.20.0 with: github_token: ${{ secrets.GITHUB_TOKEN }} - uses: bufbuild/buf-lint-action@v1 From 90816aa39ecc7e9017a4de653615c461f1516f08 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 2 Jun 2023 00:01:29 +0000 Subject: [PATCH 197/231] build(deps): bump github.com/prometheus/prometheus Bumps [github.com/prometheus/prometheus](https://github.com/prometheus/prometheus) from 0.43.0 to 0.44.0. - [Release notes](https://github.com/prometheus/prometheus/releases) - [Changelog](https://github.com/prometheus/prometheus/blob/main/CHANGELOG.md) - [Commits](https://github.com/prometheus/prometheus/compare/v0.43.0...v0.44.0) --- updated-dependencies: - dependency-name: github.com/prometheus/prometheus dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- documentation/examples/remote_storage/go.mod | 14 ++--- documentation/examples/remote_storage/go.sum | 60 ++++++++++---------- 2 files changed, 37 insertions(+), 37 deletions(-) diff --git a/documentation/examples/remote_storage/go.mod b/documentation/examples/remote_storage/go.mod index 1db3e9a9e4..c0d4331968 100644 --- a/documentation/examples/remote_storage/go.mod +++ b/documentation/examples/remote_storage/go.mod @@ -10,13 +10,13 @@ require ( github.com/influxdata/influxdb v1.11.0 github.com/prometheus/client_golang v1.15.0 github.com/prometheus/common v0.42.0 - github.com/prometheus/prometheus v0.43.0 + github.com/prometheus/prometheus v0.44.0 github.com/stretchr/testify v1.8.2 ) require ( github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 // indirect - github.com/aws/aws-sdk-go v1.44.217 // indirect + github.com/aws/aws-sdk-go v1.44.245 // indirect github.com/beorn7/perks v1.0.1 // indirect github.com/cespare/xxhash/v2 v2.2.0 // indirect github.com/davecgh/go-spew v1.1.1 // indirect @@ -44,11 +44,11 @@ require ( go.opentelemetry.io/otel/trace v1.14.0 // indirect go.uber.org/atomic v1.10.0 // indirect go.uber.org/goleak v1.2.1 // indirect - golang.org/x/exp v0.0.0-20230307190834-24139beb5833 // indirect - golang.org/x/net v0.8.0 // indirect - golang.org/x/oauth2 v0.6.0 // indirect - golang.org/x/sys v0.6.0 // indirect - golang.org/x/text v0.8.0 // indirect + golang.org/x/exp v0.0.0-20230321023759-10a507213a29 // indirect + golang.org/x/net v0.9.0 // indirect + golang.org/x/oauth2 v0.7.0 // indirect + golang.org/x/sys v0.7.0 // indirect + golang.org/x/text v0.9.0 // indirect golang.org/x/time v0.3.0 // indirect google.golang.org/appengine v1.6.7 // indirect google.golang.org/protobuf v1.30.0 // indirect diff --git a/documentation/examples/remote_storage/go.sum b/documentation/examples/remote_storage/go.sum index c4350e78b0..e0eac05c10 100644 --- a/documentation/examples/remote_storage/go.sum +++ b/documentation/examples/remote_storage/go.sum @@ -1,7 +1,7 @@ github.com/Azure/azure-sdk-for-go v65.0.0+incompatible h1:HzKLt3kIwMm4KeJYTdx9EbjRYTySD/t8i1Ee/W5EGXw= github.com/Azure/go-autorest v14.2.0+incompatible h1:V5VMDjClD3GiElqLWO7mz2MxNAK/vTfRHdAubSIPRgs= github.com/Azure/go-autorest/autorest v0.11.28 h1:ndAExarwr5Y+GaHE6VCaY1kyS/HwwGGyuimVhWsHOEM= -github.com/Azure/go-autorest/autorest/adal v0.9.22 h1:/GblQdIudfEM3AWWZ0mrYJQSd7JS4S/Mbzh6F0ov0Xc= +github.com/Azure/go-autorest/autorest/adal v0.9.23 h1:Yepx8CvFxwNKpH6ja7RZ+sKX+DWYNldbLiALMC3BTz8= github.com/Azure/go-autorest/autorest/date v0.3.0 h1:7gUk1U5M/CQbp9WoqinNzJar+8KY+LPI6wiWrP/myHw= github.com/Azure/go-autorest/autorest/to v0.4.0 h1:oXVqrxakqqV1UZdSazDOPOLvOIz+XA683u8EctwboHk= github.com/Azure/go-autorest/autorest/validation v0.3.1 h1:AgyqjAd94fwNAoTjl/WQXg4VvFeRFpO+UhNyRXqF1ac= @@ -19,8 +19,8 @@ github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 h1:s6gZFSlWYmbqAu github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137/go.mod h1:OMCwj8VM1Kc9e19TLln2VL61YJF0x1XFtfdL4JdbSyE= github.com/armon/go-metrics v0.4.1 h1:hR91U9KYmb6bLBYLQjyM+3j+rcd/UhE+G78SFnF8gJA= github.com/aws/aws-sdk-go v1.38.35/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro= -github.com/aws/aws-sdk-go v1.44.217 h1:FcWC56MRl+k756aH3qeMQTylSdeJ58WN0iFz3fkyRz0= -github.com/aws/aws-sdk-go v1.44.217/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI= +github.com/aws/aws-sdk-go v1.44.245 h1:KtY2s4q31/kn33AdV63R5t77mdxsI7rq3YT7Mgo805M= +github.com/aws/aws-sdk-go v1.44.245/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI= github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8= github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= @@ -35,15 +35,15 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/dennwc/varint v1.0.0 h1:kGNFFSSw8ToIy3obO/kKr8U9GZYUAxQEVuix4zfDWzE= github.com/dennwc/varint v1.0.0/go.mod h1:hnItb35rvZvJrbTALZtY/iQfDs48JKRG1RPpgziApxA= -github.com/digitalocean/godo v1.97.0 h1:p9w1yCcWMZcxFSLPToNGXA96WfUVLXqoHti6GzVomL4= +github.com/digitalocean/godo v1.98.0 h1:potyC1eD0N9n5/P4/WmJuKgg+OGYZOBWEW+/aKTX6QQ= github.com/docker/distribution v2.8.1+incompatible h1:Q50tZOPR6T/hjNsyc9g8/syEs6bk8XXApsHjKukMl68= -github.com/docker/docker v23.0.1+incompatible h1:vjgvJZxprTTE1A37nm+CLNAdwu6xZekyoiVlUZEINcY= +github.com/docker/docker v23.0.4+incompatible h1:Kd3Bh9V/rO+XpTP/BLqM+gx8z7+Yb0AA2Ibj+nNo4ek= github.com/docker/go-connections v0.4.0 h1:El9xVISelRB7BuFusrZozjnkIM5YnzCViNKohAFqRJQ= github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4= github.com/edsrzf/mmap-go v1.1.0 h1:6EUwBLQ/Mcr1EYLE4Tn1VdW1A4ckqCQWZBw8Hr0kjpQ= github.com/emicklei/go-restful/v3 v3.10.1 h1:rc42Y5YTp7Am7CS630D7JmhRjq4UlEUuEKfrDac4bSQ= github.com/envoyproxy/go-control-plane v0.11.0 h1:jtLewhRR2vMRNnq2ZZUoCjUlgut+Y0+sDDWPOfwOi1o= -github.com/envoyproxy/protoc-gen-validate v0.9.1 h1:PS7VIOgmSVhWUEeZwTe7z7zouA22Cr590PzXKbZHOVY= +github.com/envoyproxy/protoc-gen-validate v0.10.1 h1:c0g45+xCJhdgFGw7a5QAfdS4byAbud7miNWJ1WwEVf8= github.com/fatih/color v1.14.1 h1:qfhVLaG5s+nCROl1zJsZRxFeYrHLqWroPOQ8BWiNb4w= github.com/felixge/httpsnoop v1.0.3 h1:s/nj+GCswXYzN5v2DpNMuMQYe+0DDwt5WVCU6CWBdXk= github.com/felixge/httpsnoop v1.0.3/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U= @@ -100,7 +100,7 @@ github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38= github.com/google/go-querystring v1.1.0 h1:AnCroh3fv4ZBgVIf1Iwtovgjaw/GiKJo8M8yD/fhyJ8= github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0= -github.com/gophercloud/gophercloud v1.2.0 h1:1oXyj4g54KBg/kFtCdMM6jtxSzeIyg8wv4z1HoGPp1E= +github.com/gophercloud/gophercloud v1.3.0 h1:RUKyCMiZoQR3VlVR5E3K7PK1AC3/qppsWYo6dtBiqs8= github.com/gorilla/websocket v1.5.0 h1:PPwGk2jz7EePpoHN/+ClbZu8SPxiqlu12wZP/3sWmnc= github.com/grafana/regexp v0.0.0-20221122212121-6b5c0a4cb7fd h1:PpuIBO5P3e9hpqBD0O/HjhShYuM6XE0i/lbE6J94kww= github.com/grafana/regexp v0.0.0-20221122212121-6b5c0a4cb7fd/go.mod h1:M5qHK+eWfAv8VR/265dIuEpL3fNfeC21tXXp9itM24A= @@ -114,13 +114,13 @@ github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+l github.com/hashicorp/go-retryablehttp v0.7.2 h1:AcYqCvkpalPnPF2pn0KamgwamS42TqUDDYFRKq/RAd0= github.com/hashicorp/go-rootcerts v1.0.2 h1:jzhAVGtqPKbwpyCPELlgNWhE1znq+qwJtW5Oi2viEzc= github.com/hashicorp/golang-lru v0.6.0 h1:uL2shRDx7RTrOrTCUZEGP/wJUFiUI8QT6E7z5o8jga4= -github.com/hashicorp/nomad/api v0.0.0-20230308192510-48e7d70fcd4b h1:EkuSTU8c/63q4LMayj8ilgg/4I5PXDFVcnqKfs9qcwI= +github.com/hashicorp/nomad/api v0.0.0-20230418003350-3067191c5197 h1:I5xhKLePXpXgM6pZ4xZNTiurLLS3sGuZrZFFzAbM67A= github.com/hashicorp/serf v0.10.1 h1:Z1H2J60yRKvfDYAOZLd2MU0ND4AH/WDz7xYHDWQsIPY= -github.com/hetznercloud/hcloud-go v1.41.0 h1:KJGFRRc68QiVu4PrEP5BmCQVveCP2CM26UGQUKGpIUs= +github.com/hetznercloud/hcloud-go v1.42.0 h1:Es/CDOForQN3nOOP5Vxh1N/YHjpCg386iYEX5zCgi+A= github.com/imdario/mergo v0.3.13 h1:lFzP57bqS/wsqKssCGmtLAb8A0wKjLGrve2q3PPVcBk= github.com/influxdata/influxdb v1.11.0 h1:0X+ZsbcOWc6AEi5MHee9BYqXCKmz8IZsljrRYjmV8Qg= github.com/influxdata/influxdb v1.11.0/go.mod h1:V93tJcidY0Zh0LtSONZWnXXGDyt20dtVf+Ddp4EnhaA= -github.com/ionos-cloud/sdk-go/v6 v6.1.4 h1:BJHhFA8Q1SZC7VOXqKKr2BV2ysQ2/4hlk1e4hZte7GY= +github.com/ionos-cloud/sdk-go/v6 v6.1.6 h1:0n4irdqNska+1s3YMCRhrAqKbibEgQ7SwwhAlHzYT5A= github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg= github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo= github.com/jmespath/go-jmespath/internal/testify v1.5.1 h1:shLQSRRSCCPj3f2gpwzGwWFoC7ycTf1rcQZHOlsJ6N8= @@ -146,14 +146,14 @@ github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= -github.com/linode/linodego v1.14.1 h1:uGxQyy0BidoEpLGdvfi4cPgEW+0YUFsEGrLEhcTfjNc= +github.com/linode/linodego v1.16.1 h1:5otq57M4PdHycPERRfSFZ0s1yz1ETVWGjCp3hh7+F9w= github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0= github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA= github.com/mattn/go-isatty v0.0.17 h1:BTarxUcIeDqL27Mc+vyvdWYSL28zpIhv3RoTdsLMPng= github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0= github.com/matttproud/golang_protobuf_extensions v1.0.4 h1:mmDVorXM7PCGKw94cs5zkfA9PSy5pEvNWRP0ET0TIVo= github.com/matttproud/golang_protobuf_extensions v1.0.4/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4= -github.com/miekg/dns v1.1.51 h1:0+Xg7vObnhrz/4ZCZcZh7zPXlmU0aveS2HDBd0m0qSo= +github.com/miekg/dns v1.1.53 h1:ZBkuHr5dxHtB1caEOlZTLPo7D3L3TWckgUUs/RHfDxw= github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y= github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= @@ -169,7 +169,7 @@ github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f/go.mod h1:qRW github.com/oklog/ulid v1.3.1 h1:EGfNDEx6MqHz8B3uNV6QAib1UR2Lm97sHi3ocA6ESJ4= github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U= github.com/opencontainers/image-spec v1.0.2 h1:9yCKha/T5XdGtO0q9Q9a6T5NUCsTn/DrBg0D7ufOcFM= -github.com/ovh/go-ovh v1.3.0 h1:mvZaddk4E4kLcXhzb+cxBsMPYp2pHqiQpWYkInsuZPQ= +github.com/ovh/go-ovh v1.4.1 h1:VBGa5wMyQtTP7Zb+w97zRCh9sLtM/2YKRyy+MEJmWaM= github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= @@ -201,10 +201,10 @@ github.com/prometheus/procfs v0.1.3/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4O github.com/prometheus/procfs v0.6.0/go.mod h1:cz+aTbrPOrUb4q7XlbU9ygM+/jj0fzG6c1xBZuNvfVA= github.com/prometheus/procfs v0.9.0 h1:wzCHvIvM5SxWqYvwgVL7yJY8Lz3PKn49KQtpgMYJfhI= github.com/prometheus/procfs v0.9.0/go.mod h1:+pB4zwohETzFnmlpe6yd2lSc+0/46IYZRB/chUwxUZY= -github.com/prometheus/prometheus v0.43.0 h1:18iCSfrbAHbXvYFvR38U1Pt4uZmU9SmDcCpCrBKUiGg= -github.com/prometheus/prometheus v0.43.0/go.mod h1:2BA14LgBeqlPuzObSEbh+Y+JwLH2GcqDlJKbF2sA6FM= -github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8= -github.com/scaleway/scaleway-sdk-go v1.0.0-beta.14 h1:yFl3jyaSVLNYXlnNYM5z2pagEk1dYQhfr1p20T1NyKY= +github.com/prometheus/prometheus v0.44.0 h1:sgn8Fdx+uE5tHQn0/622swlk2XnIj6udoZCnbVjHIgc= +github.com/prometheus/prometheus v0.44.0/go.mod h1:aPsmIK3py5XammeTguyqTmuqzX/jeCdyOWWobLHNKQg= +github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= +github.com/scaleway/scaleway-sdk-go v1.0.0-beta.15 h1:Y7xOFbD+3jaPw+VN7lkakNJ/pa+ZSQVFp1ONtJaBxns= github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo= github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE= github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88= @@ -245,12 +245,12 @@ golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8U golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.7.0 h1:AvwMYaRytfdeVt3u6mLaxYtErKYjxA2OXjJ1HHq6t3A= -golang.org/x/exp v0.0.0-20230307190834-24139beb5833 h1:SChBja7BCQewoTAU7IgvucQKMIXrEpFxNMs0spT3/5s= -golang.org/x/exp v0.0.0-20230307190834-24139beb5833/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc= +golang.org/x/exp v0.0.0-20230321023759-10a507213a29 h1:ooxPy7fPvB4kwsA2h+iBNHkAbp/4JxTSwCmvdjEYmug= +golang.org/x/exp v0.0.0-20230321023759-10a507213a29/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= -golang.org/x/mod v0.9.0 h1:KENHtAZL2y3NLMYZeHY9DW8HW8V+kQyJsY/V9JlKvCs= +golang.org/x/mod v0.10.0 h1:lFO9qtOdlre5W1jxS3r/4szv2/6iXxScdzjoBMXNhYk= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -267,12 +267,12 @@ golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.1.0/go.mod h1:Cx3nUiGt4eDBEyega/BKRp+/AlGL8hYe7U9odMt2Cco= -golang.org/x/net v0.8.0 h1:Zrh2ngAOFYneWTAIAPethzeaQLuHwhuBkuV6ZiRnUaQ= -golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc= +golang.org/x/net v0.9.0 h1:aWJ/m6xSmxWBx+V0XRHTlrYrPG56jKsLdTFmsSsCzOM= +golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.6.0 h1:Lh8GPgSKBfWSwFvtuWOfeI3aAAnbXTSutYxJiOJFgIw= -golang.org/x/oauth2 v0.6.0/go.mod h1:ycmewcwgD4Rpr3eZJLSB4Kyyljb3qDh40vJ8STE5HKw= +golang.org/x/oauth2 v0.7.0 h1:qe6s0zUXlPX80/dITx3440hWZ7GwMwgDDyrSGTPJG/g= +golang.org/x/oauth2 v0.7.0/go.mod h1:hPLQkd9LyjfXTiRohC/41GhcFqxisoUQ99sCUOHO9x4= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -299,20 +299,20 @@ golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.6.0 h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ= -golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.7.0 h1:3jlCCIQZPdOYu1h8BkNvLz8Kgwtae2cagcG/VamtZRU= +golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.1.0/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= -golang.org/x/term v0.6.0 h1:clScbb1cHjoCkyRbWwBEUZ5H/tIFu5TAXIqaZD0Gcjw= +golang.org/x/term v0.7.0 h1:BEvjmm5fURWqcfbSKTdpkDXYBrUS1c0m8agp14W48vQ= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= -golang.org/x/text v0.8.0 h1:57P1ETyNKtuIjB4SRd15iJxuhj8Gc416Y78H3qgMh68= -golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= +golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE= +golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4= golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= @@ -320,7 +320,7 @@ golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtn golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= -golang.org/x/tools v0.7.0 h1:W4OVu8VVOaIO0yzWMNdepAulS7YfoS3Zabrm8DOXXU4= +golang.org/x/tools v0.8.0 h1:vSDcovVPld282ceKgDimkRSC8kpaH1dgyc9UMzlt84Y= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= From ac8abdaacda605bdcd358ec4b6d30e634a0a569a Mon Sep 17 00:00:00 2001 From: Julius Volz Date: Mon, 5 Jun 2023 17:36:11 +0200 Subject: [PATCH 198/231] Rename remaining jitterSeed -> offsetSeed variables (#12414) I had changed the naming from "jitter" to "offset" in: https://github.com/prometheus/prometheus/commit/cb045c0e4b94bbf3eee174d91b5ef2b8553948d5 ...but I forgot to add this file to the commit to complete the renaming, doing that now. Signed-off-by: Julius Volz --- scrape/target.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/scrape/target.go b/scrape/target.go index a655e85413..8b745a9c49 100644 --- a/scrape/target.go +++ b/scrape/target.go @@ -154,14 +154,14 @@ func (t *Target) hash() uint64 { } // offset returns the time until the next scrape cycle for the target. -// It includes the global server jitterSeed for scrapes from multiple Prometheus to try to be at different times. -func (t *Target) offset(interval time.Duration, jitterSeed uint64) time.Duration { +// It includes the global server offsetSeed for scrapes from multiple Prometheus to try to be at different times. +func (t *Target) offset(interval time.Duration, offsetSeed uint64) time.Duration { now := time.Now().UnixNano() // Base is a pinned to absolute time, no matter how often offset is called. var ( base = int64(interval) - now%int64(interval) - offset = (t.hash() ^ jitterSeed) % uint64(interval) + offset = (t.hash() ^ offsetSeed) % uint64(interval) next = base + int64(offset) ) From bfa466d00f152aa8dd58494c171872d7f95e3d5e Mon Sep 17 00:00:00 2001 From: Jesus Vazquez Date: Wed, 7 Jun 2023 12:29:04 +0200 Subject: [PATCH 199/231] Create release candidate 2.45.0-rc.0 (#12435) Signed-off-by: Jesus Vazquez --- CHANGELOG.md | 24 + VERSION | 2 +- go.mod | 59 +- go.sum | 118 ++-- web/api/v1/api.go | 2 +- web/ui/module/codemirror-promql/package.json | 18 +- web/ui/module/lezer-promql/package.json | 6 +- web/ui/package-lock.json | 592 ++++++++++--------- web/ui/package.json | 12 +- web/ui/react-app/package.json | 44 +- web/web.go | 2 +- 11 files changed, 464 insertions(+), 415 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7ff7022156..3b4ee3fc17 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,29 @@ # Changelog +## 2.45.0-rc.0 / 2023-06-05 + +This release is a LTS (Long-Term Support) release of Prometheus and will +receive security, documentation and bugfix patches for at least 12 months. +Please read more about our LTS release cycle at +. + +* [FEATURE] API: New limit parameter to limit the number of items returned by `/api/v1/status/tsdb` endpoint. #12336 +* [FEATURE] Config: Add limits to global config. #12126 +* [FEATURE] Consul SD: Added support for `path_prefix`. #12372 +* [FEATURE] Native histograms: Add option to scrape both classic and native histograms. #12350 +* [FEATURE] Native histograms: Added support for two more arithmetic operators `avg_over_time` and `sum_over_time`. #12262 +* [FEATURE] Promtool: When providing the block id, only one block will be loaded and analyzed. #12031 +* [FEATURE] Remote-write: New Azure ad configuration to support remote writing directly to Azure Monitor workspace. #11944 +* [FEATURE] TSDB: Samples per chunk are now configurable with flag `storage.tsdb.samples-per-chunk`. By default set to its former value 120. #12055 +* [ENHANCEMENT] API: Improving Performance on the API Gzip Handler. #12363 +* [ENHANCEMENT] Native histograms: bucket size can now be limited to avoid scrape fails. #12254 +* [ENHANCEMENT] TSDB: Dropped series are now deleted from the WAL sooner. #12297 +* [BUGFIX] Native histograms: ChunkSeries iterator now checks if a new sample can be appended to the open chunk. #12185 +* [BUGFIX] Native histograms: Fix Histogram Appender `Appendable()` segfault. #12357 +* [BUGFIX] Native histograms: Fix setting reset header to gauge histograms in seriesToChunkEncoder. #12329 +* [BUGFIX] TSDB: Tombstone intervals are not modified after Get() call. #12245 +* [BUGFIX] TSDB: Use path/filepath to set the WAL directory. #12349 + ## 2.44.0 / 2023-05-13 This version is built with Go tag `stringlabels`, to use the smaller data diff --git a/VERSION b/VERSION index 3e197472e2..ae2ba732af 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.44.0 +2.45.0-rc.0 diff --git a/go.mod b/go.mod index 71f6a2b8db..dd75cc333d 100644 --- a/go.mod +++ b/go.mod @@ -6,18 +6,18 @@ require ( github.com/Azure/azure-sdk-for-go v65.0.0+incompatible github.com/Azure/azure-sdk-for-go/sdk/azcore v1.3.1 github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.1 - github.com/Azure/go-autorest/autorest v0.11.28 + github.com/Azure/go-autorest/autorest v0.11.29 github.com/Azure/go-autorest/autorest/adal v0.9.23 github.com/alecthomas/kingpin/v2 v2.3.2 github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 - github.com/aws/aws-sdk-go v1.44.245 + github.com/aws/aws-sdk-go v1.44.276 github.com/cespare/xxhash/v2 v2.2.0 github.com/dennwc/varint v1.0.0 github.com/digitalocean/godo v1.99.0 - github.com/docker/docker v23.0.4+incompatible + github.com/docker/docker v24.0.2+incompatible github.com/edsrzf/mmap-go v1.1.0 github.com/envoyproxy/go-control-plane v0.11.0 - github.com/envoyproxy/protoc-gen-validate v0.10.1 + github.com/envoyproxy/protoc-gen-validate v1.0.1 github.com/fsnotify/fsnotify v1.6.0 github.com/go-kit/log v0.2.1 github.com/go-logfmt/logfmt v0.6.0 @@ -25,19 +25,19 @@ require ( github.com/go-zookeeper/zk v1.0.3 github.com/gogo/protobuf v1.3.2 github.com/golang/snappy v0.0.4 - github.com/google/pprof v0.0.0-20230406165453-00490a63f317 - github.com/gophercloud/gophercloud v1.3.0 + github.com/google/pprof v0.0.0-20230602150820-91b7bce49751 + github.com/gophercloud/gophercloud v1.4.0 github.com/grafana/regexp v0.0.0-20221122212121-6b5c0a4cb7fd github.com/grpc-ecosystem/grpc-gateway v1.16.0 - github.com/hashicorp/consul/api v1.20.0 - github.com/hashicorp/nomad/api v0.0.0-20230418003350-3067191c5197 + github.com/hashicorp/consul/api v1.21.0 + github.com/hashicorp/nomad/api v0.0.0-20230605233119-67e39d5d248f github.com/hetznercloud/hcloud-go v1.45.1 - github.com/ionos-cloud/sdk-go/v6 v6.1.6 + github.com/ionos-cloud/sdk-go/v6 v6.1.7 github.com/json-iterator/go v1.1.12 - github.com/klauspost/compress v1.13.6 + github.com/klauspost/compress v1.16.5 github.com/kolo/xmlrpc v0.0.0-20220921171641-a4b6fa1dd06b - github.com/linode/linodego v1.16.1 - github.com/miekg/dns v1.1.53 + github.com/linode/linodego v1.17.0 + github.com/miekg/dns v1.1.54 github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f github.com/oklog/run v1.1.0 github.com/oklog/ulid v1.3.1 @@ -45,14 +45,14 @@ require ( github.com/pkg/errors v0.9.1 github.com/prometheus/alertmanager v0.25.0 github.com/prometheus/client_golang v1.15.1 - github.com/prometheus/client_model v0.3.0 - github.com/prometheus/common v0.42.0 + github.com/prometheus/client_model v0.4.0 + github.com/prometheus/common v0.44.0 github.com/prometheus/common/assets v0.2.0 github.com/prometheus/common/sigv4 v0.1.0 github.com/prometheus/exporter-toolkit v0.10.0 - github.com/scaleway/scaleway-sdk-go v1.0.0-beta.15 + github.com/scaleway/scaleway-sdk-go v1.0.0-beta.17 github.com/shurcooL/httpfs v0.0.0-20190707220628-8d4bc4ba7749 - github.com/stretchr/testify v1.8.3 + github.com/stretchr/testify v1.8.4 github.com/vultr/govultr/v2 v2.17.2 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.42.0 go.opentelemetry.io/otel v1.16.0 @@ -61,17 +61,17 @@ require ( go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.16.0 go.opentelemetry.io/otel/sdk v1.16.0 go.opentelemetry.io/otel/trace v1.16.0 - go.uber.org/atomic v1.10.0 + go.uber.org/atomic v1.11.0 go.uber.org/automaxprocs v1.5.2 go.uber.org/goleak v1.2.1 golang.org/x/net v0.10.0 - golang.org/x/oauth2 v0.7.0 - golang.org/x/sync v0.1.0 + golang.org/x/oauth2 v0.8.0 + golang.org/x/sync v0.2.0 golang.org/x/sys v0.8.0 golang.org/x/time v0.3.0 - golang.org/x/tools v0.8.0 + golang.org/x/tools v0.9.3 google.golang.org/api v0.114.0 - google.golang.org/genproto/googleapis/api v0.0.0-20230526203410-71b5a4ffd15e + google.golang.org/genproto v0.0.0-20230320184635-7606e756e683 google.golang.org/grpc v1.55.0 google.golang.org/protobuf v1.30.0 gopkg.in/yaml.v2 v2.4.0 @@ -80,23 +80,20 @@ require ( k8s.io/apimachinery v0.26.2 k8s.io/client-go v0.26.2 k8s.io/klog v1.0.0 - k8s.io/klog/v2 v2.90.1 + k8s.io/klog/v2 v2.100.1 ) require ( cloud.google.com/go/compute/metadata v0.2.3 // indirect - github.com/Azure/azure-sdk-for-go/sdk/internal v1.1.2 // indirect + github.com/Azure/azure-sdk-for-go/sdk/internal v1.3.0 // indirect github.com/AzureAD/microsoft-authentication-library-for-go v0.8.1 // indirect github.com/coreos/go-systemd/v22 v22.5.0 // indirect github.com/hashicorp/errwrap v1.1.0 // indirect github.com/hashicorp/go-multierror v1.1.1 // indirect github.com/kylelemons/godebug v1.1.0 // indirect - github.com/pkg/browser v0.0.0-20210115035449-ce105d075bb4 // indirect - github.com/rogpeppe/go-internal v1.10.0 // indirect + github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect github.com/stretchr/objx v0.5.0 // indirect github.com/xhit/go-str2duration/v2 v2.1.0 // indirect - google.golang.org/genproto v0.0.0-20230526203410-71b5a4ffd15e // indirect - google.golang.org/genproto/googleapis/rpc v0.0.0-20230526203410-71b5a4ffd15e // indirect ) require ( @@ -112,7 +109,7 @@ require ( github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect github.com/beorn7/perks v1.0.1 // indirect github.com/cenkalti/backoff/v4 v4.2.1 // indirect - github.com/cncf/xds/go v0.0.0-20230310173818-32f1caf87195 // indirect + github.com/cncf/xds/go v0.0.0-20230428030218-4003588d1b74 // indirect github.com/davecgh/go-spew v1.1.1 // indirect github.com/docker/distribution v2.8.1+incompatible // indirect github.com/docker/go-connections v0.4.0 // indirect @@ -146,7 +143,7 @@ require ( github.com/googleapis/enterprise-certificate-proxy v0.2.3 // indirect github.com/googleapis/gax-go/v2 v2.7.1 // indirect github.com/gorilla/websocket v1.5.0 // indirect - github.com/grpc-ecosystem/grpc-gateway/v2 v2.15.2 // indirect + github.com/grpc-ecosystem/grpc-gateway/v2 v2.15.0 // indirect github.com/hashicorp/cronexpr v1.1.1 // indirect github.com/hashicorp/go-cleanhttp v0.5.2 // indirect github.com/hashicorp/go-hclog v1.4.0 // indirect @@ -182,7 +179,7 @@ require ( go.opentelemetry.io/otel/metric v1.16.0 // indirect go.opentelemetry.io/proto/otlp v0.19.0 // indirect golang.org/x/crypto v0.8.0 // indirect - golang.org/x/exp v0.0.0-20230321023759-10a507213a29 + golang.org/x/exp v0.0.0-20230522175609-2e198f4a06a1 golang.org/x/mod v0.10.0 // indirect golang.org/x/term v0.8.0 // indirect golang.org/x/text v0.9.0 // indirect @@ -190,7 +187,7 @@ require ( gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/ini.v1 v1.67.0 // indirect gotest.tools/v3 v3.0.3 // indirect - k8s.io/kube-openapi v0.0.0-20230303024457-afdc3dddf62d // indirect + k8s.io/kube-openapi v0.0.0-20230501164219-8b0f38b5fd1f // indirect k8s.io/utils v0.0.0-20230308161112-d77c459e9343 // indirect sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect sigs.k8s.io/structured-merge-diff/v4 v4.2.3 // indirect diff --git a/go.sum b/go.sum index 859ce81ca2..774deb85be 100644 --- a/go.sum +++ b/go.sum @@ -42,15 +42,15 @@ github.com/Azure/azure-sdk-for-go/sdk/azcore v1.3.1 h1:gVXuXcWd1i4C2Ruxe321aU+IK github.com/Azure/azure-sdk-for-go/sdk/azcore v1.3.1/go.mod h1:DffdKW9RFqa5VgmsjUOsS7UE7eiA5iAvYUs63bhKQ0M= github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.1 h1:T8quHYlUGyb/oqtSTwqlCr1ilJHrDv+ZtpSfo+hm1BU= github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.1/go.mod h1:gLa1CL2RNE4s7M3yopJ/p0iq5DdY6Yv5ZUt9MTRZOQM= -github.com/Azure/azure-sdk-for-go/sdk/internal v1.1.2 h1:+5VZ72z0Qan5Bog5C+ZkgSqUbeVUd9wgtHOrIKuc5b8= -github.com/Azure/azure-sdk-for-go/sdk/internal v1.1.2/go.mod h1:eWRD7oawr1Mu1sLCawqVc0CUiF43ia3qQMxLscsKQ9w= +github.com/Azure/azure-sdk-for-go/sdk/internal v1.3.0 h1:sXr+ck84g/ZlZUOZiNELInmMgOsuGwdjjVkEIde0OtY= +github.com/Azure/azure-sdk-for-go/sdk/internal v1.3.0/go.mod h1:okt5dMMTOFjX/aovMlrjvvXoPMBVSPzk9185BT0+eZM= github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 h1:UQHMgLO+TxOElx5B5HZ4hJQsoJ/PvUvKRhJHDQXO8P8= github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E= github.com/Azure/go-autorest v14.2.0+incompatible h1:V5VMDjClD3GiElqLWO7mz2MxNAK/vTfRHdAubSIPRgs= github.com/Azure/go-autorest v14.2.0+incompatible/go.mod h1:r+4oMnoxhatjLLJ6zxSWATqVooLgysK6ZNox3g/xq24= -github.com/Azure/go-autorest/autorest v0.11.28 h1:ndAExarwr5Y+GaHE6VCaY1kyS/HwwGGyuimVhWsHOEM= -github.com/Azure/go-autorest/autorest v0.11.28/go.mod h1:MrkzG3Y3AH668QyF9KRk5neJnGgmhQ6krbhR8Q5eMvA= -github.com/Azure/go-autorest/autorest/adal v0.9.18/go.mod h1:XVVeme+LZwABT8K5Lc3hA4nAe8LDBVle26gTrguhhPQ= +github.com/Azure/go-autorest/autorest v0.11.29 h1:I4+HL/JDvErx2LjyzaVxllw2lRDB5/BT2Bm4g20iqYw= +github.com/Azure/go-autorest/autorest v0.11.29/go.mod h1:ZtEzC4Jy2JDrZLxvWs8LrBWEBycl1hbT1eknI8MtfAs= +github.com/Azure/go-autorest/autorest/adal v0.9.22/go.mod h1:XuAbAEUv2Tta//+voMI038TrJBqjKam0me7qR+L8Cmk= github.com/Azure/go-autorest/autorest/adal v0.9.23 h1:Yepx8CvFxwNKpH6ja7RZ+sKX+DWYNldbLiALMC3BTz8= github.com/Azure/go-autorest/autorest/adal v0.9.23/go.mod h1:5pcMqFkdPhviJdlEy3kC/v1ZLnQl0MH6XA5YCcMhy4c= github.com/Azure/go-autorest/autorest/date v0.3.0 h1:7gUk1U5M/CQbp9WoqinNzJar+8KY+LPI6wiWrP/myHw= @@ -106,8 +106,8 @@ github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2/go.mod h1:W github.com/aws/aws-lambda-go v1.13.3/go.mod h1:4UKl9IzQMoD+QF79YdCuzCwp8VbmG4VAQwij/eHl5CU= github.com/aws/aws-sdk-go v1.27.0/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo= github.com/aws/aws-sdk-go v1.38.35/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro= -github.com/aws/aws-sdk-go v1.44.245 h1:KtY2s4q31/kn33AdV63R5t77mdxsI7rq3YT7Mgo805M= -github.com/aws/aws-sdk-go v1.44.245/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI= +github.com/aws/aws-sdk-go v1.44.276 h1:ywPlx9C5Yc482dUgAZ9bHpQ6onVvJvYE9FJWsNDCEy0= +github.com/aws/aws-sdk-go v1.44.276/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI= github.com/aws/aws-sdk-go-v2 v0.18.0/go.mod h1:JWVYvqSMppoMJC0x5wdwiImzgXTI9FuZwxzkQq9wy+g= github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8= @@ -138,8 +138,8 @@ github.com/cncf/xds/go v0.0.0-20210312221358-fbca930ec8ed/go.mod h1:eXthEFrGJvWH github.com/cncf/xds/go v0.0.0-20210805033703-aa0b78936158/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= github.com/cncf/xds/go v0.0.0-20210922020428-25de7278fc84/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= github.com/cncf/xds/go v0.0.0-20211011173535-cb28da3451f1/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= -github.com/cncf/xds/go v0.0.0-20230310173818-32f1caf87195 h1:58f1tJ1ra+zFINPlwLWvQsR9CzAKt2e+EWV2yX9oXQ4= -github.com/cncf/xds/go v0.0.0-20230310173818-32f1caf87195/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= +github.com/cncf/xds/go v0.0.0-20230428030218-4003588d1b74 h1:zlUubfBUxApscKFsF4VSvvfhsBNTBu0eF/ddvpo96yk= +github.com/cncf/xds/go v0.0.0-20230428030218-4003588d1b74/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= github.com/cockroachdb/datadriven v0.0.0-20190809214429-80d97fb3cbaa/go.mod h1:zn76sxSg3SzpJ0PPJaLDCu+Bu0Lg3sKTORVIj19EIF8= github.com/codahale/hdrhistogram v0.0.0-20161010025455-3a0bb77429bd/go.mod h1:sE/e/2PUdi/liOCUjSTXgM1o87ZssimdTWN964YiIeI= github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= @@ -162,8 +162,8 @@ github.com/digitalocean/godo v1.99.0/go.mod h1:SsS2oXo2rznfM/nORlZ/6JaUJZFhmKTib github.com/dnaeon/go-vcr v1.2.0 h1:zHCHvJYTMh1N7xnV7zf1m1GPBF9Ad0Jk/whtQ1663qI= github.com/docker/distribution v2.8.1+incompatible h1:Q50tZOPR6T/hjNsyc9g8/syEs6bk8XXApsHjKukMl68= github.com/docker/distribution v2.8.1+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w= -github.com/docker/docker v23.0.4+incompatible h1:Kd3Bh9V/rO+XpTP/BLqM+gx8z7+Yb0AA2Ibj+nNo4ek= -github.com/docker/docker v23.0.4+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= +github.com/docker/docker v24.0.2+incompatible h1:eATx+oLz9WdNVkQrr0qjQ8HvRJ4bOOxfzEo8R+dA3cg= +github.com/docker/docker v24.0.2+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= github.com/docker/go-connections v0.4.0 h1:El9xVISelRB7BuFusrZozjnkIM5YnzCViNKohAFqRJQ= github.com/docker/go-connections v0.4.0/go.mod h1:Gbd7IOopHjR8Iph03tsViu4nIes5XhDvyHbTtUxmeec= github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4= @@ -188,8 +188,8 @@ github.com/envoyproxy/go-control-plane v0.9.10-0.20210907150352-cf90f659a021/go. github.com/envoyproxy/go-control-plane v0.11.0 h1:jtLewhRR2vMRNnq2ZZUoCjUlgut+Y0+sDDWPOfwOi1o= github.com/envoyproxy/go-control-plane v0.11.0/go.mod h1:VnHyVMpzcLvCFt9yUz1UnCwHLhwx1WguiVDV7pTG/tI= github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= -github.com/envoyproxy/protoc-gen-validate v0.10.1 h1:c0g45+xCJhdgFGw7a5QAfdS4byAbud7miNWJ1WwEVf8= -github.com/envoyproxy/protoc-gen-validate v0.10.1/go.mod h1:DRjgyB0I43LtJapqN6NiRwroiAU2PaFuvk/vjgh61ss= +github.com/envoyproxy/protoc-gen-validate v1.0.1 h1:kt9FtLiooDc0vbwTLhdg3dyNX1K9Qwa1EK9LcD4jVUQ= +github.com/envoyproxy/protoc-gen-validate v1.0.1/go.mod h1:0vj8bNkYbSTNS2PIyH87KZaeN4x9zpL9Qt8fQC7d+vs= github.com/evanphx/json-patch v4.12.0+incompatible h1:4onqiflcdA9EOZ4RxV643DvftH5pOlLGNtQ5lPWQu84= github.com/evanphx/json-patch v4.12.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4= @@ -301,7 +301,6 @@ github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zV github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= github.com/golang-jwt/jwt/v4 v4.0.0/go.mod h1:/xlHOz8bRuivTWchD4jCa+NbatV+wEUSzwAxVc6locg= -github.com/golang-jwt/jwt/v4 v4.2.0/go.mod h1:/xlHOz8bRuivTWchD4jCa+NbatV+wEUSzwAxVc6locg= github.com/golang-jwt/jwt/v4 v4.5.0 h1:7cYmW1XlMY7h7ii7UhUyChSgS5wUJEnm9uZVTGqOWzg= github.com/golang-jwt/jwt/v4 v4.5.0/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= @@ -376,8 +375,8 @@ github.com/google/pprof v0.0.0-20200212024743-f11f1df84d12/go.mod h1:ZgVRPoUq/hf github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= github.com/google/pprof v0.0.0-20200430221834-fc25d7d30c6d/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= github.com/google/pprof v0.0.0-20200708004538-1a94d8640e99/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= -github.com/google/pprof v0.0.0-20230406165453-00490a63f317 h1:hFhpt7CTmR3DX+b4R19ydQFtofxT0Sv3QsKNMVQYTMQ= -github.com/google/pprof v0.0.0-20230406165453-00490a63f317/go.mod h1:79YE0hCXdHag9sBkw2o+N/YnZtTkXi0UT9Nnixa5eYk= +github.com/google/pprof v0.0.0-20230602150820-91b7bce49751 h1:hR7/MlvK23p6+lIw9SN1TigNLn9ZnF3W4SYRKq2gAHs= +github.com/google/pprof v0.0.0-20230602150820-91b7bce49751/go.mod h1:Jh3hGz2jkYak8qXPD19ryItVnUgpgeqzdkY/D0EaeuA= github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= @@ -390,8 +389,8 @@ github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+ github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= github.com/googleapis/gax-go/v2 v2.7.1 h1:gF4c0zjUP2H/s/hEGyLA3I0fA2ZWjzYiONAD6cvPr8A= github.com/googleapis/gax-go/v2 v2.7.1/go.mod h1:4orTrqY6hXxxaUL4LHIPl6lGo8vAE38/qKbhSAKP6QI= -github.com/gophercloud/gophercloud v1.3.0 h1:RUKyCMiZoQR3VlVR5E3K7PK1AC3/qppsWYo6dtBiqs8= -github.com/gophercloud/gophercloud v1.3.0/go.mod h1:aAVqcocTSXh2vYFZ1JTvx4EQmfgzxRcNupUfxZbBNDM= +github.com/gophercloud/gophercloud v1.4.0 h1:RqEu43vaX0lb0LanZr5BylK5ICVxjpFFoc0sxivyuHU= +github.com/gophercloud/gophercloud v1.4.0/go.mod h1:aAVqcocTSXh2vYFZ1JTvx4EQmfgzxRcNupUfxZbBNDM= github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY= github.com/gorilla/context v1.1.1/go.mod h1:kBGZzfjB9CEq2AlWe17Uuf7NDRt0dE0s8S51q0aT7Yg= github.com/gorilla/mux v1.6.2/go.mod h1:1lud6UwP+6orDFRuTfBEV8e9/aOM/c4fVVCaMa2zaAs= @@ -407,11 +406,11 @@ github.com/grpc-ecosystem/grpc-gateway v1.9.5/go.mod h1:vNeuVxBJEsws4ogUvrchl83t github.com/grpc-ecosystem/grpc-gateway v1.16.0 h1:gmcG1KaJ57LophUzW0Hy8NmPhnMZb4M0+kPpLofRdBo= github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw= github.com/grpc-ecosystem/grpc-gateway/v2 v2.7.0/go.mod h1:hgWBS7lorOAVIJEQMi4ZsPv9hVvWI6+ch50m39Pf2Ks= -github.com/grpc-ecosystem/grpc-gateway/v2 v2.15.2 h1:gDLXvp5S9izjldquuoAhDzccbskOL6tDC5jMSyx3zxE= -github.com/grpc-ecosystem/grpc-gateway/v2 v2.15.2/go.mod h1:7pdNwVWBBHGiCxa9lAszqCJMbfTISJ7oMftp8+UGV08= +github.com/grpc-ecosystem/grpc-gateway/v2 v2.15.0 h1:1JYBfzqrWPcCclBwxFCPAou9n+q86mfnu7NAeHfte7A= +github.com/grpc-ecosystem/grpc-gateway/v2 v2.15.0/go.mod h1:YDZoGHuwE+ov0c8smSH49WLF3F2LaWnYYuDVd+EWrc0= github.com/hashicorp/consul/api v1.3.0/go.mod h1:MmDNSzIMUjNpY/mQ398R4bk2FnqQLoPndWW5VkKPlCE= -github.com/hashicorp/consul/api v1.20.0 h1:9IHTjNVSZ7MIwjlW3N3a7iGiykCMDpxZu8jsxFJh0yc= -github.com/hashicorp/consul/api v1.20.0/go.mod h1:nR64eD44KQ59Of/ECwt2vUmIK2DKsDzAwTmwmLl8Wpo= +github.com/hashicorp/consul/api v1.21.0 h1:WMR2JiyuaQWRAMFaOGiYfY4Q4HRpyYRe/oYQofjyduM= +github.com/hashicorp/consul/api v1.21.0/go.mod h1:f8zVJwBcLdr1IQnfdfszjUM0xzp31Zl3bpws3pL9uFM= github.com/hashicorp/consul/sdk v0.3.0/go.mod h1:VKf9jXwCTEY1QZP2MOLRhb5i/I/ssyNV1vwHyQBF0x8= github.com/hashicorp/consul/sdk v0.13.1 h1:EygWVWWMczTzXGpO93awkHFzfUka6hLYJ0qhETd+6lY= github.com/hashicorp/cronexpr v1.1.1 h1:NJZDd87hGXjoZBdvyCF9mX4DCq5Wy7+A/w+A7q0wn6c= @@ -460,8 +459,8 @@ github.com/hashicorp/mdns v1.0.4/go.mod h1:mtBihi+LeNXGtG8L9dX59gAEa12BDtBQSp4v/ github.com/hashicorp/memberlist v0.1.3/go.mod h1:ajVTdAv/9Im8oMAAj5G31PhhMCZJV2pPBoIllUwCN7I= github.com/hashicorp/memberlist v0.5.0 h1:EtYPN8DpAURiapus508I4n9CzHs2W+8NZGbmmR/prTM= github.com/hashicorp/memberlist v0.5.0/go.mod h1:yvyXLpo0QaGE59Y7hDTsTzDD25JYBZ4mHgHUZ8lrOI0= -github.com/hashicorp/nomad/api v0.0.0-20230418003350-3067191c5197 h1:I5xhKLePXpXgM6pZ4xZNTiurLLS3sGuZrZFFzAbM67A= -github.com/hashicorp/nomad/api v0.0.0-20230418003350-3067191c5197/go.mod h1:2TCrNvonL09r7EiQ6M2rNt+Cmjbn1QbzchFoTWJFpj4= +github.com/hashicorp/nomad/api v0.0.0-20230605233119-67e39d5d248f h1:yxjcAZRuYymIDC0W4IQHgTe9EQdu2BsjPlVmKwyVZT4= +github.com/hashicorp/nomad/api v0.0.0-20230605233119-67e39d5d248f/go.mod h1:Xjd3OXUTfsWbCCBsQd3EdfPTz5evDi+fxqdvpN+WqQg= github.com/hashicorp/serf v0.8.2/go.mod h1:6hOLApaqBFA1NXqRQAsxw9QxuDEvNxSQRwA/JwenrHc= github.com/hashicorp/serf v0.10.1 h1:Z1H2J60yRKvfDYAOZLd2MU0ND4AH/WDz7xYHDWQsIPY= github.com/hashicorp/serf v0.10.1/go.mod h1:yL2t6BqATOLGc5HF7qbFkTfXoPIY0WZdWHfEvMqbG+4= @@ -474,8 +473,8 @@ github.com/imdario/mergo v0.3.13 h1:lFzP57bqS/wsqKssCGmtLAb8A0wKjLGrve2q3PPVcBk= github.com/imdario/mergo v0.3.13/go.mod h1:4lJ1jqUDcsbIECGy0RUJAXNIhg+6ocWgb1ALK2O4oXg= github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8= github.com/influxdata/influxdb1-client v0.0.0-20191209144304-8bf82d3c094d/go.mod h1:qj24IKcXYK6Iy9ceXlo3Tc+vtHo9lIhSX5JddghvEPo= -github.com/ionos-cloud/sdk-go/v6 v6.1.6 h1:0n4irdqNska+1s3YMCRhrAqKbibEgQ7SwwhAlHzYT5A= -github.com/ionos-cloud/sdk-go/v6 v6.1.6/go.mod h1:EzEgRIDxBELvfoa/uBN0kOQaqovLjUWEB7iW4/Q+t4k= +github.com/ionos-cloud/sdk-go/v6 v6.1.7 h1:uVG1Q/ZDJ7YmCI9Oevpue9xJEH5UrUMyXv8gm7NTxIw= +github.com/ionos-cloud/sdk-go/v6 v6.1.7/go.mod h1:EzEgRIDxBELvfoa/uBN0kOQaqovLjUWEB7iW4/Q+t4k= github.com/jarcoal/httpmock v1.3.0 h1:2RJ8GP0IIaWwcC9Fp2BmVi8Kog3v2Hn7VXM3fTd+nuc= github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k= github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg= @@ -507,8 +506,9 @@ github.com/karrick/godirwalk v1.10.3/go.mod h1:RoGL9dQei4vP9ilrpETWE8CLOZ1kiN0Lh github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= -github.com/klauspost/compress v1.13.6 h1:P76CopJELS0TiO2mebmnzgWaajssP/EszplttgQxcgc= github.com/klauspost/compress v1.13.6/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk= +github.com/klauspost/compress v1.16.5 h1:IFV2oUNUzZaz+XyusxpLzpzS8Pt5rh0Z16For/djlyI= +github.com/klauspost/compress v1.16.5/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE= github.com/kolo/xmlrpc v0.0.0-20220921171641-a4b6fa1dd06b h1:udzkj9S/zlT5X367kqJis0QP7YMxobob6zhzq6Yre00= github.com/kolo/xmlrpc v0.0.0-20220921171641-a4b6fa1dd06b/go.mod h1:pcaDhQK0/NJZEvtCO0qQPPropqV0sJOJ6YW7X+9kRwM= github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= @@ -527,8 +527,8 @@ github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0 github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw= github.com/lightstep/lightstep-tracer-common/golang/gogo v0.0.0-20190605223551-bc2310a04743/go.mod h1:qklhhLq1aX+mtWk9cPHPzaBjWImj5ULL6C7HFJtXQMM= github.com/lightstep/lightstep-tracer-go v0.18.1/go.mod h1:jlF1pusYV4pidLvZ+XD0UBX0ZE6WURAspgAczcDHrL4= -github.com/linode/linodego v1.16.1 h1:5otq57M4PdHycPERRfSFZ0s1yz1ETVWGjCp3hh7+F9w= -github.com/linode/linodego v1.16.1/go.mod h1:aESRAbpLY9R6IA1WGAWHikRI9DU9Lhesapv1MhKmPHM= +github.com/linode/linodego v1.17.0 h1:aWS98f0jUoY2lhsEuBxRdVkqyGM0nazPd68AEDF0EvU= +github.com/linode/linodego v1.17.0/go.mod h1:/omzPxie0/YI6S0sTw1q47qDt5IYSlbO/infRR4UG+A= github.com/lyft/protoc-gen-validate v0.0.13/go.mod h1:XbGvPuh87YZc5TdIa2/I4pLk0QoUACkjt2znoq26NVQ= github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= @@ -561,8 +561,8 @@ github.com/maxatome/go-testdeep v1.12.0 h1:Ql7Go8Tg0C1D/uMMX59LAoYK7LffeJQ6X2T04 github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg= github.com/miekg/dns v1.1.26/go.mod h1:bPDLeHnStXmXAq1m/Ch/hvfNHr14JKNPMBo3VZKjuso= github.com/miekg/dns v1.1.41/go.mod h1:p6aan82bvRIyn+zDIv9xYNUpwa73JcSh9BKwknJysuI= -github.com/miekg/dns v1.1.53 h1:ZBkuHr5dxHtB1caEOlZTLPo7D3L3TWckgUUs/RHfDxw= -github.com/miekg/dns v1.1.53/go.mod h1:uInx36IzPl7FYnDcMeVWxj9byh7DutNykX4G9Sj60FY= +github.com/miekg/dns v1.1.54 h1:5jon9mWcb0sFJGpnI99tOMhCPyJ+RPVz5b63MQG0VWI= +github.com/miekg/dns v1.1.54/go.mod h1:uInx36IzPl7FYnDcMeVWxj9byh7DutNykX4G9Sj60FY= github.com/mitchellh/cli v1.0.0/go.mod h1:hNIlj7HEI86fIcpObd7a0FcrxTWetlwJDGcceTlRvqc= github.com/mitchellh/cli v1.1.0/go.mod h1:xcISNoH86gajksDmfB23e/pu+B+GeFRMYmoHXxx3xhI= github.com/mitchellh/go-homedir v1.0.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= @@ -640,8 +640,8 @@ github.com/pelletier/go-toml v1.7.0/go.mod h1:vwGMzjaWMwyfHwgIBhI2YUM4fB6nL6lVAv github.com/performancecopilot/speed v3.0.0+incompatible/go.mod h1:/CLtqpZ5gBg1M9iaPbIdPPGyKcA8hKdoy6hAWba7Yac= github.com/pierrec/lz4 v1.0.2-0.20190131084431-473cd7ce01a1/go.mod h1:3/3N9NVKO0jef7pBehbT1qWhCMrIgbYNnFAZCqQ5LRc= github.com/pierrec/lz4 v2.0.5+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi+IEE17M5jbnwPHcY= -github.com/pkg/browser v0.0.0-20210115035449-ce105d075bb4 h1:Qj1ukM4GlMWXNdMBuXcXfz/Kw9s1qm0CLY32QxuSImI= -github.com/pkg/browser v0.0.0-20210115035449-ce105d075bb4/go.mod h1:N6UoU20jOqggOuDwUaBQpluzLNDqif3kq9z2wpdYEfQ= +github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 h1:KoWmjvw+nsYOo29YJK9vDA65RGE3NrOnUtO7a+RF9HU= +github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8/go.mod h1:HKlIX3XHQyzLZPlr7++PzdhaXEj94dEiJgZDTsxEqUI= github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= @@ -669,8 +669,8 @@ github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1: github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= github.com/prometheus/client_model v0.1.0/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= github.com/prometheus/client_model v0.2.0/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= -github.com/prometheus/client_model v0.3.0 h1:UBgGFHqYdG/TPFD1B1ogZywDqEkwp3fBMvqdiQ7Xew4= -github.com/prometheus/client_model v0.3.0/go.mod h1:LDGWKZIo7rky3hgvBe+caln+Dr3dPggB5dvjtD7w9+w= +github.com/prometheus/client_model v0.4.0 h1:5lQXD3cAg1OXBf4Wq03gTrXHeaV0TQvGfUooCfx1yqY= +github.com/prometheus/client_model v0.4.0/go.mod h1:oMQmHW1/JoDwqLtg57MGgP/Fb1CJEYF2imWWhWtMkYU= github.com/prometheus/common v0.2.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4= github.com/prometheus/common v0.4.1/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4= github.com/prometheus/common v0.7.0/go.mod h1:DjGbpBbp5NYNiECxcL/VnbXCCaQpKd3tt26CguLLsqA= @@ -678,8 +678,8 @@ github.com/prometheus/common v0.9.1/go.mod h1:yhUN8i9wzaXS3w1O07YhxHEBxD+W35wd8b github.com/prometheus/common v0.10.0/go.mod h1:Tlit/dnDKsSWFlCLTWaA1cyBgKHSMdTB80sz/V91rCo= github.com/prometheus/common v0.26.0/go.mod h1:M7rCNAaPfAosfx8veZJCuw84e35h3Cfd9VFqTh1DIvc= github.com/prometheus/common v0.29.0/go.mod h1:vu+V0TpY+O6vW9J44gczi3Ap/oXXR10b+M/gUGO4Hls= -github.com/prometheus/common v0.42.0 h1:EKsfXEYo4JpWMHH5cg+KOUWeuJSov1Id8zGR8eeI1YM= -github.com/prometheus/common v0.42.0/go.mod h1:xBwqVerjNdUDjgODMpudtOMwlOwf2SaTr1yjz4b7Zbc= +github.com/prometheus/common v0.44.0 h1:+5BrQJwiBB9xsMygAB3TNvpQKOwlkc25LbISbrdOOfY= +github.com/prometheus/common v0.44.0/go.mod h1:ofAIvZbQ1e/nugmZGz4/qCb9Ap1VoSTIO7x0VV9VvuY= github.com/prometheus/common/assets v0.2.0 h1:0P5OrzoHrYBOSM1OigWL3mY8ZvV2N4zIE/5AahrSrfM= github.com/prometheus/common/assets v0.2.0/go.mod h1:D17UVUE12bHbim7HzwUvtqm6gwBEaDQ0F+hIGbFbccI= github.com/prometheus/common/sigv4 v0.1.0 h1:qoVebwtwwEhS85Czm2dSROY5fTo2PAPEVdDeppTwGX4= @@ -701,15 +701,14 @@ github.com/rogpeppe/go-internal v1.1.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFR github.com/rogpeppe/go-internal v1.2.2/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= -github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog= github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts= github.com/samuel/go-zookeeper v0.0.0-20190923202752-2cc03de413da/go.mod h1:gi+0XIa01GRL2eRQVjQkKGqKF3SF9vZR/HnPullcV2E= -github.com/scaleway/scaleway-sdk-go v1.0.0-beta.15 h1:Y7xOFbD+3jaPw+VN7lkakNJ/pa+ZSQVFp1ONtJaBxns= -github.com/scaleway/scaleway-sdk-go v1.0.0-beta.15/go.mod h1:fCa7OJZ/9DRTnOKmxvT6pn+LPWUptQAmHF/SBJUGEcg= +github.com/scaleway/scaleway-sdk-go v1.0.0-beta.17 h1:1WuWJu7/e8SqK+uQl7lfk/N/oMZTL2NE/TJsNKRNMc4= +github.com/scaleway/scaleway-sdk-go v1.0.0-beta.17/go.mod h1:fCa7OJZ/9DRTnOKmxvT6pn+LPWUptQAmHF/SBJUGEcg= github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529 h1:nn5Wsu0esKSJiIVhscUtVbo7ada43DJhG55ua/hjS5I= github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc= -github.com/shoenig/test v0.6.3 h1:GVXWJFk9PiOjN0KoJ7VrJGH6uLPnqxR7/fe3HUPfE0c= +github.com/shoenig/test v0.6.6 h1:Oe8TPH9wAbv++YPNDKJWUnI8Q4PPWCx3UbOfH+FxiMU= github.com/shurcooL/httpfs v0.0.0-20190707220628-8d4bc4ba7749 h1:bUGsEnyNbVPw06Bs80sCeARAlK8lhwqGyi6UT8ymuGk= github.com/shurcooL/httpfs v0.0.0-20190707220628-8d4bc4ba7749/go.mod h1:ZY1cvUeJuFPAdZ/B6v7RHavJWZn2YPVFQ1OSXhCGOkg= github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc= @@ -752,8 +751,8 @@ github.com/stretchr/testify v1.7.2/go.mod h1:R6va5+xMeoiuVRoj+gSkQ7d3FALtqAAGI1F github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= -github.com/stretchr/testify v1.8.3 h1:RP3t2pwF7cMEbC1dqtB6poj3niw/9gnV4Cjg5oW5gtY= -github.com/stretchr/testify v1.8.3/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= +github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= +github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= github.com/tidwall/pretty v1.0.0 h1:HsD+QiTn7sK6flMKIvNmpqz1qrpP3Ps6jOKIKMooyg4= github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk= github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= @@ -818,8 +817,8 @@ go.opentelemetry.io/proto/otlp v0.19.0 h1:IVN6GR+mhC4s5yfcTbmzHYODqvWAp3ZedA2SJP go.opentelemetry.io/proto/otlp v0.19.0/go.mod h1:H7XAot3MsfNsj7EXtrA2q5xSNQ10UqI405h3+duxN4U= go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= go.uber.org/atomic v1.5.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ= -go.uber.org/atomic v1.10.0 h1:9qC72Qh0+3MqyJbAn8YU5xVq1frD8bn3JtD2oXtafVQ= -go.uber.org/atomic v1.10.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0= +go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE= +go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0= go.uber.org/automaxprocs v1.5.2 h1:2LxUOGiR3O6tw8ui5sZa2LAaHnsviZdVOUZw4fvbnME= go.uber.org/automaxprocs v1.5.2/go.mod h1:eRbA25aqJrxAbsLO0xy5jVwPt7FQnRgjW+efnwa1WM0= go.uber.org/goleak v1.2.1 h1:NBol2c7O1ZokfZ0LEU9K6Whx/KnwvepVetCUhtKja4A= @@ -857,8 +856,8 @@ golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u0 golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM= golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU= -golang.org/x/exp v0.0.0-20230321023759-10a507213a29 h1:ooxPy7fPvB4kwsA2h+iBNHkAbp/4JxTSwCmvdjEYmug= -golang.org/x/exp v0.0.0-20230321023759-10a507213a29/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc= +golang.org/x/exp v0.0.0-20230522175609-2e198f4a06a1 h1:k/i9J1pBpvlfR+9QsetwPyERsqu1GIbi967PQMq3Ivc= +golang.org/x/exp v0.0.0-20230522175609-2e198f4a06a1/go.mod h1:V1LtkGg67GoY2N1AnLN78QLrzxkLyJw7RJb1gzOOz9w= golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js= golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= @@ -938,8 +937,8 @@ golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4Iltr golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.7.0 h1:qe6s0zUXlPX80/dITx3440hWZ7GwMwgDDyrSGTPJG/g= -golang.org/x/oauth2 v0.7.0/go.mod h1:hPLQkd9LyjfXTiRohC/41GhcFqxisoUQ99sCUOHO9x4= +golang.org/x/oauth2 v0.8.0 h1:6dkIjl3j3LtZ/O3sTgZTMsLKSftL/B8Zgq4huOIIUu8= +golang.org/x/oauth2 v0.8.0/go.mod h1:yr7u4HXZRm1R1kBWqr/xKNqewf0plRYoB7sla+BCIXE= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -953,8 +952,8 @@ golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.1.0 h1:wsuoTGHzEhffawBOhz5CYhcrV4IdKZbEyZjBMuTp12o= -golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.2.0 h1:PUR+T4wwASmuSTYdKjYHI5TD22Wy5ogLU5qZCOLxBrI= +golang.org/x/sync v0.2.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20180823144017-11551d06cbcc/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -1010,6 +1009,7 @@ golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210616045830-e2b7044e8c71/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= @@ -1100,8 +1100,8 @@ golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roY golang.org/x/tools v0.0.0-20200825202427-b303f430e36d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA= golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= -golang.org/x/tools v0.8.0 h1:vSDcovVPld282ceKgDimkRSC8kpaH1dgyc9UMzlt84Y= -golang.org/x/tools v0.8.0/go.mod h1:JxBZ99ISMI5ViVkT1tr6tdNmXeTrcpVSD3vZ1RsRdN4= +golang.org/x/tools v0.9.3 h1:Gn1I8+64MsuTb/HpH+LmQtNas23LhUVr3rYZ0eKuaMM= +golang.org/x/tools v0.9.3/go.mod h1:owI94Op576fPu3cIGQeHs3joujW/2Oc6MtlxbF5dfNc= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= @@ -1163,12 +1163,8 @@ google.golang.org/genproto v0.0.0-20200618031413-b414f8b61790/go.mod h1:jDfRM7Fc google.golang.org/genproto v0.0.0-20200825200019-8632dd797987/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= google.golang.org/genproto v0.0.0-20211118181313-81c1377c94b1/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= google.golang.org/genproto v0.0.0-20220107163113-42d7afdf6368/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= -google.golang.org/genproto v0.0.0-20230526203410-71b5a4ffd15e h1:Ao9GzfUMPH3zjVfzXG5rlWlk+Q8MXWKwWpwVQE1MXfw= -google.golang.org/genproto v0.0.0-20230526203410-71b5a4ffd15e/go.mod h1:zqTuNwFlFRsw5zIts5VnzLQxSRqh+CGOTVMlYbY0Eyk= -google.golang.org/genproto/googleapis/api v0.0.0-20230526203410-71b5a4ffd15e h1:AZX1ra8YbFMSb7+1pI8S9v4rrgRR7jU1FmuFSSjTVcQ= -google.golang.org/genproto/googleapis/api v0.0.0-20230526203410-71b5a4ffd15e/go.mod h1:vHYtlOoi6TsQ3Uk2yxR7NI5z8uoV+3pZtR4jmHIkRig= -google.golang.org/genproto/googleapis/rpc v0.0.0-20230526203410-71b5a4ffd15e h1:NumxXLPfHSndr3wBBdeKiVHjGVFzi9RX2HwwQke94iY= -google.golang.org/genproto/googleapis/rpc v0.0.0-20230526203410-71b5a4ffd15e/go.mod h1:66JfowdXAEgad5O9NnYcsNPLCPZJD++2L9X0PCMODrA= +google.golang.org/genproto v0.0.0-20230320184635-7606e756e683 h1:khxVcsk/FhnzxMKOyD+TDGwjbEOpcPuIpmafPGFmhMA= +google.golang.org/genproto v0.0.0-20230320184635-7606e756e683/go.mod h1:NWraEVixdDnqcqQ30jipen1STv2r/n24Wb7twVTGR4s= google.golang.org/grpc v1.17.0/go.mod h1:6QZJwpn2B+Zp71q/5VxRsJ6NXXVCE5NRUHRo+f3cWCs= google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= google.golang.org/grpc v1.20.0/go.mod h1:chYK+tFQF0nDUGJgXMSgLCQk3phJEuONr2DCgLDdAQM= @@ -1259,8 +1255,8 @@ k8s.io/apimachinery v0.26.2 h1:da1u3D5wfR5u2RpLhE/ZtZS2P7QvDgLZTi9wrNZl/tQ= k8s.io/apimachinery v0.26.2/go.mod h1:ats7nN1LExKHvJ9TmwootT00Yz05MuYqPXEXaVeOy5I= k8s.io/client-go v0.26.2 h1:s1WkVujHX3kTp4Zn4yGNFK+dlDXy1bAAkIl+cFAiuYI= k8s.io/client-go v0.26.2/go.mod h1:u5EjOuSyBa09yqqyY7m3abZeovO/7D/WehVVlZ2qcqU= -k8s.io/kube-openapi v0.0.0-20230303024457-afdc3dddf62d h1:VcFq5n7wCJB2FQMCIHfC+f+jNcGgNMar1uKd6rVlifU= -k8s.io/kube-openapi v0.0.0-20230303024457-afdc3dddf62d/go.mod h1:y5VtZWM9sHHc2ZodIH/6SHzXj+TPU5USoA8lcIeKEKY= +k8s.io/kube-openapi v0.0.0-20230501164219-8b0f38b5fd1f h1:2kWPakN3i/k81b0gvD5C5FJ2kxm1WrQFanWchyKuqGg= +k8s.io/kube-openapi v0.0.0-20230501164219-8b0f38b5fd1f/go.mod h1:byini6yhqGC14c3ebc/QwanvYwhuMWF6yz2F8uwW8eg= k8s.io/utils v0.0.0-20230308161112-d77c459e9343 h1:m7tbIjXGcGIAtpmQr7/NAi7RsWoW3E7Zcm4jI1HicTc= k8s.io/utils v0.0.0-20230308161112-d77c459e9343/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0= rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8= diff --git a/web/api/v1/api.go b/web/api/v1/api.go index f7249efb04..e9b0081b36 100644 --- a/web/api/v1/api.go +++ b/web/api/v1/api.go @@ -1491,7 +1491,7 @@ func (api *API) serveTSDBStatus(r *http.Request) apiFuncResult { chunkCount := int64(math.NaN()) for _, mF := range metrics { if *mF.Name == "prometheus_tsdb_head_chunks" { - m := *mF.Metric[0] + m := mF.Metric[0] if m.Gauge != nil { chunkCount = int64(m.Gauge.GetValue()) break diff --git a/web/ui/module/codemirror-promql/package.json b/web/ui/module/codemirror-promql/package.json index e819319ee6..e9c952c9cf 100644 --- a/web/ui/module/codemirror-promql/package.json +++ b/web/ui/module/codemirror-promql/package.json @@ -33,17 +33,17 @@ "lru-cache": "^6.0.0" }, "devDependencies": { - "@codemirror/autocomplete": "^6.4.0", - "@codemirror/language": "^6.4.0", - "@codemirror/lint": "^6.1.0", - "@codemirror/state": "^6.2.0", - "@codemirror/view": "^6.7.3", - "@lezer/common": "^1.0.2", - "@lezer/lr": "^1.3.1", - "@lezer/highlight": "^1.1.3", + "@codemirror/autocomplete": "^6.7.1", + "@codemirror/language": "^6.7.0", + "@codemirror/lint": "^6.2.2", + "@codemirror/state": "^6.2.1", + "@codemirror/view": "^6.13.0", + "@lezer/common": "^1.0.3", + "@lezer/lr": "^1.3.6", + "@lezer/highlight": "^1.1.6", "@types/lru-cache": "^5.1.1", "isomorphic-fetch": "^3.0.0", - "nock": "^13.3.0" + "nock": "^13.3.1" }, "peerDependencies": { "@codemirror/autocomplete": "^6.4.0", diff --git a/web/ui/module/lezer-promql/package.json b/web/ui/module/lezer-promql/package.json index 7daa567388..08249b418d 100644 --- a/web/ui/module/lezer-promql/package.json +++ b/web/ui/module/lezer-promql/package.json @@ -30,9 +30,9 @@ "test": "NODE_OPTIONS=--experimental-vm-modules jest" }, "devDependencies": { - "@lezer/generator": "^1.2.2", - "@lezer/lr": "^1.3.1", - "@lezer/highlight": "^1.1.3" + "@lezer/generator": "^1.2.3", + "@lezer/lr": "^1.3.6", + "@lezer/highlight": "^1.1.6" }, "peerDependencies": { "@lezer/lr": "^1.2.3", diff --git a/web/ui/package-lock.json b/web/ui/package-lock.json index 1d3d1d3e34..a672a17673 100644 --- a/web/ui/package-lock.json +++ b/web/ui/package-lock.json @@ -10,17 +10,17 @@ "module/*" ], "devDependencies": { - "@types/jest": "^29.4.0", + "@types/jest": "^29.5.2", "@types/node": "^17.0.45", - "eslint-config-prettier": "^8.6.0", + "eslint-config-prettier": "^8.8.0", "eslint-config-react-app": "^7.0.1", "eslint-plugin-prettier": "^4.2.1", - "jest-canvas-mock": "^2.4.0", + "jest-canvas-mock": "^2.5.1", "jest-fetch-mock": "^3.0.3", - "prettier": "^2.8.3", + "prettier": "^2.8.8", "react-scripts": "^5.0.1", - "ts-jest": "^29.0.5", - "typescript": "^4.9.4" + "ts-jest": "^29.1.0", + "typescript": "^4.9.5" }, "engines": { "npm": ">=7.0.0" @@ -35,17 +35,17 @@ "lru-cache": "^6.0.0" }, "devDependencies": { - "@codemirror/autocomplete": "^6.4.0", - "@codemirror/language": "^6.4.0", - "@codemirror/lint": "^6.1.0", - "@codemirror/state": "^6.2.0", - "@codemirror/view": "^6.7.3", - "@lezer/common": "^1.0.2", - "@lezer/highlight": "^1.1.3", - "@lezer/lr": "^1.3.1", + "@codemirror/autocomplete": "^6.7.1", + "@codemirror/language": "^6.7.0", + "@codemirror/lint": "^6.2.2", + "@codemirror/state": "^6.2.1", + "@codemirror/view": "^6.13.0", + "@lezer/common": "^1.0.3", + "@lezer/highlight": "^1.1.6", + "@lezer/lr": "^1.3.6", "@types/lru-cache": "^5.1.1", "isomorphic-fetch": "^3.0.0", - "nock": "^13.3.0" + "nock": "^13.3.1" }, "engines": { "node": ">=12.0.0" @@ -64,9 +64,9 @@ "version": "0.44.0", "license": "Apache-2.0", "devDependencies": { - "@lezer/generator": "^1.2.2", - "@lezer/highlight": "^1.1.3", - "@lezer/lr": "^1.3.1" + "@lezer/generator": "^1.2.3", + "@lezer/highlight": "^1.1.6", + "@lezer/lr": "^1.3.6" }, "peerDependencies": { "@lezer/highlight": "^1.1.2", @@ -2080,9 +2080,9 @@ "dev": true }, "node_modules/@codemirror/autocomplete": { - "version": "6.4.0", - "resolved": "https://registry.npmjs.org/@codemirror/autocomplete/-/autocomplete-6.4.0.tgz", - "integrity": "sha512-HLF2PnZAm1s4kGs30EiqKMgD7XsYaQ0XJnMR0rofEWQ5t5D60SfqpDIkIh1ze5tiEbyUWm8+VJ6W1/erVvBMIA==", + "version": "6.7.1", + "resolved": "https://registry.npmjs.org/@codemirror/autocomplete/-/autocomplete-6.7.1.tgz", + "integrity": "sha512-hSxf9S0uB+GV+gBsjY1FZNo53e1FFdzPceRfCfD1gWOnV6o21GfB5J5Wg9G/4h76XZMPrF0A6OCK/Rz5+V1egg==", "dependencies": { "@codemirror/language": "^6.0.0", "@codemirror/state": "^6.0.0", @@ -2097,9 +2097,9 @@ } }, "node_modules/@codemirror/commands": { - "version": "6.2.0", - "resolved": "https://registry.npmjs.org/@codemirror/commands/-/commands-6.2.0.tgz", - "integrity": "sha512-+00smmZBradoGFEkRjliN7BjqPh/Hx0KCHWOEibUmflUqZz2RwBTU0MrVovEEHozhx3AUSGcO/rl3/5f9e9Biw==", + "version": "6.2.4", + "resolved": "https://registry.npmjs.org/@codemirror/commands/-/commands-6.2.4.tgz", + "integrity": "sha512-42lmDqVH0ttfilLShReLXsDfASKLXzfyC36bzwcqzox9PlHulMcsUOfHXNo2X2aFMVNUoQ7j+d4q5bnfseYoOA==", "dependencies": { "@codemirror/language": "^6.0.0", "@codemirror/state": "^6.2.0", @@ -2108,9 +2108,9 @@ } }, "node_modules/@codemirror/language": { - "version": "6.4.0", - "resolved": "https://registry.npmjs.org/@codemirror/language/-/language-6.4.0.tgz", - "integrity": "sha512-Wzb7GnNj8vnEtbPWiOy9H0m1fBtE28kepQNGLXekU2EEZv43BF865VKITUn+NoV8OpW6gRtvm29YEhqm46927Q==", + "version": "6.7.0", + "resolved": "https://registry.npmjs.org/@codemirror/language/-/language-6.7.0.tgz", + "integrity": "sha512-4SMwe6Fwn57klCUsVN0y4/h/iWT+XIXFEmop2lIHHuWO0ubjCrF3suqSZLyOQlznxkNnNbOOfKe5HQbQGCAmTg==", "dependencies": { "@codemirror/state": "^6.0.0", "@codemirror/view": "^6.0.0", @@ -2121,9 +2121,9 @@ } }, "node_modules/@codemirror/lint": { - "version": "6.1.0", - "resolved": "https://registry.npmjs.org/@codemirror/lint/-/lint-6.1.0.tgz", - "integrity": "sha512-mdvDQrjRmYPvQ3WrzF6Ewaao+NWERYtpthJvoQ3tK3t/44Ynhk8ZGjTSL9jMEv8CgSMogmt75X8ceOZRDSXHtQ==", + "version": "6.2.2", + "resolved": "https://registry.npmjs.org/@codemirror/lint/-/lint-6.2.2.tgz", + "integrity": "sha512-kHGuynBHjqinp1Bx25D2hgH8a6Fh1m9rSmZFzBVTqPIXDIcZ6j3VI67DY8USGYpGrjrJys9R52eLxtfERGNozg==", "dependencies": { "@codemirror/state": "^6.0.0", "@codemirror/view": "^6.0.0", @@ -2131,9 +2131,9 @@ } }, "node_modules/@codemirror/search": { - "version": "6.2.3", - "resolved": "https://registry.npmjs.org/@codemirror/search/-/search-6.2.3.tgz", - "integrity": "sha512-V9n9233lopQhB1dyjsBK2Wc1i+8hcCqxl1wQ46c5HWWLePoe4FluV3TGHoZ04rBRlGjNyz9DTmpJErig8UE4jw==", + "version": "6.5.0", + "resolved": "https://registry.npmjs.org/@codemirror/search/-/search-6.5.0.tgz", + "integrity": "sha512-64/M40YeJPToKvGO6p3fijo2vwUEj4nACEAXElCaYQ50HrXSvRaK+NHEhSh73WFBGdvIdhrV+lL9PdJy2RfCYA==", "dependencies": { "@codemirror/state": "^6.0.0", "@codemirror/view": "^6.0.0", @@ -2141,14 +2141,14 @@ } }, "node_modules/@codemirror/state": { - "version": "6.2.0", - "resolved": "https://registry.npmjs.org/@codemirror/state/-/state-6.2.0.tgz", - "integrity": "sha512-69QXtcrsc3RYtOtd+GsvczJ319udtBf1PTrr2KbLWM/e2CXUPnh0Nz9AUo8WfhSQ7GeL8dPVNUmhQVgpmuaNGA==" + "version": "6.2.1", + "resolved": "https://registry.npmjs.org/@codemirror/state/-/state-6.2.1.tgz", + "integrity": "sha512-RupHSZ8+OjNT38zU9fKH2sv+Dnlr8Eb8sl4NOnnqz95mCFTZUaiRP8Xv5MeeaG0px2b8Bnfe7YGwCV3nsBhbuw==" }, "node_modules/@codemirror/view": { - "version": "6.7.3", - "resolved": "https://registry.npmjs.org/@codemirror/view/-/view-6.7.3.tgz", - "integrity": "sha512-Lt+4POnhXrZFfHOdPzXEHxrzwdy7cjqYlMkOWvoFGi6/bAsjzlFfr0NY3B15B/PGx+cDFgM1hlc12wvYeZbGLw==", + "version": "6.13.0", + "resolved": "https://registry.npmjs.org/@codemirror/view/-/view-6.13.0.tgz", + "integrity": "sha512-oXTfJzHJ5Tl7f6T8ZO0HKf981zubxgKohjddLobbntbNZHlOZGMRL+pPZGtclDWFaFJWtGBYRGyNdjQ6Xsx5yA==", "dependencies": { "@codemirror/state": "^6.1.4", "style-mod": "^4.0.0", @@ -2476,42 +2476,33 @@ } }, "node_modules/@fortawesome/fontawesome-common-types": { - "version": "6.2.1", - "resolved": "https://registry.npmjs.org/@fortawesome/fontawesome-common-types/-/fontawesome-common-types-6.2.1.tgz", - "integrity": "sha512-Sz07mnQrTekFWLz5BMjOzHl/+NooTdW8F8kDQxjWwbpOJcnoSg4vUDng8d/WR1wOxM0O+CY9Zw0nR054riNYtQ==", + "version": "6.4.0", + "resolved": "https://registry.npmjs.org/@fortawesome/fontawesome-common-types/-/fontawesome-common-types-6.4.0.tgz", + "integrity": "sha512-HNii132xfomg5QVZw0HwXXpN22s7VBHQBv9CeOu9tfJnhsWQNd2lmTNi8CSrnw5B+5YOmzu1UoPAyxaXsJ6RgQ==", "hasInstallScript": true, "engines": { "node": ">=6" } }, "node_modules/@fortawesome/fontawesome-svg-core": { - "version": "6.2.1", - "resolved": "https://registry.npmjs.org/@fortawesome/fontawesome-svg-core/-/fontawesome-svg-core-6.2.1.tgz", - "integrity": "sha512-HELwwbCz6C1XEcjzyT1Jugmz2NNklMrSPjZOWMlc+ZsHIVk+XOvOXLGGQtFBwSyqfJDNgRq4xBCwWOaZ/d9DEA==", + "version": "6.4.0", + "resolved": "https://registry.npmjs.org/@fortawesome/fontawesome-svg-core/-/fontawesome-svg-core-6.4.0.tgz", + "integrity": "sha512-Bertv8xOiVELz5raB2FlXDPKt+m94MQ3JgDfsVbrqNpLU9+UE2E18GKjLKw+d3XbeYPqg1pzyQKGsrzbw+pPaw==", "hasInstallScript": true, "dependencies": { - "@fortawesome/fontawesome-common-types": "6.2.1" + "@fortawesome/fontawesome-common-types": "6.4.0" }, "engines": { "node": ">=6" } }, - "node_modules/@fortawesome/fontawesome-svg-core/node_modules/@fortawesome/fontawesome-common-types": { - "version": "6.2.1", - "resolved": "https://registry.npmjs.org/@fortawesome/fontawesome-common-types/-/fontawesome-common-types-6.2.1.tgz", - "integrity": "sha512-Sz07mnQrTekFWLz5BMjOzHl/+NooTdW8F8kDQxjWwbpOJcnoSg4vUDng8d/WR1wOxM0O+CY9Zw0nR054riNYtQ==", - "hasInstallScript": true, - "engines": { - "node": ">=6" - } - }, "node_modules/@fortawesome/free-solid-svg-icons": { - "version": "6.2.1", - "resolved": "https://registry.npmjs.org/@fortawesome/free-solid-svg-icons/-/free-solid-svg-icons-6.2.1.tgz", - "integrity": "sha512-oKuqrP5jbfEPJWTij4sM+/RvgX+RMFwx3QZCZcK9PrBDgxC35zuc7AOFsyMjMd/PIFPeB2JxyqDr5zs/DZFPPw==", + "version": "6.4.0", + "resolved": "https://registry.npmjs.org/@fortawesome/free-solid-svg-icons/-/free-solid-svg-icons-6.4.0.tgz", + "integrity": "sha512-kutPeRGWm8V5dltFP1zGjQOEAzaLZj4StdQhWVZnfGFCvAPVvHh8qk5bRrU4KXnRRRNni5tKQI9PBAdI6MP8nQ==", "hasInstallScript": true, "dependencies": { - "@fortawesome/fontawesome-common-types": "6.2.1" + "@fortawesome/fontawesome-common-types": "6.4.0" }, "engines": { "node": ">=6" @@ -3510,14 +3501,14 @@ "dev": true }, "node_modules/@lezer/common": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/@lezer/common/-/common-1.0.2.tgz", - "integrity": "sha512-SVgiGtMnMnW3ActR8SXgsDhw7a0w0ChHSYAyAUxxrOiJ1OqYWEKk/xJd84tTSPo1mo6DXLObAJALNnd0Hrv7Ng==" + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/@lezer/common/-/common-1.0.3.tgz", + "integrity": "sha512-JH4wAXCgUOcCGNekQPLhVeUtIqjH0yPBs7vvUdSjyQama9618IOKFJwkv2kcqdhF0my8hQEgCTEJU0GIgnahvA==" }, "node_modules/@lezer/generator": { - "version": "1.2.2", - "resolved": "https://registry.npmjs.org/@lezer/generator/-/generator-1.2.2.tgz", - "integrity": "sha512-O//eH9jTPM1GnbZruuD23xU68Pkuragonn1DEIom4Kt/eJN/QFt7Vzvp1YjV/XBmoUKC+2ySPgrA5fMF9FMM2g==", + "version": "1.2.3", + "resolved": "https://registry.npmjs.org/@lezer/generator/-/generator-1.2.3.tgz", + "integrity": "sha512-xRmNryYbJpWs7novjWtQLCGHOj71B4X1QHQ4SgJqwm11tl6COEVAGhuFTXKX16JMJUhumdXaX8We6hEMd4clDg==", "dev": true, "dependencies": { "@lezer/common": "^1.0.2", @@ -3528,17 +3519,17 @@ } }, "node_modules/@lezer/highlight": { - "version": "1.1.3", - "resolved": "https://registry.npmjs.org/@lezer/highlight/-/highlight-1.1.3.tgz", - "integrity": "sha512-3vLKLPThO4td43lYRBygmMY18JN3CPh9w+XS2j8WC30vR4yZeFG4z1iFe4jXE43NtGqe//zHW5q8ENLlHvz9gw==", + "version": "1.1.6", + "resolved": "https://registry.npmjs.org/@lezer/highlight/-/highlight-1.1.6.tgz", + "integrity": "sha512-cmSJYa2us+r3SePpRCjN5ymCqCPv+zyXmDl0ciWtVaNiORT/MxM7ZgOMQZADD0o51qOaOg24qc/zBViOIwAjJg==", "dependencies": { "@lezer/common": "^1.0.0" } }, "node_modules/@lezer/lr": { - "version": "1.3.1", - "resolved": "https://registry.npmjs.org/@lezer/lr/-/lr-1.3.1.tgz", - "integrity": "sha512-+GymJB/+3gThkk2zHwseaJTI5oa4AuOuj1I2LCslAVq1dFZLSX8SAe4ZlJq1TjezteDXtF/+d4qeWz9JvnrG9Q==", + "version": "1.3.6", + "resolved": "https://registry.npmjs.org/@lezer/lr/-/lr-1.3.6.tgz", + "integrity": "sha512-IDhcWjfxwWACnatUi0GzWBCbochfqxo3LZZlS27LbJh8RVYYXXyR5Ck9659IhkWkhSW/kZlaaiJpUO+YZTUK+Q==", "dependencies": { "@lezer/common": "^1.0.0" } @@ -4209,13 +4200,24 @@ } }, "node_modules/@types/enzyme": { - "version": "3.10.12", - "resolved": "https://registry.npmjs.org/@types/enzyme/-/enzyme-3.10.12.tgz", - "integrity": "sha512-xryQlOEIe1TduDWAOphR0ihfebKFSWOXpIsk+70JskCfRfW+xALdnJ0r1ZOTo85F9Qsjk6vtlU7edTYHbls9tA==", + "version": "3.10.13", + "resolved": "https://registry.npmjs.org/@types/enzyme/-/enzyme-3.10.13.tgz", + "integrity": "sha512-FCtoUhmFsud0Yx9fmZk179GkdZ4U9B0GFte64/Md+W/agx0L5SxsIIbhLBOxIb9y2UfBA4WQnaG1Od/UsUQs9Q==", "dev": true, "dependencies": { "@types/cheerio": "*", - "@types/react": "*" + "@types/react": "^16" + } + }, + "node_modules/@types/enzyme/node_modules/@types/react": { + "version": "16.14.42", + "resolved": "https://registry.npmjs.org/@types/react/-/react-16.14.42.tgz", + "integrity": "sha512-r6lbqQBJsQ5JJ0fp5I1+F3weosNhk7jOEcKeusIlCDYUK6kCpvIkYCamBNqGyS6WEztYlT8wmAVgblV0HxOFoA==", + "dev": true, + "dependencies": { + "@types/prop-types": "*", + "@types/scheduler": "*", + "csstype": "^3.0.2" } }, "node_modules/@types/eslint": { @@ -4330,9 +4332,9 @@ } }, "node_modules/@types/jest": { - "version": "29.4.0", - "resolved": "https://registry.npmjs.org/@types/jest/-/jest-29.4.0.tgz", - "integrity": "sha512-VaywcGQ9tPorCX/Jkkni7RWGFfI11whqzs8dvxF41P17Z+z872thvEvlIbznjPJ02kl1HMX3LmLOonsj2n7HeQ==", + "version": "29.5.2", + "resolved": "https://registry.npmjs.org/@types/jest/-/jest-29.5.2.tgz", + "integrity": "sha512-mSoZVJF5YzGVCk+FsDxzDuH7s+SCkzrgKZzf0Z0T2WudhBUPoF6ktoTPC4R0ZoCPCV5xUvuU6ias5NvxcBcMMg==", "dev": true, "dependencies": { "expect": "^29.0.0", @@ -4466,9 +4468,9 @@ } }, "node_modules/@types/react-dom": { - "version": "17.0.18", - "resolved": "https://registry.npmjs.org/@types/react-dom/-/react-dom-17.0.18.tgz", - "integrity": "sha512-rLVtIfbwyur2iFKykP2w0pl/1unw26b5td16d5xMgp7/yjTHomkyxPYChFoCr/FtEX1lN9wY6lFj1qvKdS5kDw==", + "version": "17.0.20", + "resolved": "https://registry.npmjs.org/@types/react-dom/-/react-dom-17.0.20.tgz", + "integrity": "sha512-4pzIjSxDueZZ90F52mU3aPoogkHIoSIDG+oQ+wQK7Cy2B9S+MvOqY0uEA/qawKz381qrEDkvpwyt8Bm31I8sbA==", "dev": true, "dependencies": { "@types/react": "^17" @@ -4520,9 +4522,9 @@ "dev": true }, "node_modules/@types/sanitize-html": { - "version": "2.8.0", - "resolved": "https://registry.npmjs.org/@types/sanitize-html/-/sanitize-html-2.8.0.tgz", - "integrity": "sha512-Uih6caOm3DsBYnVGOYn0A9NoTNe1c4aPStmHC/YA2JrpP9kx//jzaRcIklFvSpvVQEcpl/ZCr4DgISSf/YxTvg==", + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/@types/sanitize-html/-/sanitize-html-2.9.0.tgz", + "integrity": "sha512-4fP/kEcKNj2u39IzrxWYuf/FnCCwwQCpif6wwY6ROUS1EPRIfWJjGkY3HIowY1EX/VbX5e86yq8AAE7UPMgATg==", "dev": true, "dependencies": { "htmlparser2": "^8.0.0" @@ -4573,9 +4575,9 @@ } }, "node_modules/@types/sinon": { - "version": "10.0.13", - "resolved": "https://registry.npmjs.org/@types/sinon/-/sinon-10.0.13.tgz", - "integrity": "sha512-UVjDqJblVNQYvVNUsj0PuYYw0ELRmgt1Nt5Vk0pT5f16ROGfcKJY8o1HVuMOJOpD727RrGB9EGvoaTQE5tgxZQ==", + "version": "10.0.15", + "resolved": "https://registry.npmjs.org/@types/sinon/-/sinon-10.0.15.tgz", + "integrity": "sha512-3lrFNQG0Kr2LDzvjyjB6AMJk4ge+8iYhQfdnSwIwlG88FUOV43kPcQqDZkDa/h3WSZy6i8Fr0BSjfQtB1B3xuQ==", "dev": true, "dependencies": { "@types/sinonjs__fake-timers": "*" @@ -7557,9 +7559,9 @@ "dev": true }, "node_modules/downshift": { - "version": "7.2.0", - "resolved": "https://registry.npmjs.org/downshift/-/downshift-7.2.0.tgz", - "integrity": "sha512-dEn1Sshe7iTelUhmdbmiJhtIiwIBxBV8p15PuvEBh0qZcHXZnEt0geuCIIkCL4+ooaKRuLE0Wc+Fz9SwWuBIyg==", + "version": "7.6.0", + "resolved": "https://registry.npmjs.org/downshift/-/downshift-7.6.0.tgz", + "integrity": "sha512-VSoTVynTAsabou/hbZ6HJHUVhtBiVOjQoBsCPcQq5eAROIGP+9XKMp9asAKQ3cEcUP4oe0fFdD2pziUjhFY33Q==", "dependencies": { "@babel/runtime": "^7.14.8", "compute-scroll-into-view": "^2.0.4", @@ -7984,9 +7986,9 @@ } }, "node_modules/eslint-config-prettier": { - "version": "8.6.0", - "resolved": "https://registry.npmjs.org/eslint-config-prettier/-/eslint-config-prettier-8.6.0.tgz", - "integrity": "sha512-bAF0eLpLVqP5oEVUFKpMA+NnRFICwn9X8B5jrR9FcqnYBuPbqWEjTEspPWMj5ye6czoSLDweCzSo3Ko7gGrZaA==", + "version": "8.8.0", + "resolved": "https://registry.npmjs.org/eslint-config-prettier/-/eslint-config-prettier-8.8.0.tgz", + "integrity": "sha512-wLbQiFre3tdGgpDv67NQKnJuTlcUVYHas3k+DZCc2U2BadthoEY4B7hLPvAxaqdyOGCzuLfii2fqGph10va7oA==", "dev": true, "bin": { "eslint-config-prettier": "bin/cli.js" @@ -10550,9 +10552,9 @@ } }, "node_modules/jest-canvas-mock": { - "version": "2.4.0", - "resolved": "https://registry.npmjs.org/jest-canvas-mock/-/jest-canvas-mock-2.4.0.tgz", - "integrity": "sha512-mmMpZzpmLzn5vepIaHk5HoH3Ka4WykbSoLuG/EKoJd0x0ID/t+INo1l8ByfcUJuDM+RIsL4QDg/gDnBbrj2/IQ==", + "version": "2.5.1", + "resolved": "https://registry.npmjs.org/jest-canvas-mock/-/jest-canvas-mock-2.5.1.tgz", + "integrity": "sha512-IVnRiz+v4EYn3ydM/pBo8GW/J+nU/Hg5gHBQQOUQhdRyNfvHnabB8ReqARLO0p+kvQghqr4V0tA92CF3JcUSRg==", "dev": true, "dependencies": { "cssfontparser": "^1.2.1", @@ -12836,9 +12838,9 @@ } }, "node_modules/jquery": { - "version": "3.6.3", - "resolved": "https://registry.npmjs.org/jquery/-/jquery-3.6.3.tgz", - "integrity": "sha512-bZ5Sy3YzKo9Fyc8wH2iIQK4JImJ6R0GWI9kL1/k7Z91ZBNgkRXE6U0JfHIizZbort8ZunhSI3jw9I6253ahKfg==" + "version": "3.7.0", + "resolved": "https://registry.npmjs.org/jquery/-/jquery-3.7.0.tgz", + "integrity": "sha512-umpJ0/k8X0MvD1ds0P9SfowREz2LenHsQaxSohMZ5OMNEU2r0tf8pdeEFTHMFxWVxKNyU9rTtK3CWzUCTKJUeQ==" }, "node_modules/jquery.flot.tooltip": { "version": "0.9.0", @@ -13488,11 +13490,11 @@ } }, "node_modules/moment-timezone": { - "version": "0.5.40", - "resolved": "https://registry.npmjs.org/moment-timezone/-/moment-timezone-0.5.40.tgz", - "integrity": "sha512-tWfmNkRYmBkPJz5mr9GVDn9vRlVZOTe6yqY92rFxiOdWXbjaR0+9LwQnZGGuNR63X456NqmEkbskte8tWL5ePg==", + "version": "0.5.43", + "resolved": "https://registry.npmjs.org/moment-timezone/-/moment-timezone-0.5.43.tgz", + "integrity": "sha512-72j3aNyuIsDxdF1i7CEgV2FfxM1r6aaqJyLB2vwb33mXYyoyLly+F1zbWqhA3/bVIoJ4szlUoMbUnVdid32NUQ==", "dependencies": { - "moment": ">= 2.9.0" + "moment": "^2.29.4" }, "engines": { "node": "*" @@ -13640,9 +13642,9 @@ } }, "node_modules/nock": { - "version": "13.3.0", - "resolved": "https://registry.npmjs.org/nock/-/nock-13.3.0.tgz", - "integrity": "sha512-HHqYQ6mBeiMc+N038w8LkMpDCRquCHWeNmN3v6645P3NhN2+qXOBqvPqo7Rt1VyCMzKhJ733wZqw5B7cQVFNPg==", + "version": "13.3.1", + "resolved": "https://registry.npmjs.org/nock/-/nock-13.3.1.tgz", + "integrity": "sha512-vHnopocZuI93p2ccivFyGuUfzjq2fxNyNurp7816mlT5V5HF4SzXu8lvLrVzBbNqzs+ODooZ6OksuSUNM7Njkw==", "dev": true, "dependencies": { "debug": "^4.1.0", @@ -15708,9 +15710,9 @@ } }, "node_modules/prettier": { - "version": "2.8.3", - "resolved": "https://registry.npmjs.org/prettier/-/prettier-2.8.3.tgz", - "integrity": "sha512-tJ/oJ4amDihPoufT5sM0Z1SKEuKay8LfVAMlbbhnnkvt6BUserZylqo2PN+p9KeljLr0OHa2rXHU1T8reeoTrw==", + "version": "2.8.8", + "resolved": "https://registry.npmjs.org/prettier/-/prettier-2.8.8.tgz", + "integrity": "sha512-tdN8qQGvNjw4CHbY+XXk0JgCXn9QiF21a55rBe5LJAU+kDyC4WQn4+awm2Xfk2lQMk5fKup9XgzTZtGkjBdP9Q==", "dev": true, "bin": { "prettier": "bin-prettier.js" @@ -18022,9 +18024,9 @@ "dev": true }, "node_modules/sanitize-html": { - "version": "2.8.1", - "resolved": "https://registry.npmjs.org/sanitize-html/-/sanitize-html-2.8.1.tgz", - "integrity": "sha512-qK5neD0SaMxGwVv5txOYv05huC3o6ZAA4h5+7nJJgWMNFUNRjcjLO6FpwAtKzfKCZ0jrG6xTk6eVFskbvOGblg==", + "version": "2.10.0", + "resolved": "https://registry.npmjs.org/sanitize-html/-/sanitize-html-2.10.0.tgz", + "integrity": "sha512-JqdovUd81dG4k87vZt6uA6YhDfWkUGruUu/aPmXLxXi45gZExnt9Bnw/qeQU8oGf82vPyaE0vO4aH0PbobB9JQ==", "dependencies": { "deepmerge": "^4.2.2", "escape-string-regexp": "^4.0.0", @@ -18059,9 +18061,9 @@ "dev": true }, "node_modules/sass": { - "version": "1.57.1", - "resolved": "https://registry.npmjs.org/sass/-/sass-1.57.1.tgz", - "integrity": "sha512-O2+LwLS79op7GI0xZ8fqzF7X2m/m8WFfI02dHOdsK5R2ECeS5F62zrwg/relM1rjSLy7Vd/DiMNIvPrQGsA0jw==", + "version": "1.62.1", + "resolved": "https://registry.npmjs.org/sass/-/sass-1.62.1.tgz", + "integrity": "sha512-NHpxIzN29MXvWiuswfc1W3I0N8SXBd8UR26WntmDlRYf0bSADnwnOjsyMZ3lMezSlArD33Vs3YFhp7dWvL770A==", "dependencies": { "chokidar": ">=3.0.0 <4.0.0", "immutable": "^4.0.0", @@ -18071,7 +18073,7 @@ "sass": "sass.js" }, "engines": { - "node": ">=12.0.0" + "node": ">=14.0.0" } }, "node_modules/sass-loader": { @@ -19378,9 +19380,9 @@ "dev": true }, "node_modules/ts-jest": { - "version": "29.0.5", - "resolved": "https://registry.npmjs.org/ts-jest/-/ts-jest-29.0.5.tgz", - "integrity": "sha512-PL3UciSgIpQ7f6XjVOmbi96vmDHUqAyqDr8YxzopDqX3kfgYtX1cuNeBjP+L9sFXi6nzsGGA6R3fP3DDDJyrxA==", + "version": "29.1.0", + "resolved": "https://registry.npmjs.org/ts-jest/-/ts-jest-29.1.0.tgz", + "integrity": "sha512-ZhNr7Z4PcYa+JjMl62ir+zPiNJfXJN6E8hSLnaUKhOgqcn8vb3e537cpkd0FuAfRK3sR1LSqM1MOhliXNgOFPA==", "dev": true, "dependencies": { "bs-logger": "0.x", @@ -19403,7 +19405,7 @@ "@jest/types": "^29.0.0", "babel-jest": "^29.0.0", "jest": "^29.0.0", - "typescript": ">=4.3" + "typescript": ">=4.3 <6" }, "peerDependenciesMeta": { "@babel/core": { @@ -19572,9 +19574,9 @@ } }, "node_modules/typescript": { - "version": "4.9.4", - "resolved": "https://registry.npmjs.org/typescript/-/typescript-4.9.4.tgz", - "integrity": "sha512-Uz+dTXYzxXXbsFpM86Wh3dKCxrQqUcVMxwU54orwlJjOpO3ao8L7j5lH+dWfTwgCwIuM9GQ2kvVotzYJMXTBZg==", + "version": "4.9.5", + "resolved": "https://registry.npmjs.org/typescript/-/typescript-4.9.5.tgz", + "integrity": "sha512-1FXk9E2Hm+QzZQ7z+McJiHL4NW1F2EzMu9Nq9i3zAaGqibafqYwCVU6WyWAuyQRRzOlxou8xZSyXLEN8oKj24g==", "dev": true, "bin": { "tsc": "bin/tsc", @@ -20765,31 +20767,31 @@ "name": "@prometheus-io/app", "version": "0.44.0", "dependencies": { - "@codemirror/autocomplete": "^6.4.0", - "@codemirror/commands": "^6.2.0", - "@codemirror/language": "^6.4.0", - "@codemirror/lint": "^6.1.0", - "@codemirror/search": "^6.2.3", - "@codemirror/state": "^6.2.0", - "@codemirror/view": "^6.7.3", + "@codemirror/autocomplete": "^6.7.1", + "@codemirror/commands": "^6.2.4", + "@codemirror/language": "^6.7.0", + "@codemirror/lint": "^6.2.2", + "@codemirror/search": "^6.5.0", + "@codemirror/state": "^6.2.1", + "@codemirror/view": "^6.13.0", "@forevolve/bootstrap-dark": "^2.1.1", - "@fortawesome/fontawesome-svg-core": "6.2.1", - "@fortawesome/free-solid-svg-icons": "6.2.1", + "@fortawesome/fontawesome-svg-core": "6.4.0", + "@fortawesome/free-solid-svg-icons": "6.4.0", "@fortawesome/react-fontawesome": "0.2.0", - "@lezer/common": "^1.0.2", - "@lezer/highlight": "^1.1.3", - "@lezer/lr": "^1.3.1", + "@lezer/common": "^1.0.3", + "@lezer/highlight": "^1.1.6", + "@lezer/lr": "^1.3.6", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", "@prometheus-io/codemirror-promql": "0.44.0", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", - "downshift": "^7.2.0", + "downshift": "^7.6.0", "http-proxy-middleware": "^2.0.6", - "jquery": "^3.6.3", + "jquery": "^3.7.0", "jquery.flot.tooltip": "^0.9.0", "moment": "^2.29.4", - "moment-timezone": "^0.5.40", + "moment-timezone": "^0.5.43", "popper.js": "^1.14.3", "react": "^17.0.2", "react-copy-to-clipboard": "^5.1.0", @@ -20799,22 +20801,22 @@ "react-router-dom": "^5.3.4", "react-test-renderer": "^17.0.2", "reactstrap": "^8.10.1", - "sanitize-html": "^2.8.1", - "sass": "1.57.1", + "sanitize-html": "^2.10.0", + "sass": "1.62.1", "tempusdominus-bootstrap-4": "^5.39.2", "tempusdominus-core": "^5.19.3" }, "devDependencies": { "@testing-library/react-hooks": "^7.0.2", - "@types/enzyme": "^3.10.12", + "@types/enzyme": "^3.10.13", "@types/flot": "0.0.32", "@types/jquery": "^3.5.16", - "@types/react": "^17.0.53", + "@types/react": "^17.0.60", "@types/react-copy-to-clipboard": "^5.0.4", - "@types/react-dom": "^17.0.18", + "@types/react-dom": "^17.0.20", "@types/react-router-dom": "^5.3.3", - "@types/sanitize-html": "^2.8.0", - "@types/sinon": "^10.0.13", + "@types/sanitize-html": "^2.9.0", + "@types/sinon": "^10.0.15", "@wojtekmaj/enzyme-adapter-react-17": "^0.8.0", "enzyme": "^3.11.0", "enzyme-to-json": "^3.6.2", @@ -20824,6 +20826,17 @@ "optionalDependencies": { "fsevents": "^2.3.2" } + }, + "react-app/node_modules/@types/react": { + "version": "17.0.60", + "resolved": "https://registry.npmjs.org/@types/react/-/react-17.0.60.tgz", + "integrity": "sha512-pCH7bqWIfzHs3D+PDs3O/COCQJka+Kcw3RnO9rFA2zalqoXg7cNjJDh6mZ7oRtY1wmY4LVwDdAbA1F7Z8tv3BQ==", + "dev": true, + "dependencies": { + "@types/prop-types": "*", + "@types/scheduler": "*", + "csstype": "^3.0.2" + } } }, "dependencies": { @@ -22224,9 +22237,9 @@ "dev": true }, "@codemirror/autocomplete": { - "version": "6.4.0", - "resolved": "https://registry.npmjs.org/@codemirror/autocomplete/-/autocomplete-6.4.0.tgz", - "integrity": "sha512-HLF2PnZAm1s4kGs30EiqKMgD7XsYaQ0XJnMR0rofEWQ5t5D60SfqpDIkIh1ze5tiEbyUWm8+VJ6W1/erVvBMIA==", + "version": "6.7.1", + "resolved": "https://registry.npmjs.org/@codemirror/autocomplete/-/autocomplete-6.7.1.tgz", + "integrity": "sha512-hSxf9S0uB+GV+gBsjY1FZNo53e1FFdzPceRfCfD1gWOnV6o21GfB5J5Wg9G/4h76XZMPrF0A6OCK/Rz5+V1egg==", "requires": { "@codemirror/language": "^6.0.0", "@codemirror/state": "^6.0.0", @@ -22235,9 +22248,9 @@ } }, "@codemirror/commands": { - "version": "6.2.0", - "resolved": "https://registry.npmjs.org/@codemirror/commands/-/commands-6.2.0.tgz", - "integrity": "sha512-+00smmZBradoGFEkRjliN7BjqPh/Hx0KCHWOEibUmflUqZz2RwBTU0MrVovEEHozhx3AUSGcO/rl3/5f9e9Biw==", + "version": "6.2.4", + "resolved": "https://registry.npmjs.org/@codemirror/commands/-/commands-6.2.4.tgz", + "integrity": "sha512-42lmDqVH0ttfilLShReLXsDfASKLXzfyC36bzwcqzox9PlHulMcsUOfHXNo2X2aFMVNUoQ7j+d4q5bnfseYoOA==", "requires": { "@codemirror/language": "^6.0.0", "@codemirror/state": "^6.2.0", @@ -22246,9 +22259,9 @@ } }, "@codemirror/language": { - "version": "6.4.0", - "resolved": "https://registry.npmjs.org/@codemirror/language/-/language-6.4.0.tgz", - "integrity": "sha512-Wzb7GnNj8vnEtbPWiOy9H0m1fBtE28kepQNGLXekU2EEZv43BF865VKITUn+NoV8OpW6gRtvm29YEhqm46927Q==", + "version": "6.7.0", + "resolved": "https://registry.npmjs.org/@codemirror/language/-/language-6.7.0.tgz", + "integrity": "sha512-4SMwe6Fwn57klCUsVN0y4/h/iWT+XIXFEmop2lIHHuWO0ubjCrF3suqSZLyOQlznxkNnNbOOfKe5HQbQGCAmTg==", "requires": { "@codemirror/state": "^6.0.0", "@codemirror/view": "^6.0.0", @@ -22259,9 +22272,9 @@ } }, "@codemirror/lint": { - "version": "6.1.0", - "resolved": "https://registry.npmjs.org/@codemirror/lint/-/lint-6.1.0.tgz", - "integrity": "sha512-mdvDQrjRmYPvQ3WrzF6Ewaao+NWERYtpthJvoQ3tK3t/44Ynhk8ZGjTSL9jMEv8CgSMogmt75X8ceOZRDSXHtQ==", + "version": "6.2.2", + "resolved": "https://registry.npmjs.org/@codemirror/lint/-/lint-6.2.2.tgz", + "integrity": "sha512-kHGuynBHjqinp1Bx25D2hgH8a6Fh1m9rSmZFzBVTqPIXDIcZ6j3VI67DY8USGYpGrjrJys9R52eLxtfERGNozg==", "requires": { "@codemirror/state": "^6.0.0", "@codemirror/view": "^6.0.0", @@ -22269,9 +22282,9 @@ } }, "@codemirror/search": { - "version": "6.2.3", - "resolved": "https://registry.npmjs.org/@codemirror/search/-/search-6.2.3.tgz", - "integrity": "sha512-V9n9233lopQhB1dyjsBK2Wc1i+8hcCqxl1wQ46c5HWWLePoe4FluV3TGHoZ04rBRlGjNyz9DTmpJErig8UE4jw==", + "version": "6.5.0", + "resolved": "https://registry.npmjs.org/@codemirror/search/-/search-6.5.0.tgz", + "integrity": "sha512-64/M40YeJPToKvGO6p3fijo2vwUEj4nACEAXElCaYQ50HrXSvRaK+NHEhSh73WFBGdvIdhrV+lL9PdJy2RfCYA==", "requires": { "@codemirror/state": "^6.0.0", "@codemirror/view": "^6.0.0", @@ -22279,14 +22292,14 @@ } }, "@codemirror/state": { - "version": "6.2.0", - "resolved": "https://registry.npmjs.org/@codemirror/state/-/state-6.2.0.tgz", - "integrity": "sha512-69QXtcrsc3RYtOtd+GsvczJ319udtBf1PTrr2KbLWM/e2CXUPnh0Nz9AUo8WfhSQ7GeL8dPVNUmhQVgpmuaNGA==" + "version": "6.2.1", + "resolved": "https://registry.npmjs.org/@codemirror/state/-/state-6.2.1.tgz", + "integrity": "sha512-RupHSZ8+OjNT38zU9fKH2sv+Dnlr8Eb8sl4NOnnqz95mCFTZUaiRP8Xv5MeeaG0px2b8Bnfe7YGwCV3nsBhbuw==" }, "@codemirror/view": { - "version": "6.7.3", - "resolved": "https://registry.npmjs.org/@codemirror/view/-/view-6.7.3.tgz", - "integrity": "sha512-Lt+4POnhXrZFfHOdPzXEHxrzwdy7cjqYlMkOWvoFGi6/bAsjzlFfr0NY3B15B/PGx+cDFgM1hlc12wvYeZbGLw==", + "version": "6.13.0", + "resolved": "https://registry.npmjs.org/@codemirror/view/-/view-6.13.0.tgz", + "integrity": "sha512-oXTfJzHJ5Tl7f6T8ZO0HKf981zubxgKohjddLobbntbNZHlOZGMRL+pPZGtclDWFaFJWtGBYRGyNdjQ6Xsx5yA==", "requires": { "@codemirror/state": "^6.1.4", "style-mod": "^4.0.0", @@ -22463,31 +22476,24 @@ } }, "@fortawesome/fontawesome-common-types": { - "version": "6.2.1", - "resolved": "https://registry.npmjs.org/@fortawesome/fontawesome-common-types/-/fontawesome-common-types-6.2.1.tgz", - "integrity": "sha512-Sz07mnQrTekFWLz5BMjOzHl/+NooTdW8F8kDQxjWwbpOJcnoSg4vUDng8d/WR1wOxM0O+CY9Zw0nR054riNYtQ==" + "version": "6.4.0", + "resolved": "https://registry.npmjs.org/@fortawesome/fontawesome-common-types/-/fontawesome-common-types-6.4.0.tgz", + "integrity": "sha512-HNii132xfomg5QVZw0HwXXpN22s7VBHQBv9CeOu9tfJnhsWQNd2lmTNi8CSrnw5B+5YOmzu1UoPAyxaXsJ6RgQ==" }, "@fortawesome/fontawesome-svg-core": { - "version": "6.2.1", - "resolved": "https://registry.npmjs.org/@fortawesome/fontawesome-svg-core/-/fontawesome-svg-core-6.2.1.tgz", - "integrity": "sha512-HELwwbCz6C1XEcjzyT1Jugmz2NNklMrSPjZOWMlc+ZsHIVk+XOvOXLGGQtFBwSyqfJDNgRq4xBCwWOaZ/d9DEA==", + "version": "6.4.0", + "resolved": "https://registry.npmjs.org/@fortawesome/fontawesome-svg-core/-/fontawesome-svg-core-6.4.0.tgz", + "integrity": "sha512-Bertv8xOiVELz5raB2FlXDPKt+m94MQ3JgDfsVbrqNpLU9+UE2E18GKjLKw+d3XbeYPqg1pzyQKGsrzbw+pPaw==", "requires": { - "@fortawesome/fontawesome-common-types": "6.2.1" - }, - "dependencies": { - "@fortawesome/fontawesome-common-types": { - "version": "6.2.1", - "resolved": "https://registry.npmjs.org/@fortawesome/fontawesome-common-types/-/fontawesome-common-types-6.2.1.tgz", - "integrity": "sha512-Sz07mnQrTekFWLz5BMjOzHl/+NooTdW8F8kDQxjWwbpOJcnoSg4vUDng8d/WR1wOxM0O+CY9Zw0nR054riNYtQ==" - } + "@fortawesome/fontawesome-common-types": "6.4.0" } }, "@fortawesome/free-solid-svg-icons": { - "version": "6.2.1", - "resolved": "https://registry.npmjs.org/@fortawesome/free-solid-svg-icons/-/free-solid-svg-icons-6.2.1.tgz", - "integrity": "sha512-oKuqrP5jbfEPJWTij4sM+/RvgX+RMFwx3QZCZcK9PrBDgxC35zuc7AOFsyMjMd/PIFPeB2JxyqDr5zs/DZFPPw==", + "version": "6.4.0", + "resolved": "https://registry.npmjs.org/@fortawesome/free-solid-svg-icons/-/free-solid-svg-icons-6.4.0.tgz", + "integrity": "sha512-kutPeRGWm8V5dltFP1zGjQOEAzaLZj4StdQhWVZnfGFCvAPVvHh8qk5bRrU4KXnRRRNni5tKQI9PBAdI6MP8nQ==", "requires": { - "@fortawesome/fontawesome-common-types": "6.2.1" + "@fortawesome/fontawesome-common-types": "6.4.0" } }, "@fortawesome/react-fontawesome": { @@ -23277,14 +23283,14 @@ "dev": true }, "@lezer/common": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/@lezer/common/-/common-1.0.2.tgz", - "integrity": "sha512-SVgiGtMnMnW3ActR8SXgsDhw7a0w0ChHSYAyAUxxrOiJ1OqYWEKk/xJd84tTSPo1mo6DXLObAJALNnd0Hrv7Ng==" + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/@lezer/common/-/common-1.0.3.tgz", + "integrity": "sha512-JH4wAXCgUOcCGNekQPLhVeUtIqjH0yPBs7vvUdSjyQama9618IOKFJwkv2kcqdhF0my8hQEgCTEJU0GIgnahvA==" }, "@lezer/generator": { - "version": "1.2.2", - "resolved": "https://registry.npmjs.org/@lezer/generator/-/generator-1.2.2.tgz", - "integrity": "sha512-O//eH9jTPM1GnbZruuD23xU68Pkuragonn1DEIom4Kt/eJN/QFt7Vzvp1YjV/XBmoUKC+2ySPgrA5fMF9FMM2g==", + "version": "1.2.3", + "resolved": "https://registry.npmjs.org/@lezer/generator/-/generator-1.2.3.tgz", + "integrity": "sha512-xRmNryYbJpWs7novjWtQLCGHOj71B4X1QHQ4SgJqwm11tl6COEVAGhuFTXKX16JMJUhumdXaX8We6hEMd4clDg==", "dev": true, "requires": { "@lezer/common": "^1.0.2", @@ -23292,17 +23298,17 @@ } }, "@lezer/highlight": { - "version": "1.1.3", - "resolved": "https://registry.npmjs.org/@lezer/highlight/-/highlight-1.1.3.tgz", - "integrity": "sha512-3vLKLPThO4td43lYRBygmMY18JN3CPh9w+XS2j8WC30vR4yZeFG4z1iFe4jXE43NtGqe//zHW5q8ENLlHvz9gw==", + "version": "1.1.6", + "resolved": "https://registry.npmjs.org/@lezer/highlight/-/highlight-1.1.6.tgz", + "integrity": "sha512-cmSJYa2us+r3SePpRCjN5ymCqCPv+zyXmDl0ciWtVaNiORT/MxM7ZgOMQZADD0o51qOaOg24qc/zBViOIwAjJg==", "requires": { "@lezer/common": "^1.0.0" } }, "@lezer/lr": { - "version": "1.3.1", - "resolved": "https://registry.npmjs.org/@lezer/lr/-/lr-1.3.1.tgz", - "integrity": "sha512-+GymJB/+3gThkk2zHwseaJTI5oa4AuOuj1I2LCslAVq1dFZLSX8SAe4ZlJq1TjezteDXtF/+d4qeWz9JvnrG9Q==", + "version": "1.3.6", + "resolved": "https://registry.npmjs.org/@lezer/lr/-/lr-1.3.6.tgz", + "integrity": "sha512-IDhcWjfxwWACnatUi0GzWBCbochfqxo3LZZlS27LbJh8RVYYXXyR5Ck9659IhkWkhSW/kZlaaiJpUO+YZTUK+Q==", "requires": { "@lezer/common": "^1.0.0" } @@ -23401,45 +23407,45 @@ "@prometheus-io/app": { "version": "file:react-app", "requires": { - "@codemirror/autocomplete": "^6.4.0", - "@codemirror/commands": "^6.2.0", - "@codemirror/language": "^6.4.0", - "@codemirror/lint": "^6.1.0", - "@codemirror/search": "^6.2.3", - "@codemirror/state": "^6.2.0", - "@codemirror/view": "^6.7.3", + "@codemirror/autocomplete": "^6.7.1", + "@codemirror/commands": "^6.2.4", + "@codemirror/language": "^6.7.0", + "@codemirror/lint": "^6.2.2", + "@codemirror/search": "^6.5.0", + "@codemirror/state": "^6.2.1", + "@codemirror/view": "^6.13.0", "@forevolve/bootstrap-dark": "^2.1.1", - "@fortawesome/fontawesome-svg-core": "6.2.1", - "@fortawesome/free-solid-svg-icons": "6.2.1", + "@fortawesome/fontawesome-svg-core": "6.4.0", + "@fortawesome/free-solid-svg-icons": "6.4.0", "@fortawesome/react-fontawesome": "0.2.0", - "@lezer/common": "^1.0.2", - "@lezer/highlight": "^1.1.3", - "@lezer/lr": "^1.3.1", + "@lezer/common": "^1.0.3", + "@lezer/highlight": "^1.1.6", + "@lezer/lr": "^1.3.6", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", "@prometheus-io/codemirror-promql": "0.44.0", "@testing-library/react-hooks": "^7.0.2", - "@types/enzyme": "^3.10.12", + "@types/enzyme": "^3.10.13", "@types/flot": "0.0.32", "@types/jquery": "^3.5.16", - "@types/react": "^17.0.53", + "@types/react": "^17.0.60", "@types/react-copy-to-clipboard": "^5.0.4", - "@types/react-dom": "^17.0.18", + "@types/react-dom": "^17.0.20", "@types/react-router-dom": "^5.3.3", - "@types/sanitize-html": "^2.8.0", - "@types/sinon": "^10.0.13", + "@types/sanitize-html": "^2.9.0", + "@types/sinon": "^10.0.15", "@wojtekmaj/enzyme-adapter-react-17": "^0.8.0", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", - "downshift": "^7.2.0", + "downshift": "^7.6.0", "enzyme": "^3.11.0", "enzyme-to-json": "^3.6.2", "fsevents": "^2.3.2", "http-proxy-middleware": "^2.0.6", - "jquery": "^3.6.3", + "jquery": "^3.7.0", "jquery.flot.tooltip": "^0.9.0", "moment": "^2.29.4", - "moment-timezone": "^0.5.40", + "moment-timezone": "^0.5.43", "mutationobserver-shim": "^0.3.7", "popper.js": "^1.14.3", "react": "^17.0.2", @@ -23450,37 +23456,50 @@ "react-router-dom": "^5.3.4", "react-test-renderer": "^17.0.2", "reactstrap": "^8.10.1", - "sanitize-html": "^2.8.1", - "sass": "1.57.1", + "sanitize-html": "^2.10.0", + "sass": "1.62.1", "sinon": "^14.0.2", "tempusdominus-bootstrap-4": "^5.39.2", "tempusdominus-core": "^5.19.3" + }, + "dependencies": { + "@types/react": { + "version": "17.0.60", + "resolved": "https://registry.npmjs.org/@types/react/-/react-17.0.60.tgz", + "integrity": "sha512-pCH7bqWIfzHs3D+PDs3O/COCQJka+Kcw3RnO9rFA2zalqoXg7cNjJDh6mZ7oRtY1wmY4LVwDdAbA1F7Z8tv3BQ==", + "dev": true, + "requires": { + "@types/prop-types": "*", + "@types/scheduler": "*", + "csstype": "^3.0.2" + } + } } }, "@prometheus-io/codemirror-promql": { "version": "file:module/codemirror-promql", "requires": { - "@codemirror/autocomplete": "^6.4.0", - "@codemirror/language": "^6.4.0", - "@codemirror/lint": "^6.1.0", - "@codemirror/state": "^6.2.0", - "@codemirror/view": "^6.7.3", - "@lezer/common": "^1.0.2", - "@lezer/highlight": "^1.1.3", - "@lezer/lr": "^1.3.1", + "@codemirror/autocomplete": "^6.7.1", + "@codemirror/language": "^6.7.0", + "@codemirror/lint": "^6.2.2", + "@codemirror/state": "^6.2.1", + "@codemirror/view": "^6.13.0", + "@lezer/common": "^1.0.3", + "@lezer/highlight": "^1.1.6", + "@lezer/lr": "^1.3.6", "@prometheus-io/lezer-promql": "0.44.0", "@types/lru-cache": "^5.1.1", "isomorphic-fetch": "^3.0.0", "lru-cache": "^6.0.0", - "nock": "^13.3.0" + "nock": "^13.3.1" } }, "@prometheus-io/lezer-promql": { "version": "file:module/lezer-promql", "requires": { - "@lezer/generator": "^1.2.2", - "@lezer/highlight": "^1.1.3", - "@lezer/lr": "^1.3.1" + "@lezer/generator": "^1.2.3", + "@lezer/highlight": "^1.1.6", + "@lezer/lr": "^1.3.6" } }, "@rollup/plugin-babel": { @@ -23851,13 +23870,26 @@ } }, "@types/enzyme": { - "version": "3.10.12", - "resolved": "https://registry.npmjs.org/@types/enzyme/-/enzyme-3.10.12.tgz", - "integrity": "sha512-xryQlOEIe1TduDWAOphR0ihfebKFSWOXpIsk+70JskCfRfW+xALdnJ0r1ZOTo85F9Qsjk6vtlU7edTYHbls9tA==", + "version": "3.10.13", + "resolved": "https://registry.npmjs.org/@types/enzyme/-/enzyme-3.10.13.tgz", + "integrity": "sha512-FCtoUhmFsud0Yx9fmZk179GkdZ4U9B0GFte64/Md+W/agx0L5SxsIIbhLBOxIb9y2UfBA4WQnaG1Od/UsUQs9Q==", "dev": true, "requires": { "@types/cheerio": "*", - "@types/react": "*" + "@types/react": "^16" + }, + "dependencies": { + "@types/react": { + "version": "16.14.42", + "resolved": "https://registry.npmjs.org/@types/react/-/react-16.14.42.tgz", + "integrity": "sha512-r6lbqQBJsQ5JJ0fp5I1+F3weosNhk7jOEcKeusIlCDYUK6kCpvIkYCamBNqGyS6WEztYlT8wmAVgblV0HxOFoA==", + "dev": true, + "requires": { + "@types/prop-types": "*", + "@types/scheduler": "*", + "csstype": "^3.0.2" + } + } } }, "@types/eslint": { @@ -23972,9 +24004,9 @@ } }, "@types/jest": { - "version": "29.4.0", - "resolved": "https://registry.npmjs.org/@types/jest/-/jest-29.4.0.tgz", - "integrity": "sha512-VaywcGQ9tPorCX/Jkkni7RWGFfI11whqzs8dvxF41P17Z+z872thvEvlIbznjPJ02kl1HMX3LmLOonsj2n7HeQ==", + "version": "29.5.2", + "resolved": "https://registry.npmjs.org/@types/jest/-/jest-29.5.2.tgz", + "integrity": "sha512-mSoZVJF5YzGVCk+FsDxzDuH7s+SCkzrgKZzf0Z0T2WudhBUPoF6ktoTPC4R0ZoCPCV5xUvuU6ias5NvxcBcMMg==", "dev": true, "requires": { "expect": "^29.0.0", @@ -24101,9 +24133,9 @@ } }, "@types/react-dom": { - "version": "17.0.18", - "resolved": "https://registry.npmjs.org/@types/react-dom/-/react-dom-17.0.18.tgz", - "integrity": "sha512-rLVtIfbwyur2iFKykP2w0pl/1unw26b5td16d5xMgp7/yjTHomkyxPYChFoCr/FtEX1lN9wY6lFj1qvKdS5kDw==", + "version": "17.0.20", + "resolved": "https://registry.npmjs.org/@types/react-dom/-/react-dom-17.0.20.tgz", + "integrity": "sha512-4pzIjSxDueZZ90F52mU3aPoogkHIoSIDG+oQ+wQK7Cy2B9S+MvOqY0uEA/qawKz381qrEDkvpwyt8Bm31I8sbA==", "dev": true, "requires": { "@types/react": "^17" @@ -24155,9 +24187,9 @@ "dev": true }, "@types/sanitize-html": { - "version": "2.8.0", - "resolved": "https://registry.npmjs.org/@types/sanitize-html/-/sanitize-html-2.8.0.tgz", - "integrity": "sha512-Uih6caOm3DsBYnVGOYn0A9NoTNe1c4aPStmHC/YA2JrpP9kx//jzaRcIklFvSpvVQEcpl/ZCr4DgISSf/YxTvg==", + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/@types/sanitize-html/-/sanitize-html-2.9.0.tgz", + "integrity": "sha512-4fP/kEcKNj2u39IzrxWYuf/FnCCwwQCpif6wwY6ROUS1EPRIfWJjGkY3HIowY1EX/VbX5e86yq8AAE7UPMgATg==", "dev": true, "requires": { "htmlparser2": "^8.0.0" @@ -24203,9 +24235,9 @@ } }, "@types/sinon": { - "version": "10.0.13", - "resolved": "https://registry.npmjs.org/@types/sinon/-/sinon-10.0.13.tgz", - "integrity": "sha512-UVjDqJblVNQYvVNUsj0PuYYw0ELRmgt1Nt5Vk0pT5f16ROGfcKJY8o1HVuMOJOpD727RrGB9EGvoaTQE5tgxZQ==", + "version": "10.0.15", + "resolved": "https://registry.npmjs.org/@types/sinon/-/sinon-10.0.15.tgz", + "integrity": "sha512-3lrFNQG0Kr2LDzvjyjB6AMJk4ge+8iYhQfdnSwIwlG88FUOV43kPcQqDZkDa/h3WSZy6i8Fr0BSjfQtB1B3xuQ==", "dev": true, "requires": { "@types/sinonjs__fake-timers": "*" @@ -26441,9 +26473,9 @@ "dev": true }, "downshift": { - "version": "7.2.0", - "resolved": "https://registry.npmjs.org/downshift/-/downshift-7.2.0.tgz", - "integrity": "sha512-dEn1Sshe7iTelUhmdbmiJhtIiwIBxBV8p15PuvEBh0qZcHXZnEt0geuCIIkCL4+ooaKRuLE0Wc+Fz9SwWuBIyg==", + "version": "7.6.0", + "resolved": "https://registry.npmjs.org/downshift/-/downshift-7.6.0.tgz", + "integrity": "sha512-VSoTVynTAsabou/hbZ6HJHUVhtBiVOjQoBsCPcQq5eAROIGP+9XKMp9asAKQ3cEcUP4oe0fFdD2pziUjhFY33Q==", "requires": { "@babel/runtime": "^7.14.8", "compute-scroll-into-view": "^2.0.4", @@ -26779,9 +26811,9 @@ } }, "eslint-config-prettier": { - "version": "8.6.0", - "resolved": "https://registry.npmjs.org/eslint-config-prettier/-/eslint-config-prettier-8.6.0.tgz", - "integrity": "sha512-bAF0eLpLVqP5oEVUFKpMA+NnRFICwn9X8B5jrR9FcqnYBuPbqWEjTEspPWMj5ye6czoSLDweCzSo3Ko7gGrZaA==", + "version": "8.8.0", + "resolved": "https://registry.npmjs.org/eslint-config-prettier/-/eslint-config-prettier-8.8.0.tgz", + "integrity": "sha512-wLbQiFre3tdGgpDv67NQKnJuTlcUVYHas3k+DZCc2U2BadthoEY4B7hLPvAxaqdyOGCzuLfii2fqGph10va7oA==", "dev": true, "requires": {} }, @@ -28650,9 +28682,9 @@ } }, "jest-canvas-mock": { - "version": "2.4.0", - "resolved": "https://registry.npmjs.org/jest-canvas-mock/-/jest-canvas-mock-2.4.0.tgz", - "integrity": "sha512-mmMpZzpmLzn5vepIaHk5HoH3Ka4WykbSoLuG/EKoJd0x0ID/t+INo1l8ByfcUJuDM+RIsL4QDg/gDnBbrj2/IQ==", + "version": "2.5.1", + "resolved": "https://registry.npmjs.org/jest-canvas-mock/-/jest-canvas-mock-2.5.1.tgz", + "integrity": "sha512-IVnRiz+v4EYn3ydM/pBo8GW/J+nU/Hg5gHBQQOUQhdRyNfvHnabB8ReqARLO0p+kvQghqr4V0tA92CF3JcUSRg==", "dev": true, "requires": { "cssfontparser": "^1.2.1", @@ -30498,9 +30530,9 @@ } }, "jquery": { - "version": "3.6.3", - "resolved": "https://registry.npmjs.org/jquery/-/jquery-3.6.3.tgz", - "integrity": "sha512-bZ5Sy3YzKo9Fyc8wH2iIQK4JImJ6R0GWI9kL1/k7Z91ZBNgkRXE6U0JfHIizZbort8ZunhSI3jw9I6253ahKfg==" + "version": "3.7.0", + "resolved": "https://registry.npmjs.org/jquery/-/jquery-3.7.0.tgz", + "integrity": "sha512-umpJ0/k8X0MvD1ds0P9SfowREz2LenHsQaxSohMZ5OMNEU2r0tf8pdeEFTHMFxWVxKNyU9rTtK3CWzUCTKJUeQ==" }, "jquery.flot.tooltip": { "version": "0.9.0", @@ -31015,11 +31047,11 @@ "integrity": "sha512-5LC9SOxjSc2HF6vO2CyuTDNivEdoz2IvyJJGj6X8DJ0eFyfszE0QiEd+iXmBvUP3WHxSjFH/vIsA0EN00cgr8w==" }, "moment-timezone": { - "version": "0.5.40", - "resolved": "https://registry.npmjs.org/moment-timezone/-/moment-timezone-0.5.40.tgz", - "integrity": "sha512-tWfmNkRYmBkPJz5mr9GVDn9vRlVZOTe6yqY92rFxiOdWXbjaR0+9LwQnZGGuNR63X456NqmEkbskte8tWL5ePg==", + "version": "0.5.43", + "resolved": "https://registry.npmjs.org/moment-timezone/-/moment-timezone-0.5.43.tgz", + "integrity": "sha512-72j3aNyuIsDxdF1i7CEgV2FfxM1r6aaqJyLB2vwb33mXYyoyLly+F1zbWqhA3/bVIoJ4szlUoMbUnVdid32NUQ==", "requires": { - "moment": ">= 2.9.0" + "moment": "^2.29.4" } }, "moo": { @@ -31146,9 +31178,9 @@ } }, "nock": { - "version": "13.3.0", - "resolved": "https://registry.npmjs.org/nock/-/nock-13.3.0.tgz", - "integrity": "sha512-HHqYQ6mBeiMc+N038w8LkMpDCRquCHWeNmN3v6645P3NhN2+qXOBqvPqo7Rt1VyCMzKhJ733wZqw5B7cQVFNPg==", + "version": "13.3.1", + "resolved": "https://registry.npmjs.org/nock/-/nock-13.3.1.tgz", + "integrity": "sha512-vHnopocZuI93p2ccivFyGuUfzjq2fxNyNurp7816mlT5V5HF4SzXu8lvLrVzBbNqzs+ODooZ6OksuSUNM7Njkw==", "dev": true, "requires": { "debug": "^4.1.0", @@ -32479,9 +32511,9 @@ "dev": true }, "prettier": { - "version": "2.8.3", - "resolved": "https://registry.npmjs.org/prettier/-/prettier-2.8.3.tgz", - "integrity": "sha512-tJ/oJ4amDihPoufT5sM0Z1SKEuKay8LfVAMlbbhnnkvt6BUserZylqo2PN+p9KeljLr0OHa2rXHU1T8reeoTrw==", + "version": "2.8.8", + "resolved": "https://registry.npmjs.org/prettier/-/prettier-2.8.8.tgz", + "integrity": "sha512-tdN8qQGvNjw4CHbY+XXk0JgCXn9QiF21a55rBe5LJAU+kDyC4WQn4+awm2Xfk2lQMk5fKup9XgzTZtGkjBdP9Q==", "dev": true }, "prettier-linter-helpers": { @@ -34265,9 +34297,9 @@ "dev": true }, "sanitize-html": { - "version": "2.8.1", - "resolved": "https://registry.npmjs.org/sanitize-html/-/sanitize-html-2.8.1.tgz", - "integrity": "sha512-qK5neD0SaMxGwVv5txOYv05huC3o6ZAA4h5+7nJJgWMNFUNRjcjLO6FpwAtKzfKCZ0jrG6xTk6eVFskbvOGblg==", + "version": "2.10.0", + "resolved": "https://registry.npmjs.org/sanitize-html/-/sanitize-html-2.10.0.tgz", + "integrity": "sha512-JqdovUd81dG4k87vZt6uA6YhDfWkUGruUu/aPmXLxXi45gZExnt9Bnw/qeQU8oGf82vPyaE0vO4aH0PbobB9JQ==", "requires": { "deepmerge": "^4.2.2", "escape-string-regexp": "^4.0.0", @@ -34297,9 +34329,9 @@ "dev": true }, "sass": { - "version": "1.57.1", - "resolved": "https://registry.npmjs.org/sass/-/sass-1.57.1.tgz", - "integrity": "sha512-O2+LwLS79op7GI0xZ8fqzF7X2m/m8WFfI02dHOdsK5R2ECeS5F62zrwg/relM1rjSLy7Vd/DiMNIvPrQGsA0jw==", + "version": "1.62.1", + "resolved": "https://registry.npmjs.org/sass/-/sass-1.62.1.tgz", + "integrity": "sha512-NHpxIzN29MXvWiuswfc1W3I0N8SXBd8UR26WntmDlRYf0bSADnwnOjsyMZ3lMezSlArD33Vs3YFhp7dWvL770A==", "requires": { "chokidar": ">=3.0.0 <4.0.0", "immutable": "^4.0.0", @@ -35312,9 +35344,9 @@ "dev": true }, "ts-jest": { - "version": "29.0.5", - "resolved": "https://registry.npmjs.org/ts-jest/-/ts-jest-29.0.5.tgz", - "integrity": "sha512-PL3UciSgIpQ7f6XjVOmbi96vmDHUqAyqDr8YxzopDqX3kfgYtX1cuNeBjP+L9sFXi6nzsGGA6R3fP3DDDJyrxA==", + "version": "29.1.0", + "resolved": "https://registry.npmjs.org/ts-jest/-/ts-jest-29.1.0.tgz", + "integrity": "sha512-ZhNr7Z4PcYa+JjMl62ir+zPiNJfXJN6E8hSLnaUKhOgqcn8vb3e537cpkd0FuAfRK3sR1LSqM1MOhliXNgOFPA==", "dev": true, "requires": { "bs-logger": "0.x", @@ -35449,9 +35481,9 @@ } }, "typescript": { - "version": "4.9.4", - "resolved": "https://registry.npmjs.org/typescript/-/typescript-4.9.4.tgz", - "integrity": "sha512-Uz+dTXYzxXXbsFpM86Wh3dKCxrQqUcVMxwU54orwlJjOpO3ao8L7j5lH+dWfTwgCwIuM9GQ2kvVotzYJMXTBZg==", + "version": "4.9.5", + "resolved": "https://registry.npmjs.org/typescript/-/typescript-4.9.5.tgz", + "integrity": "sha512-1FXk9E2Hm+QzZQ7z+McJiHL4NW1F2EzMu9Nq9i3zAaGqibafqYwCVU6WyWAuyQRRzOlxou8xZSyXLEN8oKj24g==", "dev": true }, "unbox-primitive": { diff --git a/web/ui/package.json b/web/ui/package.json index 22f2615ff8..d5663ee75e 100644 --- a/web/ui/package.json +++ b/web/ui/package.json @@ -16,16 +16,16 @@ "npm": ">=7.0.0" }, "devDependencies": { - "@types/jest": "^29.4.0", + "@types/jest": "^29.5.2", "@types/node": "^17.0.45", - "eslint-config-prettier": "^8.6.0", + "eslint-config-prettier": "^8.8.0", "eslint-config-react-app": "^7.0.1", "eslint-plugin-prettier": "^4.2.1", - "jest-canvas-mock": "^2.4.0", + "jest-canvas-mock": "^2.5.1", "jest-fetch-mock": "^3.0.3", "react-scripts": "^5.0.1", - "prettier": "^2.8.3", - "ts-jest": "^29.0.5", - "typescript": "^4.9.4" + "prettier": "^2.8.8", + "ts-jest": "^29.1.0", + "typescript": "^4.9.5" } } diff --git a/web/ui/react-app/package.json b/web/ui/react-app/package.json index fc0d1e5fc3..d67209d8ef 100644 --- a/web/ui/react-app/package.json +++ b/web/ui/react-app/package.json @@ -3,31 +3,31 @@ "version": "0.44.0", "private": true, "dependencies": { - "@codemirror/autocomplete": "^6.4.0", - "@codemirror/commands": "^6.2.0", - "@codemirror/language": "^6.4.0", - "@codemirror/lint": "^6.1.0", - "@codemirror/search": "^6.2.3", - "@codemirror/state": "^6.2.0", - "@codemirror/view": "^6.7.3", + "@codemirror/autocomplete": "^6.7.1", + "@codemirror/commands": "^6.2.4", + "@codemirror/language": "^6.7.0", + "@codemirror/lint": "^6.2.2", + "@codemirror/search": "^6.5.0", + "@codemirror/state": "^6.2.1", + "@codemirror/view": "^6.13.0", "@forevolve/bootstrap-dark": "^2.1.1", - "@fortawesome/fontawesome-svg-core": "6.2.1", - "@fortawesome/free-solid-svg-icons": "6.2.1", + "@fortawesome/fontawesome-svg-core": "6.4.0", + "@fortawesome/free-solid-svg-icons": "6.4.0", "@fortawesome/react-fontawesome": "0.2.0", - "@lezer/lr": "^1.3.1", - "@lezer/highlight": "^1.1.3", - "@lezer/common": "^1.0.2", + "@lezer/lr": "^1.3.6", + "@lezer/highlight": "^1.1.6", + "@lezer/common": "^1.0.3", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", "@prometheus-io/codemirror-promql": "0.44.0", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", - "downshift": "^7.2.0", + "downshift": "^7.6.0", "http-proxy-middleware": "^2.0.6", - "jquery": "^3.6.3", + "jquery": "^3.7.0", "jquery.flot.tooltip": "^0.9.0", "moment": "^2.29.4", - "moment-timezone": "^0.5.40", + "moment-timezone": "^0.5.43", "popper.js": "^1.14.3", "react": "^17.0.2", "react-copy-to-clipboard": "^5.1.0", @@ -37,8 +37,8 @@ "react-router-dom": "^5.3.4", "react-test-renderer": "^17.0.2", "reactstrap": "^8.10.1", - "sanitize-html": "^2.8.1", - "sass": "1.57.1", + "sanitize-html": "^2.10.0", + "sass": "1.62.1", "tempusdominus-bootstrap-4": "^5.39.2", "tempusdominus-core": "^5.19.3" }, @@ -66,15 +66,15 @@ ], "devDependencies": { "@testing-library/react-hooks": "^7.0.2", - "@types/enzyme": "^3.10.12", + "@types/enzyme": "^3.10.13", "@types/flot": "0.0.32", "@types/jquery": "^3.5.16", - "@types/react": "^17.0.53", + "@types/react": "^17.0.60", "@types/react-copy-to-clipboard": "^5.0.4", - "@types/react-dom": "^17.0.18", + "@types/react-dom": "^17.0.20", "@types/react-router-dom": "^5.3.3", - "@types/sanitize-html": "^2.8.0", - "@types/sinon": "^10.0.13", + "@types/sanitize-html": "^2.9.0", + "@types/sinon": "^10.0.15", "@wojtekmaj/enzyme-adapter-react-17": "^0.8.0", "enzyme": "^3.11.0", "enzyme-to-json": "^3.6.2", diff --git a/web/web.go b/web/web.go index 27378b3b81..b9af2819b7 100644 --- a/web/web.go +++ b/web/web.go @@ -744,7 +744,7 @@ func (h *Handler) runtimeInfo() (api_v1.RuntimeInfo, error) { } func toFloat64(f *io_prometheus_client.MetricFamily) float64 { - m := *f.Metric[0] + m := f.Metric[0] if m.Gauge != nil { return m.Gauge.GetValue() } From 4268feb9d7b6be75359e9fe0dcf765cf73fedd17 Mon Sep 17 00:00:00 2001 From: Leo Q Date: Wed, 7 Jun 2023 20:28:13 +0800 Subject: [PATCH 200/231] add alert for sd refresh failure (#12410) * add alert for sd refresh failure Due to config error or sd service down, prometheus may fail to refresh sd resource, which may lead to scrape fail or irrelavant metrics. Signed-off-by: Leo Q * apply suggestions Signed-off-by: Leo Q --------- Signed-off-by: Leo Q --- documentation/prometheus-mixin/alerts.libsonnet | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/documentation/prometheus-mixin/alerts.libsonnet b/documentation/prometheus-mixin/alerts.libsonnet index 0ee5d83c7a..3efb0f27d1 100644 --- a/documentation/prometheus-mixin/alerts.libsonnet +++ b/documentation/prometheus-mixin/alerts.libsonnet @@ -20,6 +20,20 @@ description: 'Prometheus %(prometheusName)s has failed to reload its configuration.' % $._config, }, }, + { + alert: 'PrometheusSDRefreshFailure', + expr: ||| + increase(prometheus_sd_refresh_failures_total{%(prometheusSelector)s}[10m]) > 0 + ||| % $._config, + 'for': '20m', + labels: { + severity: 'warning', + }, + annotations: { + summary: 'Failed Prometheus SD refresh.', + description: 'Prometheus %(prometheusName)s has failed to refresh SD with mechanism {{$labels.mechanism}}.' % $._config, + }, + }, { alert: 'PrometheusNotificationQueueRunningFull', expr: ||| From edfc97a77e74d0f3996fed16c8a1b29c2b10ed35 Mon Sep 17 00:00:00 2001 From: Jesus Vazquez Date: Wed, 7 Jun 2023 16:00:15 +0200 Subject: [PATCH 201/231] Bump UI version (#12440) Signed-off-by: Jesus Vazquez --- web/ui/module/codemirror-promql/package.json | 4 ++-- web/ui/module/lezer-promql/package.json | 2 +- web/ui/package-lock.json | 14 +++++++------- web/ui/react-app/package.json | 4 ++-- 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/web/ui/module/codemirror-promql/package.json b/web/ui/module/codemirror-promql/package.json index e9c952c9cf..6c404a6c46 100644 --- a/web/ui/module/codemirror-promql/package.json +++ b/web/ui/module/codemirror-promql/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/codemirror-promql", - "version": "0.44.0", + "version": "0.45.0-rc.0", "description": "a CodeMirror mode for the PromQL language", "types": "dist/esm/index.d.ts", "module": "dist/esm/index.js", @@ -29,7 +29,7 @@ }, "homepage": "https://github.com/prometheus/prometheus/blob/main/web/ui/module/codemirror-promql/README.md", "dependencies": { - "@prometheus-io/lezer-promql": "0.44.0", + "@prometheus-io/lezer-promql": "0.45.0-rc.0", "lru-cache": "^6.0.0" }, "devDependencies": { diff --git a/web/ui/module/lezer-promql/package.json b/web/ui/module/lezer-promql/package.json index 08249b418d..389eb88f4f 100644 --- a/web/ui/module/lezer-promql/package.json +++ b/web/ui/module/lezer-promql/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/lezer-promql", - "version": "0.44.0", + "version": "0.45.0-rc.0", "description": "lezer-based PromQL grammar", "main": "dist/index.cjs", "type": "module", diff --git a/web/ui/package-lock.json b/web/ui/package-lock.json index a672a17673..f80d2fd6e4 100644 --- a/web/ui/package-lock.json +++ b/web/ui/package-lock.json @@ -28,10 +28,10 @@ }, "module/codemirror-promql": { "name": "@prometheus-io/codemirror-promql", - "version": "0.44.0", + "version": "0.45.0-rc.0", "license": "Apache-2.0", "dependencies": { - "@prometheus-io/lezer-promql": "0.44.0", + "@prometheus-io/lezer-promql": "0.45.0-rc.0", "lru-cache": "^6.0.0" }, "devDependencies": { @@ -61,7 +61,7 @@ }, "module/lezer-promql": { "name": "@prometheus-io/lezer-promql", - "version": "0.44.0", + "version": "0.45.0-rc.0", "license": "Apache-2.0", "devDependencies": { "@lezer/generator": "^1.2.3", @@ -20765,7 +20765,7 @@ }, "react-app": { "name": "@prometheus-io/app", - "version": "0.44.0", + "version": "0.45.0-rc.0", "dependencies": { "@codemirror/autocomplete": "^6.7.1", "@codemirror/commands": "^6.2.4", @@ -20783,7 +20783,7 @@ "@lezer/lr": "^1.3.6", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.44.0", + "@prometheus-io/codemirror-promql": "0.45.0-rc.0", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", "downshift": "^7.6.0", @@ -23423,7 +23423,7 @@ "@lezer/lr": "^1.3.6", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.44.0", + "@prometheus-io/codemirror-promql": "0.45.0-rc.0", "@testing-library/react-hooks": "^7.0.2", "@types/enzyme": "^3.10.13", "@types/flot": "0.0.32", @@ -23487,7 +23487,7 @@ "@lezer/common": "^1.0.3", "@lezer/highlight": "^1.1.6", "@lezer/lr": "^1.3.6", - "@prometheus-io/lezer-promql": "0.44.0", + "@prometheus-io/lezer-promql": "0.45.0-rc.0", "@types/lru-cache": "^5.1.1", "isomorphic-fetch": "^3.0.0", "lru-cache": "^6.0.0", diff --git a/web/ui/react-app/package.json b/web/ui/react-app/package.json index d67209d8ef..d8efd56810 100644 --- a/web/ui/react-app/package.json +++ b/web/ui/react-app/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/app", - "version": "0.44.0", + "version": "0.45.0-rc.0", "private": true, "dependencies": { "@codemirror/autocomplete": "^6.7.1", @@ -19,7 +19,7 @@ "@lezer/common": "^1.0.3", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.44.0", + "@prometheus-io/codemirror-promql": "0.45.0-rc.0", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", "downshift": "^7.6.0", From 344c8ff97ce261dbaaf2720f1e5164a8fee19184 Mon Sep 17 00:00:00 2001 From: Michael Hoffmann Date: Wed, 7 Jun 2023 22:54:30 +0200 Subject: [PATCH 202/231] feat: dont compile regex matcher if we know its a literal (#12434) labels: dont compile regex matcher if we know its a literal Signed-off-by: Michael Hoffmann Co-authored-by: Sharad --- model/labels/matcher_test.go | 18 ++++++++++++++---- model/labels/regexp.go | 17 +++++++++++++++++ 2 files changed, 31 insertions(+), 4 deletions(-) diff --git a/model/labels/matcher_test.go b/model/labels/matcher_test.go index 14615a50d1..d26e9329f2 100644 --- a/model/labels/matcher_test.go +++ b/model/labels/matcher_test.go @@ -102,12 +102,12 @@ func TestInverse(t *testing.T) { expected: &Matcher{Type: MatchEqual, Name: "name2", Value: "value2"}, }, { - matcher: &Matcher{Type: MatchRegexp, Name: "name3", Value: "value3"}, - expected: &Matcher{Type: MatchNotRegexp, Name: "name3", Value: "value3"}, + matcher: &Matcher{Type: MatchRegexp, Name: "name3", Value: "value3.*"}, + expected: &Matcher{Type: MatchNotRegexp, Name: "name3", Value: "value3.*"}, }, { - matcher: &Matcher{Type: MatchNotRegexp, Name: "name4", Value: "value4"}, - expected: &Matcher{Type: MatchRegexp, Name: "name4", Value: "value4"}, + matcher: &Matcher{Type: MatchNotRegexp, Name: "name4", Value: "value4.*"}, + expected: &Matcher{Type: MatchRegexp, Name: "name4", Value: "value4.*"}, }, } @@ -123,3 +123,13 @@ func BenchmarkMatchType_String(b *testing.B) { _ = MatchType(i % int(MatchNotRegexp+1)).String() } } + +func BenchmarkNewMatcher(b *testing.B) { + b.Run("regex matcher with literal", func(b *testing.B) { + b.ReportAllocs() + b.ResetTimer() + for i := 0; i <= b.N; i++ { + NewMatcher(MatchRegexp, "foo", "bar") + } + }) +} diff --git a/model/labels/regexp.go b/model/labels/regexp.go index e09a63772f..14319c7f7a 100644 --- a/model/labels/regexp.go +++ b/model/labels/regexp.go @@ -25,9 +25,16 @@ type FastRegexMatcher struct { prefix string suffix string contains string + + // shortcut for literals + literal bool + value string } func NewFastRegexMatcher(v string) (*FastRegexMatcher, error) { + if isLiteral(v) { + return &FastRegexMatcher{literal: true, value: v}, nil + } re, err := regexp.Compile("^(?:" + v + ")$") if err != nil { return nil, err @@ -50,6 +57,9 @@ func NewFastRegexMatcher(v string) (*FastRegexMatcher, error) { } func (m *FastRegexMatcher) MatchString(s string) bool { + if m.literal { + return s == m.value + } if m.prefix != "" && !strings.HasPrefix(s, m.prefix) { return false } @@ -63,9 +73,16 @@ func (m *FastRegexMatcher) MatchString(s string) bool { } func (m *FastRegexMatcher) GetRegexString() string { + if m.literal { + return m.value + } return m.re.String() } +func isLiteral(re string) bool { + return regexp.QuoteMeta(re) == re +} + // optimizeConcatRegex returns literal prefix/suffix text that can be safely // checked against the label value before running the regexp matcher. func optimizeConcatRegex(r *syntax.Regexp) (prefix, suffix, contains string) { From f7abe27fef92adc3239c7c3e4006f570b56e71b8 Mon Sep 17 00:00:00 2001 From: sinkingpoint Date: Fri, 9 Jun 2023 14:06:09 +1000 Subject: [PATCH 203/231] Remove trailing commas from Exemplar API docs The trailing commas here make this example invalid JSON. Here we remove them. Signed-off-by: sinkingpoint --- docs/querying/api.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/querying/api.md b/docs/querying/api.md index edce366ee6..b2b72d9cde 100644 --- a/docs/querying/api.md +++ b/docs/querying/api.md @@ -407,7 +407,7 @@ $ curl -g 'http://localhost:9090/api/v1/query_exemplars?query=test_exemplar_metr "traceID": "EpTxMJ40fUus7aGY" }, "value": "6", - "timestamp": 1600096945.479, + "timestamp": 1600096945.479 } ] }, @@ -424,15 +424,15 @@ $ curl -g 'http://localhost:9090/api/v1/query_exemplars?query=test_exemplar_metr "traceID": "Olp9XHlq763ccsfa" }, "value": "19", - "timestamp": 1600096955.479, + "timestamp": 1600096955.479 }, { "labels": { "traceID": "hCtjygkIHwAN9vs4" }, "value": "20", - "timestamp": 1600096965.489, - }, + "timestamp": 1600096965.489 + } ] } ] From 1ea477f4bcd04c580e51711dca346a06421b311c Mon Sep 17 00:00:00 2001 From: Arthur Silva Sens Date: Mon, 12 Jun 2023 12:17:20 -0300 Subject: [PATCH 204/231] Add feature flag to squash metadata from /api/v1/metadata (#12391) Signed-off-by: ArthurSens --- docs/querying/api.md | 27 +++++++ web/api/v1/api.go | 16 ++++- web/api/v1/api_test.go | 160 ++++++++++++++++++++++++++++++++++++++--- 3 files changed, 193 insertions(+), 10 deletions(-) diff --git a/docs/querying/api.md b/docs/querying/api.md index b2b72d9cde..ca7f64f622 100644 --- a/docs/querying/api.md +++ b/docs/querying/api.md @@ -863,6 +863,7 @@ GET /api/v1/metadata URL query parameters: - `limit=`: Maximum number of metrics to return. +- `limit_per_metric=`: Maximum number of metadata to return per metric. - `metric=`: A metric name to filter metadata for. All metric metadata is retrieved if left empty. The `data` section of the query result consists of an object where each key is a metric name and each value is a list of unique metadata objects, as exposed for that metric name across all targets. @@ -898,6 +899,32 @@ curl -G http://localhost:9090/api/v1/metadata?limit=2 } ``` +The following example returns only one metadata entry for each metric. + +```json +curl -G http://localhost:9090/api/v1/metadata?limit_per_metric=1 + +{ + "status": "success", + "data": { + "cortex_ring_tokens": [ + { + "type": "gauge", + "help": "Number of tokens in the ring", + "unit": "" + } + ], + "http_requests_total": [ + { + "type": "counter", + "help": "Number of HTTP requests", + "unit": "" + } + ] + } +} +``` + The following example returns metadata only for the metric `http_requests_total`. ```json diff --git a/web/api/v1/api.go b/web/api/v1/api.go index f7249efb04..c74f6ee7ae 100644 --- a/web/api/v1/api.go +++ b/web/api/v1/api.go @@ -1194,16 +1194,26 @@ func (api *API) metricMetadata(r *http.Request) apiFuncResult { return apiFuncResult{nil, &apiError{errorBadData, errors.New("limit must be a number")}, nil, nil} } } + limitPerMetric := -1 + if s := r.FormValue("limit_per_metric"); s != "" { + var err error + if limitPerMetric, err = strconv.Atoi(s); err != nil { + return apiFuncResult{nil, &apiError{errorBadData, errors.New("limit_per_metric must be a number")}, nil, nil} + } + } metric := r.FormValue("metric") for _, tt := range api.targetRetriever(r.Context()).TargetsActive() { for _, t := range tt { - if metric == "" { for _, mm := range t.MetadataList() { m := metadata{Type: mm.Type, Help: mm.Help, Unit: mm.Unit} ms, ok := metrics[mm.Metric] + if limitPerMetric > 0 && len(ms) >= limitPerMetric { + continue + } + if !ok { ms = map[metadata]struct{}{} metrics[mm.Metric] = ms @@ -1217,6 +1227,10 @@ func (api *API) metricMetadata(r *http.Request) apiFuncResult { m := metadata{Type: md.Type, Help: md.Help, Unit: md.Unit} ms, ok := metrics[md.Metric] + if limitPerMetric > 0 && len(ms) >= limitPerMetric { + continue + } + if !ok { ms = map[metadata]struct{}{} metrics[md.Metric] = ms diff --git a/web/api/v1/api_test.go b/web/api/v1/api_test.go index baee2189ef..16e74071c1 100644 --- a/web/api/v1/api_test.go +++ b/web/api/v1/api_test.go @@ -1023,15 +1023,16 @@ func testEndpoints(t *testing.T, api *API, tr *testTargetRetriever, es storage.E } type test struct { - endpoint apiFunc - params map[string]string - query url.Values - response interface{} - responseLen int - errType errorType - sorter func(interface{}) - metadata []targetMetadata - exemplars []exemplar.QueryResult + endpoint apiFunc + params map[string]string + query url.Values + response interface{} + responseLen int + responseMetadataTotal int + errType errorType + sorter func(interface{}) + metadata []targetMetadata + exemplars []exemplar.QueryResult } tests := []test{ @@ -1776,6 +1777,126 @@ func testEndpoints(t *testing.T, api *API, tr *testTargetRetriever, es storage.E }, responseLen: 2, }, + // With a limit for the number of metadata per metric. + { + endpoint: api.metricMetadata, + query: url.Values{"limit_per_metric": []string{"1"}}, + metadata: []targetMetadata{ + { + identifier: "test", + metadata: []scrape.MetricMetadata{ + { + Metric: "go_threads", + Type: textparse.MetricTypeGauge, + Help: "Number of OS threads created", + Unit: "", + }, + { + Metric: "go_threads", + Type: textparse.MetricTypeGauge, + Help: "Repeated metadata", + Unit: "", + }, + { + Metric: "go_gc_duration_seconds", + Type: textparse.MetricTypeSummary, + Help: "A summary of the GC invocation durations.", + Unit: "", + }, + }, + }, + }, + response: map[string][]metadata{ + "go_threads": { + {textparse.MetricTypeGauge, "Number of OS threads created", ""}, + }, + "go_gc_duration_seconds": { + {textparse.MetricTypeSummary, "A summary of the GC invocation durations.", ""}, + }, + }, + }, + // With a limit for the number of metadata per metric and per metric. + { + endpoint: api.metricMetadata, + query: url.Values{"limit_per_metric": []string{"1"}, "limit": []string{"1"}}, + metadata: []targetMetadata{ + { + identifier: "test", + metadata: []scrape.MetricMetadata{ + { + Metric: "go_threads", + Type: textparse.MetricTypeGauge, + Help: "Number of OS threads created", + Unit: "", + }, + { + Metric: "go_threads", + Type: textparse.MetricTypeGauge, + Help: "Repeated metadata", + Unit: "", + }, + { + Metric: "go_gc_duration_seconds", + Type: textparse.MetricTypeSummary, + Help: "A summary of the GC invocation durations.", + Unit: "", + }, + }, + }, + }, + responseLen: 1, + responseMetadataTotal: 1, + }, + + // With a limit for the number of metadata per metric and per metric, while having multiple targets. + { + endpoint: api.metricMetadata, + query: url.Values{"limit_per_metric": []string{"1"}, "limit": []string{"1"}}, + metadata: []targetMetadata{ + { + identifier: "test", + metadata: []scrape.MetricMetadata{ + { + Metric: "go_threads", + Type: textparse.MetricTypeGauge, + Help: "Number of OS threads created", + Unit: "", + }, + { + Metric: "go_threads", + Type: textparse.MetricTypeGauge, + Help: "Repeated metadata", + Unit: "", + }, + { + Metric: "go_gc_duration_seconds", + Type: textparse.MetricTypeSummary, + Help: "A summary of the GC invocation durations.", + Unit: "", + }, + }, + }, + { + identifier: "secondTarget", + metadata: []scrape.MetricMetadata{ + { + Metric: "go_threads", + Type: textparse.MetricTypeGauge, + Help: "Number of OS threads created, but from a different target", + Unit: "", + }, + { + Metric: "go_gc_duration_seconds", + Type: textparse.MetricTypeSummary, + Help: "A summary of the GC invocation durations, but from a different target.", + Unit: "", + }, + }, + }, + }, + responseLen: 1, + responseMetadataTotal: 1, + }, // When requesting a specific metric that is present. { endpoint: api.metricMetadata, @@ -2565,6 +2686,9 @@ func testEndpoints(t *testing.T, api *API, tr *testTargetRetriever, es storage.E if test.responseLen != 0 { assertAPIResponseLength(t, res.data, test.responseLen) + if test.responseMetadataTotal != 0 { + assertAPIResponseMetadataLen(t, res.data, test.responseMetadataTotal) + } } else { assertAPIResponse(t, res.data, test.response) } @@ -2615,6 +2739,24 @@ func assertAPIResponseLength(t *testing.T, got interface{}, expLen int) { } } +func assertAPIResponseMetadataLen(t *testing.T, got interface{}, expLen int) { + t.Helper() + + var gotLen int + response := got.(map[string][]metadata) + for _, m := range response { + gotLen += len(m) + } + + if gotLen != expLen { + t.Fatalf( + "Amount of metadata in the response does not match, expected:\n%d\ngot:\n%d", + expLen, + gotLen, + ) + } +} + type fakeDB struct { err error } From 81bf3e63a4663e27a935bab0167f73b9da3afeb5 Mon Sep 17 00:00:00 2001 From: Jayapriya Pai Date: Mon, 12 Jun 2023 21:39:19 +0530 Subject: [PATCH 205/231] docs: update prometheus-operator link Signed-off-by: Jayapriya Pai --- docs/configuration/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configuration/configuration.md b/docs/configuration/configuration.md index 3a9ace2b6c..4287272f65 100644 --- a/docs/configuration/configuration.md +++ b/docs/configuration/configuration.md @@ -2160,7 +2160,7 @@ attach_metadata: See [this example Prometheus configuration file](/documentation/examples/prometheus-kubernetes.yml) for a detailed example of configuring Prometheus for Kubernetes. -You may wish to check out the 3rd party [Prometheus Operator](https://github.com/coreos/prometheus-operator), +You may wish to check out the 3rd party [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator), which automates the Prometheus setup on top of Kubernetes. ### `` From 06843db80a58bc679cf3c1863891abdd1c7cdeba Mon Sep 17 00:00:00 2001 From: Bartol Deak Date: Mon, 12 Jun 2023 22:56:44 +0200 Subject: [PATCH 206/231] Hide `which` stderr output Signed-off-by: Bartol Deak --- Makefile | 2 +- Makefile.common | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 3877ee719e..2816c98014 100644 --- a/Makefile +++ b/Makefile @@ -82,7 +82,7 @@ assets-tarball: assets .PHONY: parser parser: @echo ">> running goyacc to generate the .go file." -ifeq (, $(shell which goyacc)) +ifeq (, $(shell which goyacc 2>/dev/null)) @echo "goyacc not installed so skipping" @echo "To install: go install golang.org/x/tools/cmd/goyacc@v0.6.0" else diff --git a/Makefile.common b/Makefile.common index b111d25620..2bae0efab5 100644 --- a/Makefile.common +++ b/Makefile.common @@ -178,7 +178,7 @@ endif .PHONY: common-yamllint common-yamllint: @echo ">> running yamllint on all YAML files in the repository" -ifeq (, $(shell which yamllint)) +ifeq (, $(shell which yamllint 2>/dev/null)) @echo "yamllint not installed so skipping" else yamllint . From 6a18962cfac3445693c78b157976e31c2bb81f12 Mon Sep 17 00:00:00 2001 From: Oleg Zaytsev Date: Tue, 13 Jun 2023 10:38:00 +0200 Subject: [PATCH 207/231] mv labels_string.go labels_stringlabels.go (#12328) This is a minor cosmetical change, but my IDE (and I guess many of them) nests `labels_string.go` under `labels.go` because it assumes it's the file generated by the `stringer` tool, which follows that naming pattern. Signed-off-by: Oleg Zaytsev --- model/labels/{labels_string.go => labels_stringlabels.go} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename model/labels/{labels_string.go => labels_stringlabels.go} (100%) diff --git a/model/labels/labels_string.go b/model/labels/labels_stringlabels.go similarity index 100% rename from model/labels/labels_string.go rename to model/labels/labels_stringlabels.go From 95b7d592bae8542c58efa92bb67777a047add231 Mon Sep 17 00:00:00 2001 From: Bartol Deak Date: Tue, 13 Jun 2023 11:57:52 +0200 Subject: [PATCH 208/231] rewrite `which` with `command -v` Signed-off-by: Bartol Deak --- Makefile | 2 +- Makefile.common | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index 2816c98014..0dd8673af3 100644 --- a/Makefile +++ b/Makefile @@ -82,7 +82,7 @@ assets-tarball: assets .PHONY: parser parser: @echo ">> running goyacc to generate the .go file." -ifeq (, $(shell which goyacc 2>/dev/null)) +ifeq (, $(shell command -v goyacc > /dev/null)) @echo "goyacc not installed so skipping" @echo "To install: go install golang.org/x/tools/cmd/goyacc@v0.6.0" else diff --git a/Makefile.common b/Makefile.common index 2bae0efab5..e372d34738 100644 --- a/Makefile.common +++ b/Makefile.common @@ -49,7 +49,7 @@ endif GOTEST := $(GO) test GOTEST_DIR := ifneq ($(CIRCLE_JOB),) -ifneq ($(shell which gotestsum),) +ifneq ($(shell command -v gotestsum > /dev/null),) GOTEST_DIR := test-results GOTEST := gotestsum --junitfile $(GOTEST_DIR)/unit-tests.xml -- endif @@ -178,7 +178,7 @@ endif .PHONY: common-yamllint common-yamllint: @echo ">> running yamllint on all YAML files in the repository" -ifeq (, $(shell which yamllint 2>/dev/null)) +ifeq (, $(shell command -v yamllint > /dev/null)) @echo "yamllint not installed so skipping" else yamllint . From 0941ea4afcc6df7d70fa9578bc9403c5fec8ea1c Mon Sep 17 00:00:00 2001 From: tyltr Date: Fri, 16 Jun 2023 11:09:19 +0800 Subject: [PATCH 209/231] typo Signed-off-by: tyltr --- cmd/prometheus/main.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmd/prometheus/main.go b/cmd/prometheus/main.go index e05ac79570..3d723f1529 100644 --- a/cmd/prometheus/main.go +++ b/cmd/prometheus/main.go @@ -429,7 +429,7 @@ func main() { _, err := a.Parse(os.Args[1:]) if err != nil { - fmt.Fprintln(os.Stderr, fmt.Errorf("Error parsing commandline arguments: %w", err)) + fmt.Fprintln(os.Stderr, fmt.Errorf("Error parsing command line arguments: %w", err)) a.Usage(os.Args[1:]) os.Exit(2) } From 0c6cf868287b47b01272caa1a73e21a07befc0a0 Mon Sep 17 00:00:00 2001 From: timmartin-stripe <131782471+timmartin-stripe@users.noreply.github.com> Date: Fri, 16 Jun 2023 06:55:41 -0400 Subject: [PATCH 210/231] Add sentence explaining what happens when the `for` clause is omitted (#12457) Just adding a statement here explaining that the default is an immediate move to "active" without a pending state. Signed-off-by: Tim Martin --- docs/configuration/alerting_rules.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/configuration/alerting_rules.md b/docs/configuration/alerting_rules.md index 74f6c02b12..3c1ec84f0f 100644 --- a/docs/configuration/alerting_rules.md +++ b/docs/configuration/alerting_rules.md @@ -32,7 +32,11 @@ groups: ``` The optional `for` clause causes Prometheus to wait for a certain duration -between first encountering a new expression output vector element and counting an alert as firing for this element. In this case, Prometheus will check that the alert continues to be active during each evaluation for 10 minutes before firing the alert. Elements that are active, but not firing yet, are in the pending state. +between first encountering a new expression output vector element and counting +an alert as firing for this element. In this case, Prometheus will check that +the alert continues to be active during each evaluation for 10 minutes before +firing the alert. Elements that are active, but not firing yet, are in the pending state. +Alerting rules without the `for` clause will become active on the first evaluation. The `labels` clause allows specifying a set of additional labels to be attached to the alert. Any existing conflicting labels will be overwritten. The label From 0de855508c314ddfd50fb48851be384c61e65501 Mon Sep 17 00:00:00 2001 From: Julien Pivotto Date: Tue, 30 May 2023 15:22:24 +0200 Subject: [PATCH 211/231] Add support for inline TLS certificates Signed-off-by: Julien Pivotto --- docs/configuration/configuration.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/configuration/configuration.md b/docs/configuration/configuration.md index 3a9ace2b6c..db41c3247d 100644 --- a/docs/configuration/configuration.md +++ b/docs/configuration/configuration.md @@ -427,11 +427,16 @@ Where `` must be unique across all scrape configurations. A `tls_config` allows configuring TLS connections. ```yaml -# CA certificate to validate API server certificate with. +# CA certificate to validate API server certificate with. At most one of ca and ca_file is allowed. +[ ca: ] [ ca_file: ] -# Certificate and key files for client cert authentication to the server. +# Certificate and key for client cert authentication to the server. +# At most one of cert and cert_file is allowed. +# At most one of key and key_file is allowed. +[ cert: ] [ cert_file: ] +[ key: ] [ key_file: ] # ServerName extension to indicate the name of the server. From 9b6355f02eee0df38a7c797a02fa5fa47242fd65 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sat, 17 Jun 2023 08:18:31 +0000 Subject: [PATCH 212/231] build(deps): bump golang.org/x/oauth2 from 0.8.0 to 0.9.0 Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.8.0 to 0.9.0. - [Commits](https://github.com/golang/oauth2/compare/v0.8.0...v0.9.0) --- updated-dependencies: - dependency-name: golang.org/x/oauth2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- go.mod | 12 ++++++------ go.sum | 24 ++++++++++++------------ 2 files changed, 18 insertions(+), 18 deletions(-) diff --git a/go.mod b/go.mod index dd75cc333d..7d3ed3ab89 100644 --- a/go.mod +++ b/go.mod @@ -64,10 +64,10 @@ require ( go.uber.org/atomic v1.11.0 go.uber.org/automaxprocs v1.5.2 go.uber.org/goleak v1.2.1 - golang.org/x/net v0.10.0 - golang.org/x/oauth2 v0.8.0 + golang.org/x/net v0.11.0 + golang.org/x/oauth2 v0.9.0 golang.org/x/sync v0.2.0 - golang.org/x/sys v0.8.0 + golang.org/x/sys v0.9.0 golang.org/x/time v0.3.0 golang.org/x/tools v0.9.3 google.golang.org/api v0.114.0 @@ -178,11 +178,11 @@ require ( go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.16.0 // indirect go.opentelemetry.io/otel/metric v1.16.0 // indirect go.opentelemetry.io/proto/otlp v0.19.0 // indirect - golang.org/x/crypto v0.8.0 // indirect + golang.org/x/crypto v0.10.0 // indirect golang.org/x/exp v0.0.0-20230522175609-2e198f4a06a1 golang.org/x/mod v0.10.0 // indirect - golang.org/x/term v0.8.0 // indirect - golang.org/x/text v0.9.0 // indirect + golang.org/x/term v0.9.0 // indirect + golang.org/x/text v0.10.0 // indirect google.golang.org/appengine v1.6.7 // indirect gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/ini.v1 v1.67.0 // indirect diff --git a/go.sum b/go.sum index 774deb85be..f3583ccb19 100644 --- a/go.sum +++ b/go.sum @@ -844,8 +844,8 @@ golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0 golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.0.0-20220829220503-c86fa9a7ed90/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58= -golang.org/x/crypto v0.8.0 h1:pd9TJtTueMTVQXzk8E2XESSMQDj/U7OUu0PqJqPXQjQ= -golang.org/x/crypto v0.8.0/go.mod h1:mRqEX+O9/h5TFCrQhkgjo2yKi0yYA+9ecGkdQoHrywE= +golang.org/x/crypto v0.10.0 h1:LKqV2xt9+kDzSTfOhx4FrkEBcMrAgHSYgzywV9zcGmM= +golang.org/x/crypto v0.10.0/go.mod h1:o4eNf7Ede1fv+hwOwZsTHl9EsPFO6q6ZvYR8vYfY45I= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= @@ -928,8 +928,8 @@ golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qx golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.1.0/go.mod h1:Cx3nUiGt4eDBEyega/BKRp+/AlGL8hYe7U9odMt2Cco= golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= -golang.org/x/net v0.10.0 h1:X2//UzNDwYmtCLn7To6G58Wr6f5ahEAQgKNzv9Y951M= -golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= +golang.org/x/net v0.11.0 h1:Gi2tvZIJyBtO9SDr1q9h5hEQCp/4L2RQ+ar0qjx2oNU= +golang.org/x/net v0.11.0/go.mod h1:2L/ixqYpgIVXmeoSA/4Lu7BzTG4KIyPIryS4IsOd1oQ= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= @@ -937,8 +937,8 @@ golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4Iltr golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.8.0 h1:6dkIjl3j3LtZ/O3sTgZTMsLKSftL/B8Zgq4huOIIUu8= -golang.org/x/oauth2 v0.8.0/go.mod h1:yr7u4HXZRm1R1kBWqr/xKNqewf0plRYoB7sla+BCIXE= +golang.org/x/oauth2 v0.9.0 h1:BPpt2kU7oMRq3kCHAA1tbSEshXRw1LpG2ztgDwrzuAs= +golang.org/x/oauth2 v0.9.0/go.mod h1:qYgFZaFiu6Wg24azG8bdV52QJXJGbZzIIsRCdVKzbLw= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -1021,14 +1021,14 @@ golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.8.0 h1:EBmGv8NaZBZTWvrbjNoL6HVt+IVy3QDQpJs7VRIw3tU= -golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.9.0 h1:KS/R3tvhPqvJvwcKfnBHJwwthS11LRhmM5D59eEXa0s= +golang.org/x/sys v0.9.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.1.0/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= -golang.org/x/term v0.8.0 h1:n5xxQn2i3PC0yLAbjTpNT85q/Kgzcr2gIoX9OrJUols= -golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo= +golang.org/x/term v0.9.0 h1:GRRCnKYhdQrD8kfRAdQ6Zcw1P0OcELxGLKJvtjVMZ28= +golang.org/x/term v0.9.0/go.mod h1:M6DEAAIenWoTxdKrOltXcmDY3rSplQUkrvaDU5FcQyo= golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= @@ -1039,8 +1039,8 @@ golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= -golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE= -golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= +golang.org/x/text v0.10.0 h1:UpjohKhiEgNc0CSauXmwYftY1+LlaC75SJwh0SgCX58= +golang.org/x/text v0.10.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= From f28d7d23cdd279f4530fcc66c8ff248aa5e57a56 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sat, 17 Jun 2023 08:18:50 +0000 Subject: [PATCH 213/231] build(deps): bump github.com/stretchr/testify Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.8.2 to 1.8.4. - [Release notes](https://github.com/stretchr/testify/releases) - [Commits](https://github.com/stretchr/testify/compare/v1.8.2...v1.8.4) --- updated-dependencies: - dependency-name: github.com/stretchr/testify dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] --- documentation/examples/remote_storage/go.mod | 2 +- documentation/examples/remote_storage/go.sum | 8 ++------ 2 files changed, 3 insertions(+), 7 deletions(-) diff --git a/documentation/examples/remote_storage/go.mod b/documentation/examples/remote_storage/go.mod index c0d4331968..2f28619474 100644 --- a/documentation/examples/remote_storage/go.mod +++ b/documentation/examples/remote_storage/go.mod @@ -11,7 +11,7 @@ require ( github.com/prometheus/client_golang v1.15.0 github.com/prometheus/common v0.42.0 github.com/prometheus/prometheus v0.44.0 - github.com/stretchr/testify v1.8.2 + github.com/stretchr/testify v1.8.4 ) require ( diff --git a/documentation/examples/remote_storage/go.sum b/documentation/examples/remote_storage/go.sum index e0eac05c10..c540d8984f 100644 --- a/documentation/examples/remote_storage/go.sum +++ b/documentation/examples/remote_storage/go.sum @@ -211,16 +211,12 @@ github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrf github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= -github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= -github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= -github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= -github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= -github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ8= -github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= +github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= +github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= github.com/vultr/govultr/v2 v2.17.2 h1:gej/rwr91Puc/tgh+j33p/BLR16UrIPnSr+AIwYWZQs= github.com/xhit/go-str2duration/v2 v2.1.0 h1:lxklc02Drh6ynqX+DdPyp5pCKLUQpRT8bp8Ydu2Bstc= github.com/xhit/go-str2duration/v2 v2.1.0/go.mod h1:ohY8p+0f07DiV6Em5LKB0s2YpLtXVyJfNt1+BlmyAsU= From 6adb64a9669e50819ac0b344aea591df3e6fbc15 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sat, 17 Jun 2023 08:19:02 +0000 Subject: [PATCH 214/231] build(deps): bump github.com/aws/aws-sdk-go from 1.44.276 to 1.44.284 Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.276 to 1.44.284. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Commits](https://github.com/aws/aws-sdk-go/compare/v1.44.276...v1.44.284) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index dd75cc333d..ea61b52151 100644 --- a/go.mod +++ b/go.mod @@ -10,7 +10,7 @@ require ( github.com/Azure/go-autorest/autorest/adal v0.9.23 github.com/alecthomas/kingpin/v2 v2.3.2 github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 - github.com/aws/aws-sdk-go v1.44.276 + github.com/aws/aws-sdk-go v1.44.284 github.com/cespare/xxhash/v2 v2.2.0 github.com/dennwc/varint v1.0.0 github.com/digitalocean/godo v1.99.0 diff --git a/go.sum b/go.sum index 774deb85be..bf091742a9 100644 --- a/go.sum +++ b/go.sum @@ -106,8 +106,8 @@ github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2/go.mod h1:W github.com/aws/aws-lambda-go v1.13.3/go.mod h1:4UKl9IzQMoD+QF79YdCuzCwp8VbmG4VAQwij/eHl5CU= github.com/aws/aws-sdk-go v1.27.0/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo= github.com/aws/aws-sdk-go v1.38.35/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro= -github.com/aws/aws-sdk-go v1.44.276 h1:ywPlx9C5Yc482dUgAZ9bHpQ6onVvJvYE9FJWsNDCEy0= -github.com/aws/aws-sdk-go v1.44.276/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI= +github.com/aws/aws-sdk-go v1.44.284 h1:Oc5Kubi43/VCkerlt3ZU3KpBju6BpNkoG3s7E8vj/O8= +github.com/aws/aws-sdk-go v1.44.284/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI= github.com/aws/aws-sdk-go-v2 v0.18.0/go.mod h1:JWVYvqSMppoMJC0x5wdwiImzgXTI9FuZwxzkQq9wy+g= github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8= From 3eaa7eb538a14bfbfbb1f6e11c4f0d5a4878b10e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fran=C3=A7ois=20Gouteroux?= Date: Mon, 19 Jun 2023 10:44:24 +0200 Subject: [PATCH 215/231] fix: apply suggested changes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: François Gouteroux --- util/fmtutil/format.go | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/util/fmtutil/format.go b/util/fmtutil/format.go index 291308dc23..5cff0516b9 100644 --- a/util/fmtutil/format.go +++ b/util/fmtutil/format.go @@ -44,8 +44,8 @@ var MetricMetadataTypeValue = map[string]int32{ "STATESET": 7, } -// FormatMetrics convert metric family to a writerequest. -func FormatMetrics(mf map[string]*dto.MetricFamily, extraLabels map[string]string) (*prompb.WriteRequest, error) { +// CreateWriteRequest convert metric family to a writerequest. +func CreateWriteRequest(mf map[string]*dto.MetricFamily, extraLabels map[string]string) (*prompb.WriteRequest, error) { wr := &prompb.WriteRequest{} // build metric list @@ -76,16 +76,16 @@ func FormatMetrics(mf map[string]*dto.MetricFamily, extraLabels map[string]strin return wr, nil } -func makeTimeserie(wr *prompb.WriteRequest, labels map[string]string, timestamp int64, value float64) { - var timeserie prompb.TimeSeries - timeserie.Labels = makeLabels(labels) - timeserie.Samples = []prompb.Sample{ +func toTimeseries(wr *prompb.WriteRequest, labels map[string]string, timestamp int64, value float64) { + var ts prompb.TimeSeries + ts.Labels = makeLabels(labels) + ts.Samples = []prompb.Sample{ { Timestamp: timestamp, Value: value, }, } - wr.Timeseries = append(wr.Timeseries, timeserie) + wr.Timeseries = append(wr.Timeseries, ts) } func makeTimeseries(wr *prompb.WriteRequest, labels map[string]string, m *dto.Metric) error { @@ -98,9 +98,9 @@ func makeTimeseries(wr *prompb.WriteRequest, labels map[string]string, m *dto.Me switch { case m.Gauge != nil: - makeTimeserie(wr, labels, timestamp, m.GetGauge().GetValue()) + toTimeseries(wr, labels, timestamp, m.GetGauge().GetValue()) case m.Counter != nil: - makeTimeserie(wr, labels, timestamp, m.GetCounter().GetValue()) + toTimeseries(wr, labels, timestamp, m.GetCounter().GetValue()) case m.Summary != nil: metricName := labels[model.MetricNameLabel] // Preserve metric name order with first quantile labels timeseries then sum suffix timeserie and finally count suffix timeserie @@ -112,15 +112,15 @@ func makeTimeseries(wr *prompb.WriteRequest, labels map[string]string, m *dto.Me for _, q := range m.GetSummary().Quantile { quantileLabels[model.QuantileLabel] = fmt.Sprint(q.GetQuantile()) - makeTimeserie(wr, quantileLabels, timestamp, q.GetValue()) + toTimeseries(wr, quantileLabels, timestamp, q.GetValue()) } // Overwrite label model.MetricNameLabel for count and sum metrics // Add Summary sum timeserie labels[model.MetricNameLabel] = metricName + sumStr - makeTimeserie(wr, labels, timestamp, m.GetSummary().GetSampleSum()) + toTimeseries(wr, labels, timestamp, m.GetSummary().GetSampleSum()) // Add Summary count timeserie labels[model.MetricNameLabel] = metricName + countStr - makeTimeserie(wr, labels, timestamp, float64(m.GetSummary().GetSampleCount())) + toTimeseries(wr, labels, timestamp, float64(m.GetSummary().GetSampleCount())) case m.Histogram != nil: metricName := labels[model.MetricNameLabel] @@ -133,18 +133,18 @@ func makeTimeseries(wr *prompb.WriteRequest, labels map[string]string, m *dto.Me for _, b := range m.GetHistogram().Bucket { bucketLabels[model.MetricNameLabel] = metricName + bucketStr bucketLabels[model.BucketLabel] = fmt.Sprint(b.GetUpperBound()) - makeTimeserie(wr, bucketLabels, timestamp, float64(b.GetCumulativeCount())) + toTimeseries(wr, bucketLabels, timestamp, float64(b.GetCumulativeCount())) } // Overwrite label model.MetricNameLabel for count and sum metrics // Add Histogram sum timeserie labels[model.MetricNameLabel] = metricName + sumStr - makeTimeserie(wr, labels, timestamp, m.GetHistogram().GetSampleSum()) + toTimeseries(wr, labels, timestamp, m.GetHistogram().GetSampleSum()) // Add Histogram count timeserie labels[model.MetricNameLabel] = metricName + countStr - makeTimeserie(wr, labels, timestamp, float64(m.GetHistogram().GetSampleCount())) + toTimeseries(wr, labels, timestamp, float64(m.GetHistogram().GetSampleCount())) case m.Untyped != nil: - makeTimeserie(wr, labels, timestamp, m.GetUntyped().GetValue()) + toTimeseries(wr, labels, timestamp, m.GetUntyped().GetValue()) default: err = errors.New("unsupported metric type") } @@ -208,5 +208,5 @@ func ParseMetricsTextAndFormat(input io.Reader, labels map[string]string) (*prom if err != nil { return nil, err } - return FormatMetrics(mf, labels) + return CreateWriteRequest(mf, labels) } From 8bc2a19469119cccf51dd067efdb413e1150ca46 Mon Sep 17 00:00:00 2001 From: Matthias Loibl Date: Mon, 19 Jun 2023 17:40:15 +0200 Subject: [PATCH 216/231] web: Initialize requestCounter metrics to 0 with handler and 200k labels. Signed-off-by: Matthias Loibl --- web/web.go | 1 + 1 file changed, 1 insertion(+) diff --git a/web/web.go b/web/web.go index b9af2819b7..ddb5430eda 100644 --- a/web/web.go +++ b/web/web.go @@ -158,6 +158,7 @@ func (m *metrics) instrumentHandlerWithPrefix(prefix string) func(handlerName st } func (m *metrics) instrumentHandler(handlerName string, handler http.HandlerFunc) http.HandlerFunc { + m.requestCounter.WithLabelValues(handlerName, "200").Add(0) return promhttp.InstrumentHandlerCounter( m.requestCounter.MustCurryWith(prometheus.Labels{"handler": handlerName}), promhttp.InstrumentHandlerDuration( From d19fa62476ada4ec6eb4399c93d0e11cca2aece9 Mon Sep 17 00:00:00 2001 From: Julien Pivotto Date: Tue, 20 Jun 2023 13:39:32 +0200 Subject: [PATCH 217/231] Revert "Improving Performance on the API Gzip Handler (#12363)" This reverts commit dfae954dc1137568f33564e8cffda321f2867925. Signed-off-by: Julien Pivotto --- go.mod | 1 - go.sum | 2 - util/httputil/compression.go | 74 +++++++++++-------- util/httputil/compression_test.go | 115 +++--------------------------- 4 files changed, 52 insertions(+), 140 deletions(-) diff --git a/go.mod b/go.mod index dd75cc333d..9e4ab7ce95 100644 --- a/go.mod +++ b/go.mod @@ -34,7 +34,6 @@ require ( github.com/hetznercloud/hcloud-go v1.45.1 github.com/ionos-cloud/sdk-go/v6 v6.1.7 github.com/json-iterator/go v1.1.12 - github.com/klauspost/compress v1.16.5 github.com/kolo/xmlrpc v0.0.0-20220921171641-a4b6fa1dd06b github.com/linode/linodego v1.17.0 github.com/miekg/dns v1.1.54 diff --git a/go.sum b/go.sum index 774deb85be..897e6acff6 100644 --- a/go.sum +++ b/go.sum @@ -507,8 +507,6 @@ github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvW github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= github.com/klauspost/compress v1.13.6/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk= -github.com/klauspost/compress v1.16.5 h1:IFV2oUNUzZaz+XyusxpLzpzS8Pt5rh0Z16For/djlyI= -github.com/klauspost/compress v1.16.5/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE= github.com/kolo/xmlrpc v0.0.0-20220921171641-a4b6fa1dd06b h1:udzkj9S/zlT5X367kqJis0QP7YMxobob6zhzq6Yre00= github.com/kolo/xmlrpc v0.0.0-20220921171641-a4b6fa1dd06b/go.mod h1:pcaDhQK0/NJZEvtCO0qQPPropqV0sJOJ6YW7X+9kRwM= github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= diff --git a/util/httputil/compression.go b/util/httputil/compression.go index 5e9276958d..b96c088cbb 100644 --- a/util/httputil/compression.go +++ b/util/httputil/compression.go @@ -14,11 +14,11 @@ package httputil import ( + "compress/gzip" + "compress/zlib" + "io" "net/http" "strings" - - "github.com/klauspost/compress/gzhttp" - "github.com/klauspost/compress/zlib" ) const ( @@ -28,27 +28,53 @@ const ( deflateEncoding = "deflate" ) -// Wrapper around http.ResponseWriter which adds deflate compression -type deflatedResponseWriter struct { +// Wrapper around http.Handler which adds suitable response compression based +// on the client's Accept-Encoding headers. +type compressedResponseWriter struct { http.ResponseWriter - writer *zlib.Writer + writer io.Writer } // Writes HTTP response content data. -func (c *deflatedResponseWriter) Write(p []byte) (int, error) { +func (c *compressedResponseWriter) Write(p []byte) (int, error) { return c.writer.Write(p) } -// Close Closes the deflatedResponseWriter and ensures to flush all data before. -func (c *deflatedResponseWriter) Close() { - c.writer.Close() +// Closes the compressedResponseWriter and ensures to flush all data before. +func (c *compressedResponseWriter) Close() { + if zlibWriter, ok := c.writer.(*zlib.Writer); ok { + zlibWriter.Flush() + } + if gzipWriter, ok := c.writer.(*gzip.Writer); ok { + gzipWriter.Flush() + } + if closer, ok := c.writer.(io.Closer); ok { + defer closer.Close() + } } -// Constructs a new deflatedResponseWriter to compress the original writer using 'deflate' compression. -func newDeflateResponseWriter(writer http.ResponseWriter) *deflatedResponseWriter { - return &deflatedResponseWriter{ +// Constructs a new compressedResponseWriter based on client request headers. +func newCompressedResponseWriter(writer http.ResponseWriter, req *http.Request) *compressedResponseWriter { + encodings := strings.Split(req.Header.Get(acceptEncodingHeader), ",") + for _, encoding := range encodings { + switch strings.TrimSpace(encoding) { + case gzipEncoding: + writer.Header().Set(contentEncodingHeader, gzipEncoding) + return &compressedResponseWriter{ + ResponseWriter: writer, + writer: gzip.NewWriter(writer), + } + case deflateEncoding: + writer.Header().Set(contentEncodingHeader, deflateEncoding) + return &compressedResponseWriter{ + ResponseWriter: writer, + writer: zlib.NewWriter(writer), + } + } + } + return &compressedResponseWriter{ ResponseWriter: writer, - writer: zlib.NewWriter(writer), + writer: writer, } } @@ -60,21 +86,7 @@ type CompressionHandler struct { // ServeHTTP adds compression to the original http.Handler's ServeHTTP() method. func (c CompressionHandler) ServeHTTP(writer http.ResponseWriter, req *http.Request) { - encodings := strings.Split(req.Header.Get(acceptEncodingHeader), ",") - for _, encoding := range encodings { - switch strings.TrimSpace(encoding) { - case gzipEncoding: - gzhttp.GzipHandler(c.Handler).ServeHTTP(writer, req) - return - case deflateEncoding: - compWriter := newDeflateResponseWriter(writer) - writer.Header().Set(contentEncodingHeader, deflateEncoding) - c.Handler.ServeHTTP(compWriter, req) - compWriter.Close() - return - default: - c.Handler.ServeHTTP(writer, req) - return - } - } + compWriter := newCompressedResponseWriter(writer, req) + c.Handler.ServeHTTP(compWriter, req) + compWriter.Close() } diff --git a/util/httputil/compression_test.go b/util/httputil/compression_test.go index b7148fc1cc..8512797613 100644 --- a/util/httputil/compression_test.go +++ b/util/httputil/compression_test.go @@ -17,30 +17,23 @@ import ( "bytes" "compress/gzip" "compress/zlib" - "encoding/json" - "fmt" "io" "net/http" "net/http/httptest" - "strings" "testing" "github.com/stretchr/testify/require" - - "github.com/prometheus/prometheus/model/labels" ) var ( - mux *http.ServeMux - server *httptest.Server - respBody = strings.Repeat("Hello World!", 500) + mux *http.ServeMux + server *httptest.Server ) func setup() func() { mux = http.NewServeMux() server = httptest.NewServer(mux) return func() { - server.CloseClientConnections() server.Close() } } @@ -48,7 +41,7 @@ func setup() func() { func getCompressionHandlerFunc() CompressionHandler { hf := func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) - w.Write([]byte(respBody)) + w.Write([]byte("Hello World!")) } return CompressionHandler{ Handler: http.HandlerFunc(hf), @@ -74,8 +67,9 @@ func TestCompressionHandler_PlainText(t *testing.T) { contents, err := io.ReadAll(resp.Body) require.NoError(t, err, "unexpected error while creating the response body reader") + expected := "Hello World!" actual := string(contents) - require.Equal(t, respBody, actual, "expected response with content") + require.Equal(t, expected, actual, "expected response with content") } func TestCompressionHandler_Gzip(t *testing.T) { @@ -109,7 +103,8 @@ func TestCompressionHandler_Gzip(t *testing.T) { require.NoError(t, err, "unexpected error while reading the response body") actual := buf.String() - require.Equal(t, respBody, actual, "unexpected response content") + expected := "Hello World!" + require.Equal(t, expected, actual, "unexpected response content") } func TestCompressionHandler_Deflate(t *testing.T) { @@ -143,98 +138,6 @@ func TestCompressionHandler_Deflate(t *testing.T) { require.NoError(t, err, "unexpected error while reading the response body") actual := buf.String() - require.Equal(t, respBody, actual, "expected response with content") -} - -func Benchmark_compression(b *testing.B) { - client := &http.Client{ - Transport: &http.Transport{ - DisableCompression: true, - }, - } - - cases := map[string]struct { - enc string - numberOfLabels int - }{ - "gzip-10-labels": { - enc: gzipEncoding, - numberOfLabels: 10, - }, - "gzip-100-labels": { - enc: gzipEncoding, - numberOfLabels: 100, - }, - "gzip-1K-labels": { - enc: gzipEncoding, - numberOfLabels: 1000, - }, - "gzip-10K-labels": { - enc: gzipEncoding, - numberOfLabels: 10000, - }, - "gzip-100K-labels": { - enc: gzipEncoding, - numberOfLabels: 100000, - }, - "gzip-1M-labels": { - enc: gzipEncoding, - numberOfLabels: 1000000, - }, - } - - for name, tc := range cases { - b.Run(name, func(b *testing.B) { - tearDown := setup() - defer tearDown() - labels := labels.ScratchBuilder{} - - for i := 0; i < tc.numberOfLabels; i++ { - labels.Add(fmt.Sprintf("Name%v", i), fmt.Sprintf("Value%v", i)) - } - - respBody, err := json.Marshal(labels.Labels()) - require.NoError(b, err) - - hf := func(w http.ResponseWriter, r *http.Request) { - w.WriteHeader(http.StatusOK) - w.Write(respBody) - } - h := CompressionHandler{ - Handler: http.HandlerFunc(hf), - } - - mux.Handle("/foo_endpoint", h) - - req, _ := http.NewRequest("GET", server.URL+"/foo_endpoint", nil) - req.Header.Set(acceptEncodingHeader, tc.enc) - - b.ReportAllocs() - b.ResetTimer() - - // Reusing the array to read the body and avoid allocation on the test - encRespBody := make([]byte, len(respBody)) - - for i := 0; i < b.N; i++ { - resp, err := client.Do(req) - - require.NoError(b, err) - - require.NoError(b, err, "client get failed with unexpected error") - responseBodySize := 0 - for { - n, err := resp.Body.Read(encRespBody) - responseBodySize += n - if err == io.EOF { - break - } - } - - b.ReportMetric(float64(responseBodySize), "ContentLength") - resp.Body.Close() - } - - client.CloseIdleConnections() - }) - } + expected := "Hello World!" + require.Equal(t, expected, actual, "expected response with content") } From c858049744a8cdf684a54cf67b8d16a41d69d81d Mon Sep 17 00:00:00 2001 From: Jesus Vazquez Date: Tue, 20 Jun 2023 17:13:02 +0200 Subject: [PATCH 218/231] Create 2.45.0-rc.1 (#12478) Signed-off-by: Jesus Vazquez --- CHANGELOG.md | 5 +++++ VERSION | 2 +- web/ui/module/codemirror-promql/package.json | 4 ++-- web/ui/module/lezer-promql/package.json | 2 +- web/ui/package-lock.json | 14 +++++++------- web/ui/react-app/package.json | 4 ++-- 6 files changed, 18 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3b4ee3fc17..e4b61b13f1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,10 @@ # Changelog + +## 2.45.0-rc.1 / 2023-06-20 + +* [ENHANCEMENT] Reverts previous enhancement to the API Gzip Handler due to higher cpu and memory usage. #12363 + ## 2.45.0-rc.0 / 2023-06-05 This release is a LTS (Long-Term Support) release of Prometheus and will diff --git a/VERSION b/VERSION index ae2ba732af..362f077e4f 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.45.0-rc.0 +2.45.0-rc.1 diff --git a/web/ui/module/codemirror-promql/package.json b/web/ui/module/codemirror-promql/package.json index 6c404a6c46..f39187f9eb 100644 --- a/web/ui/module/codemirror-promql/package.json +++ b/web/ui/module/codemirror-promql/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/codemirror-promql", - "version": "0.45.0-rc.0", + "version": "0.45.0-rc.1", "description": "a CodeMirror mode for the PromQL language", "types": "dist/esm/index.d.ts", "module": "dist/esm/index.js", @@ -29,7 +29,7 @@ }, "homepage": "https://github.com/prometheus/prometheus/blob/main/web/ui/module/codemirror-promql/README.md", "dependencies": { - "@prometheus-io/lezer-promql": "0.45.0-rc.0", + "@prometheus-io/lezer-promql": "0.45.0-rc.1", "lru-cache": "^6.0.0" }, "devDependencies": { diff --git a/web/ui/module/lezer-promql/package.json b/web/ui/module/lezer-promql/package.json index 389eb88f4f..1b6a1cebda 100644 --- a/web/ui/module/lezer-promql/package.json +++ b/web/ui/module/lezer-promql/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/lezer-promql", - "version": "0.45.0-rc.0", + "version": "0.45.0-rc.1", "description": "lezer-based PromQL grammar", "main": "dist/index.cjs", "type": "module", diff --git a/web/ui/package-lock.json b/web/ui/package-lock.json index f80d2fd6e4..a773df1478 100644 --- a/web/ui/package-lock.json +++ b/web/ui/package-lock.json @@ -28,10 +28,10 @@ }, "module/codemirror-promql": { "name": "@prometheus-io/codemirror-promql", - "version": "0.45.0-rc.0", + "version": "0.45.0-rc.1", "license": "Apache-2.0", "dependencies": { - "@prometheus-io/lezer-promql": "0.45.0-rc.0", + "@prometheus-io/lezer-promql": "0.45.0-rc.1", "lru-cache": "^6.0.0" }, "devDependencies": { @@ -61,7 +61,7 @@ }, "module/lezer-promql": { "name": "@prometheus-io/lezer-promql", - "version": "0.45.0-rc.0", + "version": "0.45.0-rc.1", "license": "Apache-2.0", "devDependencies": { "@lezer/generator": "^1.2.3", @@ -20765,7 +20765,7 @@ }, "react-app": { "name": "@prometheus-io/app", - "version": "0.45.0-rc.0", + "version": "0.45.0-rc.1", "dependencies": { "@codemirror/autocomplete": "^6.7.1", "@codemirror/commands": "^6.2.4", @@ -20783,7 +20783,7 @@ "@lezer/lr": "^1.3.6", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.45.0-rc.0", + "@prometheus-io/codemirror-promql": "0.45.0-rc.1", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", "downshift": "^7.6.0", @@ -23423,7 +23423,7 @@ "@lezer/lr": "^1.3.6", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.45.0-rc.0", + "@prometheus-io/codemirror-promql": "0.45.0-rc.1", "@testing-library/react-hooks": "^7.0.2", "@types/enzyme": "^3.10.13", "@types/flot": "0.0.32", @@ -23487,7 +23487,7 @@ "@lezer/common": "^1.0.3", "@lezer/highlight": "^1.1.6", "@lezer/lr": "^1.3.6", - "@prometheus-io/lezer-promql": "0.45.0-rc.0", + "@prometheus-io/lezer-promql": "0.45.0-rc.1", "@types/lru-cache": "^5.1.1", "isomorphic-fetch": "^3.0.0", "lru-cache": "^6.0.0", diff --git a/web/ui/react-app/package.json b/web/ui/react-app/package.json index d8efd56810..6f75d6503f 100644 --- a/web/ui/react-app/package.json +++ b/web/ui/react-app/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/app", - "version": "0.45.0-rc.0", + "version": "0.45.0-rc.1", "private": true, "dependencies": { "@codemirror/autocomplete": "^6.7.1", @@ -19,7 +19,7 @@ "@lezer/common": "^1.0.3", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.45.0-rc.0", + "@prometheus-io/codemirror-promql": "0.45.0-rc.1", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", "downshift": "^7.6.0", From 87d08abe110406d27870d50ed4e1cfa5f8f12206 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Tue, 20 Jun 2023 20:58:47 +0100 Subject: [PATCH 219/231] labels: faster Compare function when using -tags stringlabels (#12451) Instead of unpacking every individual string, we skip to the point where there is a difference, going 8 bytes at a time where possible. Add benchmark for Compare; extend tests too. --------- Signed-off-by: Bryan Boreham Co-authored-by: Oleg Zaytsev --- model/labels/labels_stringlabels.go | 64 ++++++++++++++---------- model/labels/labels_test.go | 76 +++++++++++++++++++++-------- 2 files changed, 94 insertions(+), 46 deletions(-) diff --git a/model/labels/labels_stringlabels.go b/model/labels/labels_stringlabels.go index e232ca7a4d..eb98fd6fb7 100644 --- a/model/labels/labels_stringlabels.go +++ b/model/labels/labels_stringlabels.go @@ -422,37 +422,49 @@ func FromStrings(ss ...string) Labels { // Compare compares the two label sets. // The result will be 0 if a==b, <0 if a < b, and >0 if a > b. -// TODO: replace with Less function - Compare is never needed. -// TODO: just compare the underlying strings when we don't need alphanumeric sorting. func Compare(a, b Labels) int { - l := len(a.data) - if len(b.data) < l { - l = len(b.data) + // Find the first byte in the string where a and b differ. + shorter, longer := a.data, b.data + if len(b.data) < len(a.data) { + shorter, longer = b.data, a.data + } + i := 0 + // First, go 8 bytes at a time. Data strings are expected to be 8-byte aligned. + sp := unsafe.Pointer((*reflect.StringHeader)(unsafe.Pointer(&shorter)).Data) + lp := unsafe.Pointer((*reflect.StringHeader)(unsafe.Pointer(&longer)).Data) + for ; i < len(shorter)-8; i += 8 { + if *(*uint64)(unsafe.Add(sp, i)) != *(*uint64)(unsafe.Add(lp, i)) { + break + } + } + // Now go 1 byte at a time. + for ; i < len(shorter); i++ { + if shorter[i] != longer[i] { + break + } + } + if i == len(shorter) { + // One Labels was a prefix of the other; the set with fewer labels compares lower. + return len(a.data) - len(b.data) } - ia, ib := 0, 0 - for ia < l { - var aName, bName string - aName, ia = decodeString(a.data, ia) - bName, ib = decodeString(b.data, ib) - if aName != bName { - if aName < bName { - return -1 - } - return 1 - } - var aValue, bValue string - aValue, ia = decodeString(a.data, ia) - bValue, ib = decodeString(b.data, ib) - if aValue != bValue { - if aValue < bValue { - return -1 - } - return 1 + // Now we know that there is some difference before the end of a and b. + // Go back through the fields and find which field that difference is in. + firstCharDifferent := i + for i = 0; ; { + size, nextI := decodeSize(a.data, i) + if nextI+size > firstCharDifferent { + break } + i = nextI + size } - // If all labels so far were in common, the set with fewer labels comes first. - return len(a.data) - len(b.data) + // Difference is inside this entry. + aStr, _ := decodeString(a.data, i) + bStr, _ := decodeString(b.data, i) + if aStr < bStr { + return -1 + } + return +1 } // Copy labels from b on top of whatever was in ls previously, reusing memory or expanding if needed. diff --git a/model/labels/labels_test.go b/model/labels/labels_test.go index 108d8b0de0..f0f05734be 100644 --- a/model/labels/labels_test.go +++ b/model/labels/labels_test.go @@ -361,6 +361,18 @@ func TestLabels_Compare(t *testing.T) { "bbc", "222"), expected: -1, }, + { + compared: FromStrings( + "aaa", "111", + "bb", "222"), + expected: 1, + }, + { + compared: FromStrings( + "aaa", "111", + "bbbb", "222"), + expected: -1, + }, { compared: FromStrings( "aaa", "111"), @@ -380,6 +392,10 @@ func TestLabels_Compare(t *testing.T) { "bbb", "222"), expected: 0, }, + { + compared: EmptyLabels(), + expected: 1, + }, } sign := func(a int) int { @@ -395,6 +411,8 @@ func TestLabels_Compare(t *testing.T) { for i, test := range tests { got := Compare(labels, test.compared) require.Equal(t, sign(test.expected), sign(got), "unexpected comparison result for test case %d", i) + got = Compare(test.compared, labels) + require.Equal(t, -sign(test.expected), sign(got), "unexpected comparison result for reverse test case %d", i) } } @@ -468,27 +486,34 @@ func BenchmarkLabels_Get(b *testing.B) { } } +var comparisonBenchmarkScenarios = []struct { + desc string + base, other Labels +}{ + { + "equal", + FromStrings("a_label_name", "a_label_value", "another_label_name", "another_label_value"), + FromStrings("a_label_name", "a_label_value", "another_label_name", "another_label_value"), + }, + { + "not equal", + FromStrings("a_label_name", "a_label_value", "another_label_name", "another_label_value"), + FromStrings("a_label_name", "a_label_value", "another_label_name", "a_different_label_value"), + }, + { + "different sizes", + FromStrings("a_label_name", "a_label_value", "another_label_name", "another_label_value"), + FromStrings("a_label_name", "a_label_value"), + }, + { + "lots", + FromStrings("aaa", "bbb", "ccc", "ddd", "eee", "fff", "ggg", "hhh", "iii", "jjj", "kkk", "lll", "mmm", "nnn", "ooo", "ppp", "qqq", "rrz"), + FromStrings("aaa", "bbb", "ccc", "ddd", "eee", "fff", "ggg", "hhh", "iii", "jjj", "kkk", "lll", "mmm", "nnn", "ooo", "ppp", "qqq", "rrr"), + }, +} + func BenchmarkLabels_Equals(b *testing.B) { - for _, scenario := range []struct { - desc string - base, other Labels - }{ - { - "equal", - FromStrings("a_label_name", "a_label_value", "another_label_name", "another_label_value"), - FromStrings("a_label_name", "a_label_value", "another_label_name", "another_label_value"), - }, - { - "not equal", - FromStrings("a_label_name", "a_label_value", "another_label_name", "another_label_value"), - FromStrings("a_label_name", "a_label_value", "another_label_name", "a_different_label_value"), - }, - { - "different sizes", - FromStrings("a_label_name", "a_label_value", "another_label_name", "another_label_value"), - FromStrings("a_label_name", "a_label_value"), - }, - } { + for _, scenario := range comparisonBenchmarkScenarios { b.Run(scenario.desc, func(b *testing.B) { b.ResetTimer() for i := 0; i < b.N; i++ { @@ -498,6 +523,17 @@ func BenchmarkLabels_Equals(b *testing.B) { } } +func BenchmarkLabels_Compare(b *testing.B) { + for _, scenario := range comparisonBenchmarkScenarios { + b.Run(scenario.desc, func(b *testing.B) { + b.ResetTimer() + for i := 0; i < b.N; i++ { + _ = Compare(scenario.base, scenario.other) + } + }) + } +} + func TestLabels_Copy(t *testing.T) { require.Equal(t, FromStrings("aaa", "111", "bbb", "222"), FromStrings("aaa", "111", "bbb", "222").Copy()) } From be0b82cf2ffbe5558eaff537b23c0eaed152c76d Mon Sep 17 00:00:00 2001 From: Jesus Vazquez Date: Tue, 20 Jun 2023 22:20:01 +0200 Subject: [PATCH 220/231] Fix PR typo in 2.45.0-rc.1 CHANGELOG (#12479) Signed-off-by: Jesus Vazquez --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e4b61b13f1..946073e61e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,7 +3,7 @@ ## 2.45.0-rc.1 / 2023-06-20 -* [ENHANCEMENT] Reverts previous enhancement to the API Gzip Handler due to higher cpu and memory usage. #12363 +* [ENHANCEMENT] Reverts previous enhancement to the API Gzip Handler due to higher cpu and memory usage. #12476 ## 2.45.0-rc.0 / 2023-06-05 From 8ef767e396bf8445f009f945b0162fd71827f445 Mon Sep 17 00:00:00 2001 From: Jesus Vazquez Date: Fri, 23 Jun 2023 17:01:52 +0200 Subject: [PATCH 221/231] Release 2.45.0 (#12486) Signed-off-by: Jesus Vazquez --- CHANGELOG.md | 7 +------ VERSION | 2 +- web/ui/module/codemirror-promql/package.json | 4 ++-- web/ui/module/lezer-promql/package.json | 2 +- web/ui/package-lock.json | 14 +++++++------- web/ui/react-app/package.json | 4 ++-- 6 files changed, 14 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 946073e61e..76b6e51b5f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,11 +1,7 @@ # Changelog -## 2.45.0-rc.1 / 2023-06-20 - -* [ENHANCEMENT] Reverts previous enhancement to the API Gzip Handler due to higher cpu and memory usage. #12476 - -## 2.45.0-rc.0 / 2023-06-05 +## 2.45.0 / 2023-06-23 This release is a LTS (Long-Term Support) release of Prometheus and will receive security, documentation and bugfix patches for at least 12 months. @@ -20,7 +16,6 @@ Please read more about our LTS release cycle at * [FEATURE] Promtool: When providing the block id, only one block will be loaded and analyzed. #12031 * [FEATURE] Remote-write: New Azure ad configuration to support remote writing directly to Azure Monitor workspace. #11944 * [FEATURE] TSDB: Samples per chunk are now configurable with flag `storage.tsdb.samples-per-chunk`. By default set to its former value 120. #12055 -* [ENHANCEMENT] API: Improving Performance on the API Gzip Handler. #12363 * [ENHANCEMENT] Native histograms: bucket size can now be limited to avoid scrape fails. #12254 * [ENHANCEMENT] TSDB: Dropped series are now deleted from the WAL sooner. #12297 * [BUGFIX] Native histograms: ChunkSeries iterator now checks if a new sample can be appended to the open chunk. #12185 diff --git a/VERSION b/VERSION index 362f077e4f..e599014eab 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.45.0-rc.1 +2.45.0 diff --git a/web/ui/module/codemirror-promql/package.json b/web/ui/module/codemirror-promql/package.json index f39187f9eb..c7fafb1337 100644 --- a/web/ui/module/codemirror-promql/package.json +++ b/web/ui/module/codemirror-promql/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/codemirror-promql", - "version": "0.45.0-rc.1", + "version": "0.45.0", "description": "a CodeMirror mode for the PromQL language", "types": "dist/esm/index.d.ts", "module": "dist/esm/index.js", @@ -29,7 +29,7 @@ }, "homepage": "https://github.com/prometheus/prometheus/blob/main/web/ui/module/codemirror-promql/README.md", "dependencies": { - "@prometheus-io/lezer-promql": "0.45.0-rc.1", + "@prometheus-io/lezer-promql": "0.45.0", "lru-cache": "^6.0.0" }, "devDependencies": { diff --git a/web/ui/module/lezer-promql/package.json b/web/ui/module/lezer-promql/package.json index 1b6a1cebda..847a9610e3 100644 --- a/web/ui/module/lezer-promql/package.json +++ b/web/ui/module/lezer-promql/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/lezer-promql", - "version": "0.45.0-rc.1", + "version": "0.45.0", "description": "lezer-based PromQL grammar", "main": "dist/index.cjs", "type": "module", diff --git a/web/ui/package-lock.json b/web/ui/package-lock.json index a773df1478..ac365d0d09 100644 --- a/web/ui/package-lock.json +++ b/web/ui/package-lock.json @@ -28,10 +28,10 @@ }, "module/codemirror-promql": { "name": "@prometheus-io/codemirror-promql", - "version": "0.45.0-rc.1", + "version": "0.45.0", "license": "Apache-2.0", "dependencies": { - "@prometheus-io/lezer-promql": "0.45.0-rc.1", + "@prometheus-io/lezer-promql": "0.45.0", "lru-cache": "^6.0.0" }, "devDependencies": { @@ -61,7 +61,7 @@ }, "module/lezer-promql": { "name": "@prometheus-io/lezer-promql", - "version": "0.45.0-rc.1", + "version": "0.45.0", "license": "Apache-2.0", "devDependencies": { "@lezer/generator": "^1.2.3", @@ -20765,7 +20765,7 @@ }, "react-app": { "name": "@prometheus-io/app", - "version": "0.45.0-rc.1", + "version": "0.45.0", "dependencies": { "@codemirror/autocomplete": "^6.7.1", "@codemirror/commands": "^6.2.4", @@ -20783,7 +20783,7 @@ "@lezer/lr": "^1.3.6", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.45.0-rc.1", + "@prometheus-io/codemirror-promql": "0.45.0", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", "downshift": "^7.6.0", @@ -23423,7 +23423,7 @@ "@lezer/lr": "^1.3.6", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.45.0-rc.1", + "@prometheus-io/codemirror-promql": "0.45.0", "@testing-library/react-hooks": "^7.0.2", "@types/enzyme": "^3.10.13", "@types/flot": "0.0.32", @@ -23487,7 +23487,7 @@ "@lezer/common": "^1.0.3", "@lezer/highlight": "^1.1.6", "@lezer/lr": "^1.3.6", - "@prometheus-io/lezer-promql": "0.45.0-rc.1", + "@prometheus-io/lezer-promql": "0.45.0", "@types/lru-cache": "^5.1.1", "isomorphic-fetch": "^3.0.0", "lru-cache": "^6.0.0", diff --git a/web/ui/react-app/package.json b/web/ui/react-app/package.json index 6f75d6503f..81eba150a0 100644 --- a/web/ui/react-app/package.json +++ b/web/ui/react-app/package.json @@ -1,6 +1,6 @@ { "name": "@prometheus-io/app", - "version": "0.45.0-rc.1", + "version": "0.45.0", "private": true, "dependencies": { "@codemirror/autocomplete": "^6.7.1", @@ -19,7 +19,7 @@ "@lezer/common": "^1.0.3", "@nexucis/fuzzy": "^0.4.1", "@nexucis/kvsearch": "^0.8.1", - "@prometheus-io/codemirror-promql": "0.45.0-rc.1", + "@prometheus-io/codemirror-promql": "0.45.0", "bootstrap": "^4.6.2", "css.escape": "^1.5.1", "downshift": "^7.6.0", From 02277bbe0dd2d89fd3fe5f508399d54da9c61827 Mon Sep 17 00:00:00 2001 From: Matt Harbison Date: Fri, 23 Jun 2023 16:37:04 -0400 Subject: [PATCH 222/231] Fix path handling in File-SD watcher to allow directory monitoring on Windows Previously, `d.paths` were normalized to backslashes on Windows, even when if the config file used Unix style. The end result meant always watching `./`, so changes for this config were always ignored: scrape_configs: - job_name: 'envmsc1' file_sd_configs: - files: - 'targets/envmsc1.d/*.yml' - 'targets/envmsc1.d/*.yaml' Additionally, unlike the other platforms, no warning was emitted on startup about not being able to install the watch if the directory didn't exist. Now it is logged. Signed-off-by: Matt Harbison --- discovery/file/file.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/discovery/file/file.go b/discovery/file/file.go index c45595c6dd..60b63350f5 100644 --- a/discovery/file/file.go +++ b/discovery/file/file.go @@ -226,8 +226,8 @@ func (d *Discovery) watchFiles() { panic("no watcher configured") } for _, p := range d.paths { - if idx := strings.LastIndex(p, "/"); idx > -1 { - p = p[:idx] + if dir, _ := filepath.Split(p); dir != "" { + p = dir } else { p = "./" } From e1115ae58d069b3f7fd19ffc6b635a6c98882148 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Mon, 26 Jun 2023 18:35:22 +0100 Subject: [PATCH 223/231] labels: improve Get method for stringlabels build (#12485) Inline one call to `decodeString`, and skip decoding the value string until we find a match for the name. Do a quick check on the first character in each string, and exit early if we've gone past - labels are sorted in order. Also improve tests and benchmark: * labels: test Get with varying lengths - it's not typical for Prometheus labels to all be the same length. * extend benchmark with label not found --------- Signed-off-by: Bryan Boreham --- model/labels/labels_stringlabels.go | 24 +++++++++++++++++++----- model/labels/labels_test.go | 6 ++++-- 2 files changed, 23 insertions(+), 7 deletions(-) diff --git a/model/labels/labels_stringlabels.go b/model/labels/labels_stringlabels.go index eb98fd6fb7..223aa6ebf7 100644 --- a/model/labels/labels_stringlabels.go +++ b/model/labels/labels_stringlabels.go @@ -273,13 +273,27 @@ func (ls Labels) Copy() Labels { // Get returns the value for the label with the given name. // Returns an empty string if the label doesn't exist. func (ls Labels) Get(name string) string { + if name == "" { // Avoid crash in loop if someone asks for "". + return "" // Prometheus does not store blank label names. + } for i := 0; i < len(ls.data); { - var lName, lValue string - lName, i = decodeString(ls.data, i) - lValue, i = decodeString(ls.data, i) - if lName == name { - return lValue + var size int + size, i = decodeSize(ls.data, i) + if ls.data[i] == name[0] { + lName := ls.data[i : i+size] + i += size + if lName == name { + lValue, _ := decodeString(ls.data, i) + return lValue + } + } else { + if ls.data[i] > name[0] { // Stop looking if we've gone past. + break + } + i += size } + size, i = decodeSize(ls.data, i) + i += size } return "" } diff --git a/model/labels/labels_test.go b/model/labels/labels_test.go index f0f05734be..d91be27cbc 100644 --- a/model/labels/labels_test.go +++ b/model/labels/labels_test.go @@ -443,7 +443,8 @@ func TestLabels_Has(t *testing.T) { func TestLabels_Get(t *testing.T) { require.Equal(t, "", FromStrings("aaa", "111", "bbb", "222").Get("foo")) - require.Equal(t, "111", FromStrings("aaa", "111", "bbb", "222").Get("aaa")) + require.Equal(t, "111", FromStrings("aaaa", "111", "bbb", "222").Get("aaaa")) + require.Equal(t, "222", FromStrings("aaaa", "111", "bbb", "222").Get("bbb")) } // BenchmarkLabels_Get was written to check whether a binary search can improve the performance vs the linear search implementation @@ -463,7 +464,7 @@ func BenchmarkLabels_Get(b *testing.B) { maxLabels := 30 allLabels := make([]Label, maxLabels) for i := 0; i < maxLabels; i++ { - allLabels[i] = Label{Name: strings.Repeat(string('a'+byte(i)), 5)} + allLabels[i] = Label{Name: strings.Repeat(string('a'+byte(i)), 5+(i%5))} } for _, size := range []int{5, 10, maxLabels} { b.Run(fmt.Sprintf("with %d labels", size), func(b *testing.B) { @@ -474,6 +475,7 @@ func BenchmarkLabels_Get(b *testing.B) { {"get first label", allLabels[0].Name}, {"get middle label", allLabels[size/2].Name}, {"get last label", allLabels[size-1].Name}, + {"get not-found label", "benchmark"}, } { b.Run(scenario.desc, func(b *testing.B) { b.ResetTimer() From d78661ba108f7771e196d69dea00711d51a57893 Mon Sep 17 00:00:00 2001 From: Julien Pivotto Date: Tue, 27 Jun 2023 08:58:52 +0200 Subject: [PATCH 224/231] Stepping up as 2.46 release shepherd (#12494) Signed-off-by: Julien Pivotto --- RELEASE.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/RELEASE.md b/RELEASE.md index f5c907fe99..0d0918191b 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -50,7 +50,9 @@ Release cadence of first pre-releases being cut is 6 weeks. | v2.43 | 2023-03-08 | Julien Pivotto (GitHub: @roidelapluie) | | v2.44 | 2023-04-19 | Bryan Boreham (GitHub: @bboreham) | | v2.45 LTS | 2023-05-31 | Jesus Vazquez (Github: @jesusvazquez) | -| v2.46 | 2023-07-12 | **searching for volunteer** | +| v2.46 | 2023-07-12 | Julien Pivotto (GitHub: @roidelapluie) | +| v2.47 | 2023-08-23 | **searching for volunteer** | +| v2.48 | 2023-10-04 | **searching for volunteer** | If you are interested in volunteering please create a pull request against the [prometheus/prometheus](https://github.com/prometheus/prometheus) repository and propose yourself for the release series of your choice. From 58d38c4c5630fb73afb5403d03ede884ad7fd91c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fran=C3=A7ois=20Gouteroux?= Date: Tue, 27 Jun 2023 09:30:39 +0200 Subject: [PATCH 225/231] fix: apply suggested changes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: François Gouteroux --- cmd/promtool/metrics.go | 2 +- util/fmtutil/format.go | 33 ++++++++++++--------------------- util/fmtutil/format_test.go | 10 +++++----- 3 files changed, 18 insertions(+), 27 deletions(-) diff --git a/cmd/promtool/metrics.go b/cmd/promtool/metrics.go index 4a6fafd407..2bc2237e2f 100644 --- a/cmd/promtool/metrics.go +++ b/cmd/promtool/metrics.go @@ -102,7 +102,7 @@ func PushMetrics(url *url.URL, roundTripper http.RoundTripper, headers map[strin } func parseAndPushMetrics(client *remote.Client, data []byte, labels map[string]string) bool { - metricsData, err := fmtutil.ParseMetricsTextAndFormat(bytes.NewReader(data), labels) + metricsData, err := fmtutil.MetricTextToWriteRequest(bytes.NewReader(data), labels) if err != nil { fmt.Fprintln(os.Stderr, " FAILED:", err) return false diff --git a/util/fmtutil/format.go b/util/fmtutil/format.go index 5cff0516b9..9034a90fa7 100644 --- a/util/fmtutil/format.go +++ b/util/fmtutil/format.go @@ -44,8 +44,18 @@ var MetricMetadataTypeValue = map[string]int32{ "STATESET": 7, } -// CreateWriteRequest convert metric family to a writerequest. -func CreateWriteRequest(mf map[string]*dto.MetricFamily, extraLabels map[string]string) (*prompb.WriteRequest, error) { +// MetricTextToWriteRequest consumes an io.Reader and return the data in write request format. +func MetricTextToWriteRequest(input io.Reader, labels map[string]string) (*prompb.WriteRequest, error) { + var parser expfmt.TextParser + mf, err := parser.TextToMetricFamilies(input) + if err != nil { + return nil, err + } + return MetricFamiliesToWriteRequest(mf, labels) +} + +// MetricFamiliesToWriteRequest convert metric family to a writerequest. +func MetricFamiliesToWriteRequest(mf map[string]*dto.MetricFamily, extraLabels map[string]string) (*prompb.WriteRequest, error) { wr := &prompb.WriteRequest{} // build metric list @@ -191,22 +201,3 @@ func makeLabelsMap(m *dto.Metric, metricName string, extraLabels map[string]stri return labels } - -// ParseMetricsTextReader consumes an io.Reader and returns the MetricFamily. -func ParseMetricsTextReader(input io.Reader) (map[string]*dto.MetricFamily, error) { - var parser expfmt.TextParser - mf, err := parser.TextToMetricFamilies(input) - if err != nil { - return nil, err - } - return mf, nil -} - -// ParseMetricsTextAndFormat return the data in the expected prometheus metrics write request format. -func ParseMetricsTextAndFormat(input io.Reader, labels map[string]string) (*prompb.WriteRequest, error) { - mf, err := ParseMetricsTextReader(input) - if err != nil { - return nil, err - } - return CreateWriteRequest(mf, labels) -} diff --git a/util/fmtutil/format_test.go b/util/fmtutil/format_test.go index 5c1ab5bde0..0f052f5e75 100644 --- a/util/fmtutil/format_test.go +++ b/util/fmtutil/format_test.go @@ -201,13 +201,13 @@ func TestParseAndPushMetricsTextAndFormat(t *testing.T) { `)) labels := map[string]string{"job": "promtool"} - expected, err := ParseMetricsTextAndFormat(input, labels) + expected, err := MetricTextToWriteRequest(input, labels) require.NoError(t, err) require.Equal(t, writeRequestFixture, expected) } -func TestParseMetricsTextAndFormatErrorParsingFloatValue(t *testing.T) { +func TestMetricTextToWriteRequestErrorParsingFloatValue(t *testing.T) { input := bytes.NewReader([]byte(` # HELP http_requests_total The total number of HTTP requests. # TYPE http_requests_total counter @@ -216,11 +216,11 @@ func TestParseMetricsTextAndFormatErrorParsingFloatValue(t *testing.T) { `)) labels := map[string]string{"job": "promtool"} - _, err := ParseMetricsTextAndFormat(input, labels) + _, err := MetricTextToWriteRequest(input, labels) require.Equal(t, err.Error(), "text format parsing error in line 4: expected float as value, got \"1027Error\"") } -func TestParseMetricsTextAndFormatErrorParsingMetricType(t *testing.T) { +func TestMetricTextToWriteRequestErrorParsingMetricType(t *testing.T) { input := bytes.NewReader([]byte(` # HELP node_info node info summary. # TYPE node_info info @@ -228,6 +228,6 @@ func TestParseMetricsTextAndFormatErrorParsingMetricType(t *testing.T) { `)) labels := map[string]string{"job": "promtool"} - _, err := ParseMetricsTextAndFormat(input, labels) + _, err := MetricTextToWriteRequest(input, labels) require.Equal(t, err.Error(), "text format parsing error in line 3: unknown metric type \"info\"") } From 2c6b9c4cd4b7ce8232eaefaf6b1dbdccd6aeb130 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 27 Jun 2023 11:14:27 +0000 Subject: [PATCH 226/231] build(deps): bump github.com/influxdata/influxdb Bumps [github.com/influxdata/influxdb](https://github.com/influxdata/influxdb) from 1.11.0 to 1.11.2. - [Release notes](https://github.com/influxdata/influxdb/releases) - [Changelog](https://github.com/influxdata/influxdb/blob/master/CHANGELOG_OLD.md) - [Commits](https://github.com/influxdata/influxdb/commits) --- updated-dependencies: - dependency-name: github.com/influxdata/influxdb dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] --- documentation/examples/remote_storage/go.mod | 2 +- documentation/examples/remote_storage/go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/documentation/examples/remote_storage/go.mod b/documentation/examples/remote_storage/go.mod index c0d4331968..2972854f13 100644 --- a/documentation/examples/remote_storage/go.mod +++ b/documentation/examples/remote_storage/go.mod @@ -7,7 +7,7 @@ require ( github.com/go-kit/log v0.2.1 github.com/gogo/protobuf v1.3.2 github.com/golang/snappy v0.0.4 - github.com/influxdata/influxdb v1.11.0 + github.com/influxdata/influxdb v1.11.2 github.com/prometheus/client_golang v1.15.0 github.com/prometheus/common v0.42.0 github.com/prometheus/prometheus v0.44.0 diff --git a/documentation/examples/remote_storage/go.sum b/documentation/examples/remote_storage/go.sum index e0eac05c10..678ddfe2c1 100644 --- a/documentation/examples/remote_storage/go.sum +++ b/documentation/examples/remote_storage/go.sum @@ -118,8 +118,8 @@ github.com/hashicorp/nomad/api v0.0.0-20230418003350-3067191c5197 h1:I5xhKLePXpX github.com/hashicorp/serf v0.10.1 h1:Z1H2J60yRKvfDYAOZLd2MU0ND4AH/WDz7xYHDWQsIPY= github.com/hetznercloud/hcloud-go v1.42.0 h1:Es/CDOForQN3nOOP5Vxh1N/YHjpCg386iYEX5zCgi+A= github.com/imdario/mergo v0.3.13 h1:lFzP57bqS/wsqKssCGmtLAb8A0wKjLGrve2q3PPVcBk= -github.com/influxdata/influxdb v1.11.0 h1:0X+ZsbcOWc6AEi5MHee9BYqXCKmz8IZsljrRYjmV8Qg= -github.com/influxdata/influxdb v1.11.0/go.mod h1:V93tJcidY0Zh0LtSONZWnXXGDyt20dtVf+Ddp4EnhaA= +github.com/influxdata/influxdb v1.11.2 h1:qOF3uQN1mDfJNEKwbAgJsqehf8IXgKok2vlGm736oGo= +github.com/influxdata/influxdb v1.11.2/go.mod h1:eUMkLTE2vQwvSk6KGMrTBLKPaqSuczuelGbggigMPFw= github.com/ionos-cloud/sdk-go/v6 v6.1.6 h1:0n4irdqNska+1s3YMCRhrAqKbibEgQ7SwwhAlHzYT5A= github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg= github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo= From c18f5b64872429384ef0170bf8a39f1d809b7337 Mon Sep 17 00:00:00 2001 From: SuperQ Date: Thu, 30 Mar 2023 11:04:37 +0200 Subject: [PATCH 227/231] Update golangci-lint * Update golangci-lint to v1.53.3. * Update the sync script handler for the old golanci-lint action. Signed-off-by: SuperQ --- .github/workflows/ci.yml | 2 +- Makefile.common | 2 +- scripts/golangci-lint.yml | 32 ++++++++++++++++++++++++++++++++ scripts/sync_repo_files.sh | 23 +++++++++++++++-------- 4 files changed, 49 insertions(+), 10 deletions(-) create mode 100644 scripts/golangci-lint.yml diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index bfa1ac00d0..c0bccd0efd 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -150,7 +150,7 @@ jobs: uses: golangci/golangci-lint-action@v3.4.0 with: args: --verbose - version: v1.51.2 + version: v1.53.3 fuzzing: uses: ./.github/workflows/fuzzing.yml if: github.event_name == 'pull_request' diff --git a/Makefile.common b/Makefile.common index e372d34738..787feff089 100644 --- a/Makefile.common +++ b/Makefile.common @@ -61,7 +61,7 @@ PROMU_URL := https://github.com/prometheus/promu/releases/download/v$(PROMU_ SKIP_GOLANGCI_LINT := GOLANGCI_LINT := GOLANGCI_LINT_OPTS ?= -GOLANGCI_LINT_VERSION ?= v1.51.2 +GOLANGCI_LINT_VERSION ?= v1.53.3 # golangci-lint only supports linux, darwin and windows platforms on i386/amd64. # windows isn't included here because of the path separator being different. ifeq ($(GOHOSTOS),$(filter $(GOHOSTOS),linux darwin)) diff --git a/scripts/golangci-lint.yml b/scripts/golangci-lint.yml new file mode 100644 index 0000000000..433f71b885 --- /dev/null +++ b/scripts/golangci-lint.yml @@ -0,0 +1,32 @@ +--- +# This action is synced from https://github.com/prometheus/prometheus +name: golangci-lint +on: + push: + paths: + - "go.sum" + - "go.mod" + - "**.go" + - "scripts/errcheck_excludes.txt" + - ".github/workflows/golangci-lint.yml" + - ".golangci.yml" + pull_request: + +jobs: + golangci: + name: lint + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v3 + - name: install Go + uses: actions/setup-go@v3 + with: + go-version: 1.20.x + - name: Install snmp_exporter/generator dependencies + run: sudo apt-get update && sudo apt-get -y install libsnmp-dev + if: github.repository == 'prometheus/snmp_exporter' + - name: Lint + uses: golangci/golangci-lint-action@v3.4.0 + with: + version: v1.53.3 diff --git a/scripts/sync_repo_files.sh b/scripts/sync_repo_files.sh index 3c987f07ee..5e32eb8832 100755 --- a/scripts/sync_repo_files.sh +++ b/scripts/sync_repo_files.sh @@ -37,7 +37,7 @@ if [ -z "${GITHUB_TOKEN}" ]; then fi # List of files that should be synced. -SYNC_FILES="CODE_OF_CONDUCT.md LICENSE Makefile.common SECURITY.md .yamllint .github/workflows/golangci-lint.yml" +SYNC_FILES="CODE_OF_CONDUCT.md LICENSE Makefile.common SECURITY.md .yamllint scripts/golangci-lint.yml" # Go to the root of the repo cd "$(git rev-parse --show-cdup)" || exit 1 @@ -115,20 +115,23 @@ process_repo() { local needs_update=() for source_file in ${SYNC_FILES}; do source_checksum="$(sha256sum "${source_dir}/${source_file}" | cut -d' ' -f1)" - - target_file="$(curl -sL --fail "https://raw.githubusercontent.com/${org_repo}/${default_branch}/${source_file}")" + if [[ "${source_file}" == 'scripts/golangci-lint.yml' ]] && ! check_go "${org_repo}" "${default_branch}" ; then + echo "${org_repo} is not Go, skipping golangci-lint.yml." + continue + fi if [[ "${source_file}" == 'LICENSE' ]] && ! check_license "${target_file}" ; then echo "LICENSE in ${org_repo} is not apache, skipping." continue fi - if [[ "${source_file}" == '.github/workflows/golangci-lint.yml' ]] && ! check_go "${org_repo}" "${default_branch}" ; then - echo "${org_repo} is not Go, skipping .github/workflows/golangci-lint.yml." - continue + target_filename="${source_file}" + if [[ "${source_file}" == 'scripts/golangci-lint.yml' ]] ; then + target_filename=".github/workflows/${source_file}" fi + target_file="$(curl -sL --fail "https://raw.githubusercontent.com/${org_repo}/${default_branch}/${target_filename}")" if [[ -z "${target_file}" ]]; then echo "${source_file} doesn't exist in ${org_repo}" case "${source_file}" in - CODE_OF_CONDUCT.md | SECURITY.md | .github/workflows/golangci-lint.yml) + CODE_OF_CONDUCT.md | SECURITY.md) echo "${source_file} missing in ${org_repo}, force updating." needs_update+=("${source_file}") ;; @@ -159,8 +162,12 @@ process_repo() { # Update the files in target repo by one from prometheus/prometheus. for source_file in "${needs_update[@]}"; do + target_filename="${source_file}" + if [[ "${source_file}" == 'scripts/golangci-lint.yml' ]] ; then + target_filename=".github/workflows/${source_file}" + fi case "${source_file}" in - *) cp -f "${source_dir}/${source_file}" "./${source_file}" ;; + *) cp -f "${source_dir}/${source_file}" "./${target_filename}" ;; esac done From 16c3cd35b1dfeafd4812d76db4cfa38f7c09ae67 Mon Sep 17 00:00:00 2001 From: SuperQ Date: Tue, 4 Apr 2023 10:28:19 +0200 Subject: [PATCH 228/231] Disable revive unused-parameter Signed-off-by: SuperQ --- .golangci.yml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.golangci.yml b/.golangci.yml index fc2721455c..128ad4f04a 100644 --- a/.golangci.yml +++ b/.golangci.yml @@ -56,3 +56,9 @@ linters-settings: local-prefixes: github.com/prometheus/prometheus gofumpt: extra-rules: true + revive: + rules: + # https://github.com/mgechev/revive/blob/master/RULES_DESCRIPTIONS.md#unused-parameter + - name: unused-parameter + severity: warning + disabled: true From 4e9b89f88716d04c25ded01cc653052595073c73 Mon Sep 17 00:00:00 2001 From: SuperQ Date: Tue, 27 Jun 2023 13:51:29 +0200 Subject: [PATCH 229/231] Update depguard config syntax. Signed-off-by: SuperQ --- .golangci.yml | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/.golangci.yml b/.golangci.yml index 128ad4f04a..4a6daae594 100644 --- a/.golangci.yml +++ b/.golangci.yml @@ -31,14 +31,19 @@ issues: linters-settings: depguard: - list-type: blacklist - include-go-root: true - packages-with-error-message: - - sync/atomic: "Use go.uber.org/atomic instead of sync/atomic" - - github.com/stretchr/testify/assert: "Use github.com/stretchr/testify/require instead of github.com/stretchr/testify/assert" - - github.com/go-kit/kit/log: "Use github.com/go-kit/log instead of github.com/go-kit/kit/log" - - io/ioutil: "Use corresponding 'os' or 'io' functions instead." - - regexp: "Use github.com/grafana/regexp instead of regexp" + rules: + main: + deny: + - pkg: "sync/atomic" + desc: "Use go.uber.org/atomic instead of sync/atomic" + - pkg: "github.com/stretchr/testify/assert" + desc: "Use github.com/stretchr/testify/require instead of github.com/stretchr/testify/assert" + - pkg: "github.com/go-kit/kit/log" + desc: "Use github.com/go-kit/log instead of github.com/go-kit/kit/log" + - pkg: "io/ioutil" + desc: "Use corresponding 'os' or 'io' functions instead." + - pkg: "regexp" + desc: "Use github.com/grafana/regexp instead of regexp" errcheck: exclude-functions: # Don't flag lines such as "io.Copy(io.Discard, resp.Body)". From 686482ab34cf8cbbfdc88caa0edfd7cbdb8439b2 Mon Sep 17 00:00:00 2001 From: Matthias Loibl Date: Tue, 27 Jun 2023 18:10:38 +0200 Subject: [PATCH 230/231] Remove Add(0) Signed-off-by: Matthias Loibl --- web/web.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/web.go b/web/web.go index ddb5430eda..6a3eab3270 100644 --- a/web/web.go +++ b/web/web.go @@ -158,7 +158,7 @@ func (m *metrics) instrumentHandlerWithPrefix(prefix string) func(handlerName st } func (m *metrics) instrumentHandler(handlerName string, handler http.HandlerFunc) http.HandlerFunc { - m.requestCounter.WithLabelValues(handlerName, "200").Add(0) + m.requestCounter.WithLabelValues(handlerName, "200") return promhttp.InstrumentHandlerCounter( m.requestCounter.MustCurryWith(prometheus.Labels{"handler": handlerName}), promhttp.InstrumentHandlerDuration( From 484a9e4071dbcd83c9bc7a878816723ce7114398 Mon Sep 17 00:00:00 2001 From: cui fliter Date: Thu, 29 Jun 2023 18:28:13 +0800 Subject: [PATCH 231/231] fix some typos (#12498) Signed-off-by: cui fliter --- CHANGELOG.md | 2 +- web/ui/react-app/public/index.html | 2 +- web/ui/react-app/src/contexts/ThemeContext.tsx | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 76b6e51b5f..d316e84d37 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -63,7 +63,7 @@ improvements for testing. #10991 * [FEATURE] Promtool: Add HTTP client configuration to query commands. #11487 * [FEATURE] Scrape: Add `scrape_config_files` to include scrape configs from different files. #12019 * [FEATURE] HTTP client: Add `no_proxy` to exclude URLs from proxied requests. #12098 -* [FEATURE] HTTP client: Add `proxy_from_enviroment` to read proxies from env variables. #12098 +* [FEATURE] HTTP client: Add `proxy_from_environment` to read proxies from env variables. #12098 * [ENHANCEMENT] API: Add support for setting lookback delta per query via the API. #12088 * [ENHANCEMENT] API: Change HTTP status code from 503/422 to 499 if a request is canceled. #11897 * [ENHANCEMENT] Scrape: Allow exemplars for all metric types. #11984 diff --git a/web/ui/react-app/public/index.html b/web/ui/react-app/public/index.html index 57131a26b6..9aeb3555e1 100755 --- a/web/ui/react-app/public/index.html +++ b/web/ui/react-app/public/index.html @@ -42,7 +42,7 @@ --> TITLE_PLACEHOLDER diff --git a/web/ui/react-app/src/contexts/ThemeContext.tsx b/web/ui/react-app/src/contexts/ThemeContext.tsx index 9ee84cf5ec..2f2ee70730 100644 --- a/web/ui/react-app/src/contexts/ThemeContext.tsx +++ b/web/ui/react-app/src/contexts/ThemeContext.tsx @@ -9,7 +9,7 @@ export interface ThemeCtx { setTheme: (t: themeSetting) => void; } -// defaults, will be overriden in App.tsx +// defaults, will be overridden in App.tsx export const ThemeContext = React.createContext({ theme: 'light', userPreference: 'auto',