mirror of
https://git.haproxy.org/git/haproxy.git/
synced 2026-05-04 20:46:11 +02:00
38126ecd90
Removed the insecure-fork-wanted runtime check from the OTel filter parser and all related mentions from documentation and test configuration. The OpenTelemetry C wrapper library can now explicitly start all necessary OTel threads immediately after configuration parsing, so it is no longer affected by the HAProxy thread/process creation restriction and the insecure-fork-wanted option is no longer needed.
278 lines
9.3 KiB
Markdown
278 lines
9.3 KiB
Markdown
## HAProxy OpenTelemetry Filter (OTel)
|
|
|
|
The OTel filter enables HAProxy to emit telemetry data -- traces, metrics and
|
|
logs -- to any OpenTelemetry-compatible backend via the OpenTelemetry protocol
|
|
(OTLP).
|
|
|
|
It is the successor to the OpenTracing (OT) filter, built on the OpenTelemetry
|
|
standard which unifies distributed tracing, metrics and logging into a single
|
|
observability framework.
|
|
|
|
### Features
|
|
|
|
- **Distributed tracing** -- spans with parent-child relationships, context
|
|
propagation via HTTP headers or HAProxy variables, links, baggage and status.
|
|
- **Metrics** -- counter, histogram, up-down counter and gauge instruments with
|
|
configurable aggregation and bucket boundaries.
|
|
- **Logging** -- log records with severity levels, optional span correlation and
|
|
runtime-evaluated attributes.
|
|
- **Rate limiting** -- percentage-based sampling (0.0--100.0) for controlling
|
|
overhead.
|
|
- **ACL integration** -- fine-grained conditional execution at instrumentation,
|
|
scope and event levels.
|
|
- **CLI management** -- runtime enable/disable, rate adjustment, error mode
|
|
switching and status inspection.
|
|
- **Context propagation** -- inject/extract span contexts between cascaded
|
|
HAProxy instances or external services.
|
|
|
|
### Dependencies
|
|
|
|
The filter requires the
|
|
[OpenTelemetry C Wrapper](https://github.com/haproxytech/opentelemetry-c-wrapper)
|
|
library, which wraps the OpenTelemetry C++ SDK.
|
|
|
|
### Building
|
|
|
|
The OTel filter is compiled together with HAProxy by adding `USE_OTEL=1` to the
|
|
make command.
|
|
|
|
#### Using pkg-config
|
|
|
|
```
|
|
PKG_CONFIG_PATH=/opt/lib/pkgconfig make -j8 USE_OTEL=1 TARGET=linux-glibc
|
|
```
|
|
|
|
#### Explicit paths
|
|
|
|
```
|
|
make -j8 USE_OTEL=1 OTEL_INC=/opt/include OTEL_LIB=/opt/lib TARGET=linux-glibc
|
|
```
|
|
|
|
#### Build options
|
|
|
|
| Variable | Description |
|
|
|-----------------|-----------------------------------------------------|
|
|
| `USE_OTEL` | Enable the OpenTelemetry filter |
|
|
| `OTEL_DEBUG` | Compile in debug mode |
|
|
| `OTEL_INC` | Force path to opentelemetry-c-wrapper include files |
|
|
| `OTEL_LIB` | Force path to opentelemetry-c-wrapper library |
|
|
| `OTEL_RUNPATH` | Add opentelemetry-c-wrapper RUNPATH to executable |
|
|
| `OTEL_USE_VARS` | Enable context propagation via HAProxy variables |
|
|
|
|
#### Debug mode
|
|
|
|
```
|
|
PKG_CONFIG_PATH=/opt/lib/pkgconfig make -j8 USE_OTEL=1 OTEL_DEBUG=1 TARGET=linux-glibc
|
|
```
|
|
|
|
#### Variable-based context propagation
|
|
|
|
```
|
|
PKG_CONFIG_PATH=/opt/lib/pkgconfig make -j8 USE_OTEL=1 OTEL_USE_VARS=1 TARGET=linux-glibc
|
|
```
|
|
|
|
#### Verifying the build
|
|
|
|
```
|
|
./haproxy -vv | grep -i opentelemetry
|
|
```
|
|
|
|
If the filter is built in, the output contains:
|
|
|
|
```
|
|
Built with OpenTelemetry support (C++ version 1.26.0, C Wrapper version 1.0.0-842).
|
|
[OTEL] opentelemetry
|
|
```
|
|
|
|
#### Library path at runtime
|
|
|
|
When pkg-config is not used, the executable may not find the library at startup.
|
|
Use `LD_LIBRARY_PATH` or build with `OTEL_RUNPATH=1`:
|
|
|
|
```
|
|
LD_LIBRARY_PATH=/opt/lib ./haproxy ...
|
|
```
|
|
|
|
```
|
|
make -j8 USE_OTEL=1 OTEL_RUNPATH=1 OTEL_INC=/opt/include OTEL_LIB=/opt/lib TARGET=linux-glibc
|
|
```
|
|
|
|
### Configuration
|
|
|
|
The filter uses a two-file configuration model:
|
|
|
|
1. **OTel configuration file** (`.cfg`) -- defines the telemetry model:
|
|
instrumentation settings, scopes and groups.
|
|
2. **YAML configuration file** (`.yml`) -- defines the OpenTelemetry SDK
|
|
pipeline: exporters, samplers, processors, providers and signal routing.
|
|
|
|
#### Activating the filter
|
|
|
|
Add the filter to a HAProxy proxy section (frontend/listen/backend):
|
|
|
|
```
|
|
frontend my-frontend
|
|
...
|
|
filter opentelemetry [id <id>] config <file>
|
|
...
|
|
```
|
|
|
|
If no filter id is specified, `otel-filter` is used as default.
|
|
|
|
#### OTel configuration file structure
|
|
|
|
The OTel configuration file contains three section types:
|
|
|
|
- `otel-instrumentation` -- mandatory; references the YAML file, sets rate
|
|
limits, error modes, logging and declares groups and scopes.
|
|
- `otel-scope` -- defines actions (spans, attributes, metrics, logs) triggered
|
|
by stream events or from groups.
|
|
- `otel-group` -- a named collection of scopes triggered from HAProxy TCP/HTTP
|
|
rules.
|
|
|
|
#### Minimal YAML configuration
|
|
|
|
```yaml
|
|
exporters:
|
|
my_exporter:
|
|
type: otlp_http
|
|
endpoint: "http://localhost:4318/v1/traces"
|
|
|
|
samplers:
|
|
my_sampler:
|
|
type: always_on
|
|
|
|
processors:
|
|
my_processor:
|
|
type: batch
|
|
|
|
providers:
|
|
my_provider:
|
|
resources:
|
|
- service.name: "haproxy"
|
|
|
|
signals:
|
|
traces:
|
|
scope_name: "HAProxy OTel"
|
|
exporters: my_exporter
|
|
samplers: my_sampler
|
|
processors: my_processor
|
|
providers: my_provider
|
|
```
|
|
|
|
#### Supported YAML exporters
|
|
|
|
| Type | Description |
|
|
|-----------------|---------------------------------------|
|
|
| `otlp_grpc` | OTLP over gRPC |
|
|
| `otlp_http` | OTLP over HTTP (JSON or Protobuf) |
|
|
| `otlp_file` | Local files in OTLP format |
|
|
| `zipkin` | Zipkin-compatible backends |
|
|
| `elasticsearch` | Elasticsearch |
|
|
| `ostream` | Text output to a file (for debugging) |
|
|
| `memory` | In-memory buffer (for testing) |
|
|
|
|
### Scope keywords
|
|
|
|
| Keyword | Description |
|
|
|----------------|---------------------------------------------------------|
|
|
| `span` | Create or reference a span |
|
|
| `attribute` | Set key-value span attributes |
|
|
| `event` | Add timestamped span events |
|
|
| `baggage` | Set context propagation data |
|
|
| `status` | Set span status (ok/error/ignore/unset) |
|
|
| `link` | Add span links to related spans |
|
|
| `inject` | Inject context into headers or variables |
|
|
| `extract` | Extract context from headers or variables |
|
|
| `finish` | Close spans (supports wildcards: `*`, `*req*`, `*res*`) |
|
|
| `instrument` | Create or update metric instruments |
|
|
| `log-record` | Emit a log record with severity |
|
|
| `otel-event` | Bind scope to a filter event with optional ACL |
|
|
| `idle-timeout` | Set periodic event interval for idle streams |
|
|
|
|
### CLI commands
|
|
|
|
Available via the HAProxy CLI socket (prefix: `flt-otel`):
|
|
|
|
| Command | Description |
|
|
|----------------------------|------------------------------------|
|
|
| `flt-otel status` | Show filter status |
|
|
| `flt-otel enable` | Enable the filter |
|
|
| `flt-otel disable` | Disable the filter |
|
|
| `flt-otel hard-errors` | Enable hard-errors mode |
|
|
| `flt-otel soft-errors` | Disable hard-errors mode |
|
|
| `flt-otel logging [state]` | Set logging state |
|
|
| `flt-otel rate [value]` | Set or show the rate limit |
|
|
| `flt-otel debug [level]` | Set debug level (debug build only) |
|
|
|
|
When invoked without arguments, `rate`, `logging` and `debug` display the
|
|
current value.
|
|
|
|
### Performance
|
|
|
|
Benchmark results from the standalone (`sa`) configuration, which exercises all
|
|
events (worst-case scenario):
|
|
|
|
| Rate limit | Req/s | Avg latency | Overhead |
|
|
|------------|--------|-------------|----------|
|
|
| 100.0% | 38,202 | 213.08 us | 21.6% |
|
|
| 50.0% | 42,777 | 190.49 us | 12.2% |
|
|
| 25.0% | 45,302 | 180.46 us | 7.0% |
|
|
| 10.0% | 46,879 | 174.69 us | 3.7% |
|
|
| 2.5% | 47,993 | 170.58 us | 1.4% |
|
|
| disabled | 48,788 | 167.74 us | ~0 |
|
|
| off | 48,697 | 168.00 us | baseline |
|
|
|
|
With a rate limit of 10% or less, the performance impact is negligible.
|
|
Detailed methodology and additional results are in the `test/` directory.
|
|
|
|
### Test configurations
|
|
|
|
The `test/` directory contains ready-to-run example configurations:
|
|
|
|
- **sa** -- standalone; the most comprehensive example, demonstrating spans,
|
|
attributes, events, links, baggage, status, metrics, log records, ACL
|
|
conditions and idle-timeout events.
|
|
- **fe/be** -- distributed tracing across two cascaded HAProxy instances using
|
|
HTTP header-based context propagation.
|
|
- **ctx** -- context propagation via HAProxy variables using the inject/extract
|
|
mechanism.
|
|
- **cmp** -- minimal configuration for benchmarking comparison.
|
|
- **empty** -- filter initialized with no active telemetry.
|
|
|
|
#### Quick start with Jaeger
|
|
|
|
Start a Jaeger all-in-one container:
|
|
|
|
```
|
|
docker run -d --name jaeger -p 4317:4317 -p 4318:4318 -p 16686:16686 jaegertracing/all-in-one:latest
|
|
```
|
|
|
|
Run one of the test configurations:
|
|
|
|
```
|
|
./test/run-sa.sh
|
|
```
|
|
|
|
Open the Jaeger UI at `http://localhost:16686` to view traces.
|
|
|
|
### Documentation
|
|
|
|
Detailed documentation is available in the following files:
|
|
|
|
- [README](README) -- complete reference documentation
|
|
- [README-configuration](README-configuration) -- configuration guide
|
|
- [README-conf](README-conf) -- configuration details
|
|
- [README-design](README-design) -- cross-cutting design patterns
|
|
- [README-implementation](README-implementation) -- component architecture
|
|
- [README-func](README-func) -- function reference
|
|
- [README-misc](README-misc) -- miscellaneous notes
|
|
|
|
### Copyright
|
|
|
|
Copyright 2026 HAProxy Technologies
|
|
|
|
### Author
|
|
|
|
Miroslav Zagorac <mzagorac@haproxy.com>
|