Added the "instrument" keyword to otel-scope sections for recording metric
measurements alongside traces.
Introduced flt_otel_conf_instrument holding instrument type, description,
unit, sample expressions, and optional key-value attributes. The
supported synchronous integer-precision instrument types were counters,
histograms, up-down counters, and gauges.
Instruments followed a two-form design: a "create" form defined a new
instrument with its type and value expression, while an "update" form
recorded measurements against an existing instrument with per-scope
attributes.
Instrument creation was performed lazily at first use with HA_ATOMIC_CAS
to guarantee thread-safe one-time initialization. The configuration
check phase validated that every update-form had a matching create-form
definition and that create-form names were unique across all scopes.
The meter lifecycle was integrated into filter init and deinit, starting
the meter alongside the tracer and shutting it down during cleanup.
Added HAProxy CLI commands that allow runtime inspection and modification
of OTel filter settings without requiring a configuration reload.
The new cli.c module registers CLI keywords under the "otel" prefix and
implements the following commands: flt_otel_cli_parse_status() displays a
comprehensive status report of all OTel filter instances including filter
ID, proxy, disabled state, hard-error mode, logging state, rate limit,
analyzer bits, and SDK diagnostic message count;
flt_otel_cli_parse_disabled() enables or disables filtering across all
instances; flt_otel_cli_parse_option() toggles the hard-error mode that
controls whether errors disable the filter for a stream or are silently
ignored; flt_otel_cli_parse_logging() manages the logging state with
support for off, on, and dontlog-normal modes; flt_otel_cli_parse_rate()
adjusts the sampling rate limit as a floating-point percentage; and
flt_otel_cli_parse_debug() sets the debug verbosity level in debug builds.
All modifications are applied atomically across every OTel filter instance
in every proxy.
The CLI initialization is called from flt_otel_ops_init() during filter
startup via flt_otel_cli_init(), which registers the keyword table through
cli_register_kw().
Supporting changes include the FLT_OTEL_U32_FLOAT macro for converting the
internal uint32_t rate representation to a human-readable percentage, the
FLT_OTEL_PROXIES_LIST_START/END iteration macros for traversing all OTel
filter instances across the proxy list, and flt_otel_filters_dump() for
debug logging of filter instances.
Implemented the scope execution engine that creates OTel spans, evaluates
sample expressions to collect telemetry data, and manages span lifecycle
during request and response processing.
The scope runner flt_otel_scope_run() was expanded from a stub into a
complete implementation that evaluates ACL conditions on the scope,
extracts span contexts from HTTP headers when configured, iterates over
the scope's span definitions calling flt_otel_scope_run_span() for each,
marks and finishes completed spans, and cleans up unused runtime
resources.
The span runner flt_otel_scope_run_span() creates OTel spans via the
tracer with optional parent references (from other spans or extracted
contexts), collects telemetry by calling flt_otel_sample_add() for each
configured attribute, event, baggage and status entry, then applies the
collected data to the span (attributes, events with their own key-value
arrays, baggage items, and status code with description) and injects the
span context into HTTP headers when configured.
The sample evaluation layer converts HAProxy sample expressions into OTel
telemetry data. flt_otel_sample_add() evaluates each sample expression
against the stream, converts the result via flt_otel_sample_to_value()
which preserves native types (booleans as OTELC_VALUE_BOOL, integers as
OTELC_VALUE_INT64, all others as strings), and routes the key-value pair
to the appropriate collector based on the sample type (attribute, event,
baggage, or status). The key-value arrays grow dynamically using the
FLT_OTEL_ATTR_INIT_SIZE and FLT_OTEL_ATTR_INC_SIZE constants.
Span finishing is handled in two phases: flt_otel_scope_finish_mark()
marks spans and contexts for completion using exact name matching or
wildcards ("*" for all, "*req*" for request-direction, "*res*" for
response-direction), and flt_otel_scope_finish_marked() ends all marked
spans with a common monotonic timestamp and destroys their contexts.
Implemented the flt_otel_ops_check() callback that validates the parsed
OTel filter configuration after all HAProxy configuration sections have
been processed.
The check callback performs the following validations: resolves deferred
sample fetch arguments under full frontend and backend capabilities,
verifies uniqueness of filter IDs across all proxies, ensures the
instrumentation section and its configuration file are present, checks
for duplicate group and scope section names, verifies that groups are not
empty, resolves group-to-scope and instrumentation-to-group/scope
cross-references by linking placeholder entries to their definitions,
detects unused scopes, counts root spans and warns when the count differs
from one, and accumulates the required channel analyzer bits from all used
scopes into the instrumentation configuration.
The commit also added the flt_otel_counters structure to track per-event
diagnostic counters in debug builds, the FLT_OTEL_ALERT macro for
filter-scoped error messages, and the FLT_OTEL_DBG_LIST macro for
iterating and dumping named configuration lists.
Added the configuration structures that model the OTel filter's
instrumentation hierarchy and the utility functions that support the
configuration parser.
The configuration is organized as a tree rooted at flt_otel_conf, which
holds the proxy reference, filter identity, and lists of groups and
scopes. Below it, flt_otel_conf_instr carries the instrumentation
settings: tracer handle, rate limiting, hard-error mode, logging state,
channel analyzers, and placeholder references to groups and scopes.
Groups (flt_otel_conf_group) aggregate scopes by name. Scopes
(flt_otel_conf_scope) bind an event to its ACL condition, span context
declarations, span definitions and a list of spans scheduled for
finishing. Spans (flt_otel_conf_span) carry attributes, events,
baggages and status entries, each represented as flt_otel_conf_sample
structures that pair a key with concatenated sample-expression arguments.
All configuration types share a common header macro (FLT_OTEL_CONF_HDR)
that embeds an identifier string, its length, a configuration line number,
and a list link. Their init and free functions are generated by the
FLT_OTEL_CONF_FUNC_INIT and FLT_OTEL_CONF_FUNC_FREE macros in
conf_funcs.h, with per-type custom initialization and cleanup bodies.
The utility layer in util.c provides argument counting and concatenation
for the configuration parser, sample data to string conversion covering
boolean, integer, IPv4, IPv6, string and HTTP method types, and debug
helpers for dumping argument arrays and linked list state.
The OpenTelemetry (OTel) filter enables distributed tracing of requests
across service boundaries, export of metrics such as request rates,
latencies and error counts, and structured logging tied to trace context,
giving operators a unified view of HAProxy traffic through any
OpenTelemetry-compatible backend.
The OTel filter is implemented using the standard HAProxy stream filter
API. Stream filters attach to proxies and intercept traffic at each stage
of processing: they receive callbacks on stream creation and destruction,
channel analyzer events, HTTP header and payload processing, and TCP data
forwarding. This allows the filter to collect telemetry data at every
stage of the request/response lifecycle without modifying the core proxy
logic.
This commit added the minimum set of files required for the filter to
compile: the addon Makefile with pkg-config-based detection of the
opentelemetry-c-wrapper library, header files with configuration
constants, utility macros and type definitions, and the source files
containing stub filter operation callbacks registered through
flt_otel_ops and the "opentelemetry" keyword parser entry point.
The filter uses the opentelemetry-c-wrapper library from HAProxy
Technologies, which provides a C interface to the OpenTelemetry C++ SDK.
This wrapper allows HAProxy, a C codebase, to leverage the full
OpenTelemetry observability pipeline without direct C++ dependencies
in the HAProxy source tree.
https://github.com/haproxytech/opentelemetry-c-wrapperhttps://github.com/open-telemetry/opentelemetry-cpp
Build options:
USE_OTEL - enable the OpenTelemetry filter
OTEL_DEBUG - compile the filter in debug mode
OTEL_INC - force the include path to the C wrapper
OTEL_LIB - force the library path to the C wrapper
OTEL_RUNPATH - add the C wrapper RUNPATH to the executable
Example build with OTel and debug enabled:
make -j8 USE_OTEL=1 OTEL_DEBUG=1 TARGET=linux-glibc
