mirrors/haproxy
1
0
Fork 0
mirror of https://git.haproxy.org/git/haproxy.git/ synced 2026-04-18 12:14:56 +02:00
Code Issues Projects Releases Wiki Activity

DOC: otel: added documentation

Browse Source 
Added the full documentation set for the OpenTelemetry filter.

The main README served as the user-facing guide covering build
instructions, core OpenTelemetry concepts, the complete filter
configuration reference, usage examples with worked scenarios,
CLI commands, and known limitations.

Supplementary documents provided a detailed configuration guide with
worked examples (README-configuration), an internal C structure reference
for developers (README-conf), a function reference organized by source
file (README-func), an architecture and implementation review
(README-implementation), and miscellaneous notes (README-misc).
This commit is contained in:
Miroslav Zagorac 2026-04-12 12:07:17 +02:00 committed by William Lallemand
parent 579d1247f2
commit 8d8d7e4602
6 changed files with 4569 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1137
addons/otel/README Normal file
View File

File diff suppressed because it is too large Load Diff

454
addons/otel/README-conf Normal file
View File

@ -0,0 +1,454 @@
OpenTelemetry Filter Configuration Structures
==============================================================================
1 Overview
------------------------------------------------------------------------------
The OpenTelemetry filter configuration is a tree of C structures that mirrors
the hierarchical layout of the filter's configuration file. Each structure type
carries a common header macro, and its allocation and deallocation are performed
by macro-generated init/free function pairs defined in conf_funcs.h.
The root of the tree is flt_otel_conf, which owns the instrumentation settings,
groups, and scopes. Scopes contain the actual tracing and metrics definitions:
contexts, spans, instruments, and their sample expressions.
Source files:
include/conf.h Structure definitions and debug macros.
include/conf_funcs.h Init/free macro templates and declarations.
src/conf.c Init/free implementations for all types.
2 Common Macros
------------------------------------------------------------------------------
Two macros provide the building blocks embedded in every configuration
structure.
2.1 FLT_OTEL_CONF_STR(p)
Expands to an anonymous struct containing a string pointer and its cached
length:
struct {
char *p;
size_t p_len;
};
Used for auxiliary string fields that do not need list linkage (e.g. ref_id and
ctx_id in flt_otel_conf_span).
2.2 FLT_OTEL_CONF_HDR(p)
Expands to an anonymous struct that extends FLT_OTEL_CONF_STR with a
configuration file line number and an intrusive list node:
struct {
char *p;
size_t p_len;
int cfg_line;
struct list list;
};
Every configuration structure embeds FLT_OTEL_CONF_HDR as its first member.
The <p> parameter names the identifier field (e.g. "id", "key", "str", "span",
"fmt_expr"). The list node chains the structure into its parent's list.
The cfg_line records the source line for error reporting.
3 Structure Hierarchy
------------------------------------------------------------------------------
The complete ownership tree, from root to leaves:
flt_otel_conf
+-- flt_otel_conf_instr (one, via pointer)
| +-- flt_otel_conf_ph (ph_groups list)
| +-- flt_otel_conf_ph (ph_scopes list)
| +-- struct acl (acls list, HAProxy-owned type)
| +-- struct logger (proxy_log.loggers, HAProxy type)
+-- flt_otel_conf_group (groups list)
| +-- flt_otel_conf_ph (ph_scopes list)
+-- flt_otel_conf_scope (scopes list)
+-- flt_otel_conf_context (contexts list)
+-- flt_otel_conf_span (spans list)
| +-- flt_otel_conf_link (links list)
| +-- flt_otel_conf_sample (attributes list)
| | +-- flt_otel_conf_sample_expr (exprs list)
| +-- flt_otel_conf_sample (events list)
| | +-- flt_otel_conf_sample_expr (exprs list)
| +-- flt_otel_conf_sample (baggages list)
| | +-- flt_otel_conf_sample_expr (exprs list)
| +-- flt_otel_conf_sample (statuses list)
| +-- flt_otel_conf_sample_expr (exprs list)
+-- flt_otel_conf_str (spans_to_finish list)
+-- flt_otel_conf_instrument (instruments list)
| +-- flt_otel_conf_sample (samples list)
| +-- flt_otel_conf_sample_expr (exprs list)
+-- flt_otel_conf_log_record (log_records list)
+-- flt_otel_conf_sample (samples list)
+-- flt_otel_conf_sample_expr (exprs list)
All child lists use HAProxy's intrusive doubly-linked list (struct list)
threaded through the FLT_OTEL_CONF_HDR embedded in each child structure.
3.1 Placeholder Structures
The flt_otel_conf_ph structure serves as an indirection node. During parsing,
placeholder entries record names of groups and scopes. At check time
(flt_otel_check), these names are resolved to pointers to the actual
flt_otel_conf_group or flt_otel_conf_scope structures via the ptr field.
Two type aliases exist for clarity:
#define flt_otel_conf_ph_group flt_otel_conf_ph
#define flt_otel_conf_ph_scope flt_otel_conf_ph
Corresponding free aliases ensure the FLT_OTEL_LIST_DESTROY macro can locate
the correct free function:
#define flt_otel_conf_ph_group_free flt_otel_conf_ph_free
#define flt_otel_conf_ph_scope_free flt_otel_conf_ph_free
4 Structure Definitions
------------------------------------------------------------------------------
4.1 flt_otel_conf (root)
proxy Proxy owning the filter.
id The OpenTelemetry filter id.
cfg_file The OpenTelemetry filter configuration file name.
instr The OpenTelemetry instrumentation settings (pointer).
groups List of all available groups.
scopes List of all available scopes.
cnt Various counters related to filter operation.
smp_args Deferred sample fetch arguments to resolve at check time.
This structure does not use FLT_OTEL_CONF_HDR because it is not part of any
list -- it is the unique root, owned by the filter instance.
4.2 flt_otel_conf_instr (instrumentation)
FLT_OTEL_CONF_HDR(id) The OpenTelemetry instrumentation name.
config The OpenTelemetry configuration file name.
tracer The OpenTelemetry tracer handle.
meter The OpenTelemetry meter handle.
logger The OpenTelemetry logger handle.
rate_limit Rate limit as uint32 ([0..2^32-1] maps [0..100]%).
flag_harderr Hard-error mode flag.
flag_disabled Disabled flag.
logging Logging mode (0, 1, or 3).
proxy_log The log server list (HAProxy proxy structure).
analyzers Defined channel analyzers bitmask.
idle_timeout Minimum idle timeout across scopes (ms, 0 = off).
acls ACLs declared on this tracer.
ph_groups List of all used groups (placeholders).
ph_scopes List of all used scopes (placeholders).
Exactly one instrumentation block is allowed per filter instance. The parser
stores a pointer to it in flt_otel_conf.instr.
4.3 flt_otel_conf_group
FLT_OTEL_CONF_HDR(id) The group name.
flag_used The indication that the group is being used.
ph_scopes List of all used scopes (placeholders).
Groups bundle scopes for use with the "otel-group" HAProxy action.
4.4 flt_otel_conf_scope
FLT_OTEL_CONF_HDR(id) The scope name.
flag_used The indication that the scope is being used.
event FLT_OTEL_EVENT_* identifier.
idle_timeout Idle timeout interval in milliseconds (0 = off).
acls ACLs declared on this scope.
cond ACL condition to meet.
contexts Declared contexts.
spans Declared spans.
spans_to_finish The list of spans scheduled for finishing.
instruments The list of metric instruments.
log_records The list of log records.
Each scope binds to a single HAProxy analyzer event (or none, if used only
through groups).
4.5 flt_otel_conf_span
FLT_OTEL_CONF_HDR(id) The name of the span.
FLT_OTEL_CONF_STR(ref_id) The reference name, if used.
FLT_OTEL_CONF_STR(ctx_id) The span context name, if used.
ctx_flags The type of storage used for the span context.
flag_root Whether this is a root span.
links The set of linked span names.
attributes The set of key:value attributes.
events The set of events with key-value attributes.
baggages The set of key:value baggage items.
statuses Span status code and description.
The ref_id and ctx_id fields use FLT_OTEL_CONF_STR because they are simple name
strings without list linkage.
4.6 flt_otel_conf_instrument
FLT_OTEL_CONF_HDR(id) The name of the instrument.
idx Meter instrument index: UNSET (-1) before creation,
PENDING (-2) while another thread is creating, or >= 0
for the actual meter index.
type Instrument type (or UPDATE).
aggr_type Aggregation type for the view (create only).
description Instrument description (create only).
unit Instrument unit (create only).
samples Sample expressions for the value.
bounds Histogram bucket boundaries (create only).
bounds_num Number of histogram bucket boundaries.
attr Instrument attributes (update only).
attr_len Number of instrument attributes.
ref Resolved create-form instrument (update only).
Instruments come in two forms: create-form (defines a new metric with type,
description, unit, and optional histogram bounds) and update-form (references
an existing instrument via the ref pointer).
4.7 flt_otel_conf_log_record
FLT_OTEL_CONF_HDR(id) Required by macro; member <id> is not used directly.
severity The severity level.
event_id Optional event identifier.
event_name Optional event name.
span Optional span reference.
attr Log record attributes.
attr_len Number of log record attributes.
samples Sample expressions for the body.
Log records are emitted via the OTel logger at the configured severity. The
optional span reference associates the log record with an open span at runtime.
Attributes are stored as key-value pairs added via the 'attr' keyword, which
can be repeated.
4.8 flt_otel_conf_context
FLT_OTEL_CONF_HDR(id) The name of the context.
flags Storage type from which the span context is extracted.
4.9 flt_otel_conf_sample
FLT_OTEL_CONF_HDR(key) The list containing sample names.
fmt_string Combined sample-expression arguments string.
extra Optional supplementary data.
exprs Used to chain sample expressions.
num_exprs Number of defined expressions.
lf_expr The log-format expression.
lf_used Whether lf_expr is used instead of exprs.
The extra field carries type-specific data: event name strings (OTELC_VALUE_DATA)
for span events, status code integers (OTELC_VALUE_INT32) for span statuses.
When the sample value argument contains the "%[" sequence, the parser treats
it as a log-format string: the lf_used flag is set and the compiled result is
stored in lf_expr, while the exprs list remains empty. At runtime, if lf_used
is true, the log-format expression is evaluated via build_logline() instead of
the sample expression list.
4.10 flt_otel_conf_sample_expr
FLT_OTEL_CONF_HDR(fmt_expr) The original expression format string.
expr The sample expression (struct sample_expr).
4.11 Simple Types
flt_otel_conf_hdr Generic header; used for simple named entries.
flt_otel_conf_str String holder (identical to conf_hdr in layout, but the
HDR field is named "str" instead of "id"); used for
spans_to_finish.
flt_otel_conf_link Span link reference; HDR field named "span".
flt_otel_conf_ph Placeholder; carries a ptr field resolved at check time.
5 Initialization
------------------------------------------------------------------------------
5.1 Macro-Generated Init Functions
The FLT_OTEL_CONF_FUNC_INIT macro (conf_funcs.h) generates a function with the
following signature for each configuration type:
struct flt_otel_conf_<type> *flt_otel_conf_<type>_init(const char *id, int line, struct list *head, char **err);
The generated function performs these steps:
1. Validates that <id> is non-NULL and non-empty.
2. Checks the identifier length against FLT_OTEL_ID_MAXLEN (64).
3. If <head> is non-NULL, iterates the list to reject duplicate identifiers
(strcmp match).
4. Allocates the structure with OTELC_CALLOC (zeroed memory).
5. Records the configuration line number in cfg_line.
6. Duplicates the identifier string with OTELC_STRDUP.
7. If <head> is non-NULL, appends the structure to the list via LIST_APPEND.
8. Executes any custom initialization body provided as the third macro
argument.
If any step fails, the function sets an error message via FLT_OTEL_ERR and
returns NULL.
5.2 Custom Initialization Bodies
Several structure types require additional setup beyond what the macro template
provides. The custom init body runs after the base allocation and list
insertion succeed:
conf_sample:
LIST_INIT for exprs. Calls lf_expr_init for lf_expr.
conf_span:
LIST_INIT for links, attributes, events, baggages, statuses.
conf_scope:
LIST_INIT for acls, contexts, spans, spans_to_finish, instruments,
log_records.
conf_group:
LIST_INIT for ph_scopes.
conf_instrument:
Sets idx and type to OTELC_METRIC_INSTRUMENT_UNSET, aggr_type to
OTELC_METRIC_AGGREGATION_UNSET. LIST_INIT for samples.
conf_log_record:
LIST_INIT for samples.
conf_instr:
Sets rate_limit to FLT_OTEL_FLOAT_U32(100.0) (100%). Calls init_new_proxy
for proxy_log. LIST_INIT for acls, ph_groups, ph_scopes.
Types with no custom body (hdr, str, link, ph, sample_expr, context) rely
entirely on the zeroed OTELC_CALLOC allocation.
5.3 Extended Sample Initialization
The flt_otel_conf_sample_init_ex function (conf.c) provides a higher-level
initialization for sample structures:
1. Verifies sufficient arguments in the args[] array.
2. Calls flt_otel_conf_sample_init with the sample key.
3. Copies extra data (event name string or status code integer).
4. Concatenates remaining arguments into the fmt_string via
flt_otel_args_concat.
5. Counts the number of sample expressions.
This function is used by the parser for span attributes, events, baggages,
statuses, and instrument samples.
5.4 Top-Level Initialization
The flt_otel_conf_init function (conf.c) is hand-written rather than
macro-generated because the root structure does not follow the standard header
pattern:
1. Allocates flt_otel_conf with OTELC_CALLOC.
2. Stores the proxy reference.
3. Initializes the groups and scopes lists.
6 Deallocation
------------------------------------------------------------------------------
6.1 Macro-Generated Free Functions
The FLT_OTEL_CONF_FUNC_FREE macro (conf_funcs.h) generates a function with the
following signature:
void flt_otel_conf_<type>_free(struct flt_otel_conf_<type> **ptr);
The generated function performs these steps:
1. Checks that both <ptr> and <*ptr> are non-NULL.
2. Executes any custom cleanup body provided as the third macro argument.
3. Frees the identifier string with OTELC_SFREE.
4. Removes the structure from its list with FLT_OTEL_LIST_DEL.
5. Frees the structure with OTELC_SFREE_CLEAR and sets <*ptr> to NULL.
6.2 Custom Cleanup Bodies
Custom cleanup runs before the base teardown, allowing child structures to be
freed while the parent is still valid:
conf_sample:
Frees fmt_string. If extra is OTELC_VALUE_DATA, frees the data pointer.
Destroys the exprs list (sample_expr entries). Deinitializes lf_expr via
lf_expr_deinit.
conf_sample_expr:
Releases the HAProxy sample expression via release_sample_expr.
conf_span:
Frees ref_id and ctx_id strings.
Destroys links, attributes, events, baggages, and statuses lists.
conf_instrument:
Frees description, unit, and bounds. Destroys the samples list.
Destroys the attr key-value array via otelc_kv_destroy.
conf_log_record:
Frees event_name and span strings. Destroys the attr key-value array via
otelc_kv_destroy. Destroys the samples list.
conf_scope:
Prunes and frees each ACL entry. Frees the ACL condition via free_acl_cond.
Destroys contexts, spans, spans_to_finish, instruments, and log_records
lists.
conf_group:
Destroys the ph_scopes list.
conf_instr:
Frees the config string. Prunes and frees each ACL entry. Frees each
logger entry from proxy_log.loggers. Destroys the ph_groups and ph_scopes
lists.
Types with no custom cleanup (hdr, str, link, ph, context) only run the base
teardown: free the identifier, unlink, free the structure.
6.3 List Destruction
The FLT_OTEL_LIST_DESTROY(type, head) macro (defined in define.h) iterates a
list and calls flt_otel_conf_<type>_free for each entry. This macro drives the
recursive teardown from parent to leaf.
6.4 Top-Level Deallocation
The flt_otel_conf_free function (conf.c) is hand-written:
1. Frees the id and cfg_file strings.
2. Calls flt_otel_conf_instr_free for the instrumentation.
3. Destroys the groups list (which recursively frees placeholders).
4. Destroys the scopes list (which recursively frees contexts, spans,
instruments, log records, and all their children).
5. Frees the root structure and sets the pointer to NULL.
7 Summary of Init/Free Pairs
------------------------------------------------------------------------------
The following table lists all configuration types and their init/free function
pairs. Types marked "macro" are generated by the FLT_OTEL_CONF_FUNC_(INIT|FREE)
macros. The HDR field column shows which member name is used for the common
header.
Type HDR field Source Custom init body
--------------- --------- ------ -------------------------
conf (none) manual groups, scopes
conf_hdr id macro (none)
conf_str str macro (none)
conf_link span macro (none)
conf_ph id macro (none)
conf_sample_expr fmt_expr macro (none)
conf_sample key macro exprs, lf_expr
conf_sample (ex) key manual extra, fmt_string, exprs
conf_context id macro (none)
conf_span id macro 5 sub-lists
conf_instrument id macro idx, type, samples
conf_log_record id macro samples
conf_scope id macro 6 sub-lists
conf_group id macro ph_scopes
conf_instr id macro rate_limit, proxy_log, acls, ph_groups, ph_scopes

946
addons/otel/README-configuration Normal file
View File

@ -0,0 +1,946 @@
-----------------------------------------
HAProxy OTel filter configuration guide
Version 1.0
( Last update: 2026-03-18 )
-----------------------------------------
Author : Miroslav Zagorac
Contact : mzagorac at haproxy dot com
SUMMARY
--------
1. Overview
2. HAProxy filter declaration
3. OTel configuration file structure
3.1. OTel scope (top-level)
3.2. "otel-instrumentation" section
3.3. "otel-scope" section
3.4. "otel-group" section
4. YAML configuration file
4.1. Exporters
4.2. Samplers
4.3. Processors
4.4. Readers
4.5. Providers
4.6. Signals
5. HAProxy rule integration
6. Complete examples
6.1. Standalone example (sa)
6.2. Frontend / backend example (fe/be)
6.3. Context propagation example (ctx)
6.4. Comparison example (cmp)
6.5. Empty / minimal example (empty)
1. Overview
------------
The OTel filter configuration consists of two files:
1) An OTel configuration file (.cfg) that defines the tracing model: scopes,
groups, spans, attributes, events, instrumentation and log-records.
2) A YAML configuration file (.yml) that configures the OpenTelemetry SDK
pipeline: exporters, samplers, processors, readers, providers and signal
routing.
The OTel configuration file is referenced from the HAProxy configuration using
the 'filter opentelemetry' directive. The YAML file is in turn referenced from
the OTel configuration file using the 'config' keyword inside the
"otel-instrumentation" section.
2. HAProxy filter declaration
------------------------------
The OTel filter requires the 'insecure-fork-wanted' keyword in the HAProxy
'global' section. This is necessary because the OpenTelemetry C++ SDK creates
background threads for data export and batch processing. HAProxy will refuse
to load the configuration if this keyword is missing.
global
insecure-fork-wanted
...
The filter is activated by adding a filter directive in the HAProxy
configuration, in a proxy section (frontend / listen / backend):
frontend my-frontend
...
filter opentelemetry [id <id>] config <otel-cfg-file>
...
If no filter id is specified, 'otel-filter' is used as default. The 'config'
parameter is mandatory and specifies the path to the OTel configuration file.
Example (from test/sa/haproxy.cfg):
frontend otel-test-sa-frontend
bind *:10080
default_backend servers-backend
acl acl-http-status-ok status 100:399
filter opentelemetry id otel-test-sa config sa/otel.cfg
http-response otel-group otel-test-sa http_response_group if acl-http-status-ok
http-after-response otel-group otel-test-sa http_after_response_group if !acl-http-status-ok
backend servers-backend
server server-1 127.0.0.1:8000
3. OTel configuration file structure
--------------------------------------
The OTel configuration file uses a simple section-based format. It contains
three types of sections: one "otel-instrumentation" section (mandatory), zero
or more "otel-scope" sections, and zero or more "otel-group" sections.
3.1. OTel scope (top-level)
-----------------------------
The file is organized into top-level OTel scopes, each identified by a filter
id enclosed in square brackets. The filter id must match the id specified in
the HAProxy 'filter opentelemetry' directive.
[<filter-id>]
otel-instrumentation <name>
...
otel-group <name>
...
otel-scope <name>
...
Multiple OTel scopes (for different filter instances) can coexist in the same
file:
[my-first-filter]
otel-instrumentation instr1
...
[my-second-filter]
otel-instrumentation instr2
...
3.2. "otel-instrumentation" section
-------------------------------------
Exactly one "otel-instrumentation" section must be defined per OTel scope.
It configures the global behavior of the filter and declares which groups
and scopes are active.
Syntax:
otel-instrumentation <name>
Keywords (mandatory):
config <file>
Path to the YAML configuration file for the OpenTelemetry SDK.
Keywords (optional):
acl <aclname> <criterion> [flags] [operator] <value> ...
Declare an ACL. See section 7 of the HAProxy Configuration Manual.
debug-level <value>
Set the debug level bitmask (e.g. 0x77f). Only effective when compiled
with OTEL_DEBUG=1.
groups <name> ...
Declare one or more "otel-group" sections used by this instrumentation.
Can be repeated on multiple lines.
log global
log <addr> [len <len>] [format <fmt>] <facility> [<level> [<minlvl>]]
no log
Enable per-instance logging.
option disabled / no option disabled
Disable or enable the filter. Default: enabled.
option dontlog-normal / no option dontlog-normal
Suppress logging for normal (successful) operations. Default: disabled.
option hard-errors / no option hard-errors
Stop all filter processing in a stream after the first error. Default:
disabled (errors are non-fatal).
rate-limit <value>
Percentage of streams for which the filter is activated. Floating-point
value from 0.0 to 100.0. Default: 100.0.
scopes <name> ...
Declare one or more "otel-scope" sections used by this instrumentation.
Can be repeated on multiple lines.
Example (from test/sa/otel.cfg):
[otel-test-sa]
otel-instrumentation otel-test-instrumentation
debug-level 0x77f
log localhost:514 local7 debug
config sa/otel.yml
option dontlog-normal
option hard-errors
no option disabled
rate-limit 100.0
groups http_response_group
groups http_after_response_group
scopes on_stream_start
scopes on_stream_stop
scopes client_session_start
scopes frontend_tcp_request
...
scopes server_session_end
3.3. "otel-scope" section
---------------------------
An "otel-scope" section defines the actions that take place when a particular
event fires or when a group is triggered.
Syntax:
otel-scope <name>
Supported keywords:
span <name> [parent <ref>] [link <ref>] [root]
Create a new span or reference an already opened one.
- 'root' marks this span as the trace root (only one per trace).
- 'parent <ref>' sets the parent to an existing span or extracted context
name.
- 'link <ref>' adds an inline link to another span or context. Multiple
inline links can be specified within the argument limit.
- If no reference is given, the span becomes a root span.
A span declaration opens a "sub-context" within the scope: the keywords
'link', 'attribute', 'event', 'baggage', 'status' and 'inject' that follow
apply to that span until the next 'span' keyword or the end of the scope.
Examples:
span "HAProxy session" root
span "Client session" parent "HAProxy session"
span "HTTP request" parent "TCP request" link "HAProxy session"
span "Client session" parent "otel_ctx_1"
attribute <key> <sample> ...
Set an attribute on the currently active span. A single sample preserves
its native type; multiple samples are concatenated as a string.
Examples:
attribute "http.method" method
attribute "http.url" url
attribute "http.version" str("HTTP/") req.ver
event <name> <key> <sample> ...
Add a span event (timestamped annotation) to the currently active span.
The data type is always string.
Examples:
event "event_ip" "src" src str(":") src_port
event "event_be" "be" be_id str(" ") be_name
baggage <key> <sample> ...
Set baggage on the currently active span. Baggage propagates to all child
spans. The data type is always string.
Example:
baggage "haproxy_id" var(sess.otel.uuid)
status <code> [<sample> ...]
Set the span status. Valid codes: ignore (default), unset, ok, error.
An optional description follows the code.
Examples:
status "ok"
status "error" str("http.status_code: ") status
link <span> ...
Add non-hierarchical links to the currently active span. Multiple span
names can be specified. Use this keyword for multiple links (the inline
'link' in 'span' is limited to one).
Example:
link "HAProxy session" "Client session"
inject <name-prefix> [use-vars] [use-headers]
Inject span context into an HTTP header carrier and/or HAProxy variables.
The prefix names the context; the special prefix '-' generates the name
automatically. Default storage: use-headers. The 'use-vars' option
requires OTEL_USE_VARS=1 at compile time.
Example:
span "HAProxy session" root
inject "otel_ctx_1" use-headers use-vars
extract <name-prefix> [use-vars | use-headers]
Extract a previously injected span context from an HTTP header or HAProxy
variables. The extracted context can then be used as a parent reference
in 'span ... parent <name-prefix>'.
Example:
extract "otel_ctx_1" use-vars
span "Client session" parent "otel_ctx_1"
finish <name> ...
Close one or more spans or span contexts. Special names:
'*' - finish all open spans
'*req*' - finish all request-channel spans
'*res*' - finish all response-channel spans
Multiple names can be given on one line. A quoted context name after a
span name finishes the associated context as well.
Examples:
finish "Frontend TCP request"
finish "Client session" "otel_ctx_2"
finish *
instrument <type> <name> [aggr <aggregation>] [desc <description>] [unit <unit>] value <sample> [bounds <bounds>]
instrument update <name> [attr <key> <value> ...]
Create or update a metric instrument.
Supported types:
cnt_int - counter (uint64)
hist_int - histogram (uint64)
udcnt_int - up-down counter (int64)
gauge_int - gauge (int64)
Supported aggregation types:
drop - measurements are discarded
histogram - explicit bucket histogram
last_value - last recorded value
sum - sum of recorded values
default - SDK default for the instrument type
exp_histogram - base-2 exponential histogram
An aggregation type can be specified using the 'aggr' keyword. When
specified, a metrics view is registered with the given aggregation
strategy. If omitted, the SDK default is used.
For histogram instruments (hist_int), optional bucket boundaries can be
specified using the 'bounds' keyword followed by a double-quoted string
of space-separated numbers (order does not matter; values are sorted
internally). When bounds are specified without an explicit aggregation
type, histogram aggregation is used automatically.
Observable (asynchronous) and double-precision types are not supported.
Observable instrument callbacks are invoked by the OTel SDK from an
external background thread; HAProxy sample fetches rely on internal
per-thread-group state and return incorrect results from a non-HAProxy
thread. Double-precision types are not supported because HAProxy sample
fetches do not return double values.
Examples:
instrument cnt_int "name_cnt_int" desc "Integer Counter" value int(1),add(2) unit "unit"
instrument hist_int "name_hist" aggr exp_histogram desc "Latency" value lat_ns_tot unit "ns"
instrument hist_int "name_hist2" desc "Latency" value lat_ns_tot unit "ns" bounds "100 1000 10000"
instrument update "name_cnt_int" attr "attr_1_key" "attr_1_value"
log-record <severity> [id <integer>] [event <name>] [span <span-name>] [attr <key> <value>] ... <sample> ...
Emit an OpenTelemetry log record. The first argument is a required
severity level. Optional keywords follow in any order:
id <integer> - numeric event identifier
event <name> - event name string
span <span-name> - associate the log record with an open span
attr <key> <value> - add a key-value attribute (repeatable)
The remaining arguments at the end are sample fetch expressions that form
the log record body. A single sample preserves its native type; multiple
samples are concatenated as a string.
Supported severity levels follow the OpenTelemetry specification:
trace, trace2, trace3, trace4
debug, debug2, debug3, debug4
info, info2, info3, info4
warn, warn2, warn3, warn4
error, error2, error3, error4
fatal, fatal2, fatal3, fatal4
The log record is only emitted if the logger is enabled for the configured
severity (controlled by the 'min_severity' option in the YAML logs signal
configuration). If a 'span' reference is given but the named span is not
found at runtime, the log record is emitted without span correlation.
Examples:
log-record info str("heartbeat")
log-record info id 1001 event "http-request" span "Frontend HTTP request" attr "http.method" "GET" method url
log-record trace id 1000 event "session-start" span "Client session" attr "attr_1_key" "attr_1_value" attr "attr_2_key" "attr_2_value" src str(":") src_port
log-record warn event "server-unavailable" str("503 Service Unavailable")
log-record info event "session-stop" str("stream stopped")
acl <aclname> <criterion> [flags] [operator] <value> ...
Declare an ACL local to this scope.
Example:
acl acl-test-src-ip src 127.0.0.1
otel-event <name> [{ if | unless } <condition>]
Bind this scope to a filter event, optionally with an ACL-based condition.
Supported events (stream lifecycle):
on-stream-start
on-stream-stop
on-idle-timeout
on-backend-set
Supported events (request channel):
on-client-session-start
on-frontend-tcp-request
on-http-wait-request
on-http-body-request
on-frontend-http-request
on-switching-rules-request
on-backend-tcp-request
on-backend-http-request
on-process-server-rules-request
on-http-process-request
on-tcp-rdp-cookie-request
on-process-sticking-rules-request
on-http-headers-request
on-http-end-request
on-client-session-end
on-server-unavailable
Supported events (response channel):
on-server-session-start
on-tcp-response
on-http-wait-response
on-process-store-rules-response
on-http-response
on-http-headers-response
on-http-end-response
on-http-reply
on-server-session-end
The on-stream-start event fires from the stream_start filter callback,
before any channel processing begins. The on-stream-stop event fires from
the stream_stop callback, after all channel processing ends. No channel
is available at that point, so context injection/extraction via HTTP
headers cannot be used in scopes bound to these events.
The on-idle-timeout event fires periodically when the stream has no data
transfer activity. It requires the 'idle-timeout' keyword to set the
interval. Scopes bound to this event can create heartbeat spans, record
idle-time metrics, and emit idle-time log records.
The on-backend-set event fires when a backend is assigned to the stream.
It is not called if the frontend and backend are the same.
The on-http-headers-request and on-http-headers-response events fire after
all HTTP headers have been parsed and analyzed.
The on-http-end-request and on-http-end-response events fire when all HTTP
data has been processed and forwarded.
The on-http-reply event fires when HAProxy generates an internal reply
(error page, deny response, redirect).
Examples:
otel-event on-stream-start if acl-test-src-ip
otel-event on-stream-stop
otel-event on-client-session-start
otel-event on-client-session-start if acl-test-src-ip
otel-event on-http-response if !acl-http-status-ok
otel-event on-idle-timeout
idle-timeout <time>
Set the idle timeout interval for a scope bound to the 'on-idle-timeout'
event. The timer fires periodically at the given interval when the stream
has no data transfer activity. This keyword is mandatory for scopes using
the 'on-idle-timeout' event and cannot be used with any other event.
The <time> argument accepts the standard HAProxy time format: a number
followed by a unit suffix (ms, s, m, h, d). A value of zero is not
permitted.
Example:
scopes on_idle_timeout
..
otel-scope on_idle_timeout
idle-timeout 5s
span "heartbeat" root
attribute "idle.elapsed" str("idle-check")
instrument cnt_int "idle.count" value int(1)
log-record info str("heartbeat")
otel-event on-idle-timeout
3.4. "otel-group" section
---------------------------
An "otel-group" section defines a named collection of scopes that can be
triggered from HAProxy TCP/HTTP rules rather than from filter events.
Syntax:
otel-group <name>
Keywords:
scopes <name> ...
List the "otel-scope" sections that belong to this group. Multiple names
can be given on one line. Scopes that are used only in groups do not need
to define an 'otel-event'.
Example (from test/sa/otel.cfg):
otel-group http_response_group
scopes http_response_1
scopes http_response_2
otel-scope http_response_1
span "HTTP response"
event "event_content" "hdr.content" res.hdr("content-type") str("; length: ") res.hdr("content-length") str(" bytes")
otel-scope http_response_2
span "HTTP response"
event "event_date" "hdr.date" res.hdr("date") str(" / ") res.hdr("last-modified")
4. YAML configuration file
----------------------------
The YAML configuration file defines the OpenTelemetry SDK pipeline. It is
referenced by the 'config' keyword in the "otel-instrumentation" section.
It contains the following top-level sections: exporters, samplers, processors,
readers, providers and signals.
4.1. Exporters
---------------
Each exporter has a user-chosen name and a 'type' that determines which
additional options are available. Options marked with (*) are required.
Supported types:
otlp_grpc - Export via OTLP over gRPC.
type (*) "otlp_grpc"
thread_name exporter thread name (string)
endpoint OTLP/gRPC endpoint URL (string)
use_ssl_credentials enable SSL channel credentials (boolean)
ssl_credentials_cacert_path CA certificate file path (string)
ssl_credentials_cacert_as_string CA certificate as inline string (string)
ssl_client_key_path client private key file path (string)
ssl_client_key_string client private key as inline string (string)
ssl_client_cert_path client certificate file path (string)
ssl_client_cert_string client certificate as inline string (string)
timeout export timeout in seconds (integer)
user_agent User-Agent header value (string)
max_threads maximum exporter threads (integer)
compression compression algorithm name (string)
max_concurrent_requests concurrent request limit (integer)
otlp_http - Export via OTLP over HTTP (JSON or Protobuf).
type (*) "otlp_http"
thread_name exporter thread name (string)
endpoint OTLP/HTTP endpoint URL (string)
content_type payload format: "json" or "protobuf"
json_bytes_mapping binary encoding: "hexid", "utf8" or "base64"
debug enable debug output (boolean)
timeout export timeout in seconds (integer)
http_headers custom HTTP headers (list of key: value)
max_concurrent_requests concurrent request limit (integer)
max_requests_per_connection request limit per connection (integer)
ssl_insecure_skip_verify skip TLS certificate verification (boolean)
ssl_ca_cert_path CA certificate file path (string)
ssl_ca_cert_string CA certificate as inline string (string)
ssl_client_key_path client private key file path (string)
ssl_client_key_string client private key as inline string (string)
ssl_client_cert_path client certificate file path (string)
ssl_client_cert_string client certificate as inline string (string)
ssl_min_tls minimum TLS version (string)
ssl_max_tls maximum TLS version (string)
ssl_cipher TLS cipher list (string)
ssl_cipher_suite TLS 1.3 cipher suite list (string)
compression compression algorithm name (string)
otlp_file - Export to local files in OTLP format.
type (*) "otlp_file"
thread_name exporter thread name (string)
file_pattern output filename pattern (string)
alias_pattern symlink pattern for latest file (string)
flush_interval flush interval in microseconds (integer)
flush_count spans per flush (integer)
file_size maximum file size in bytes (integer)
rotate_size number of rotated files to keep (integer)
ostream - Write to a file (text output, useful for debugging).
type (*) "ostream"
filename output file path (string)
memory - In-memory buffer (useful for testing).
type (*) "memory"
buffer_size maximum buffered items (integer)
zipkin - Export to Zipkin-compatible backends.
type (*) "zipkin"
endpoint Zipkin collector URL (string)
format payload format: "json" or "protobuf"
service_name service name reported to Zipkin (string)
ipv4 service IPv4 address (string)
ipv6 service IPv6 address (string)
elasticsearch - Export to Elasticsearch.
type (*) "elasticsearch"
host Elasticsearch hostname (string)
port Elasticsearch port (integer)
index Elasticsearch index name (string)
response_timeout response timeout in seconds (integer)
debug enable debug output (boolean)
http_headers custom HTTP headers (list of key: value)
4.2. Samplers
--------------
Samplers control which traces are recorded. Each sampler has a user-chosen
name and a 'type' that determines its behavior.
Supported types:
always_on - Sample every trace.
type (*) "always_on"
always_off - Sample no traces.
type (*) "always_off"
trace_id_ratio_based - Sample a fraction of traces.
type (*) "trace_id_ratio_based"
ratio sampling ratio, 0.0 to 1.0 (float)
parent_based - Inherit sampling decision from parent span.
type (*) "parent_based"
delegate fallback sampler name (string)
4.3. Processors
----------------
Processors define how telemetry data is handled before export. Each
processor has a user-chosen name and a 'type' that determines its behavior.
Supported types:
batch - Batch spans before exporting.
type (*) "batch"
thread_name processor thread name (string)
max_queue_size maximum queued spans (integer)
schedule_delay export interval in milliseconds (integer)
max_export_batch_size maximum spans per export call (integer)
When the queue reaches half capacity, a preemptive notification triggers
an early export.
single - Export each span individually (no batching).
type (*) "single"
4.4. Readers
-------------
Readers define how metrics are collected and exported. Each reader has a
user-chosen name.
thread_name reader thread name (string)
export_interval collection interval in milliseconds (integer)
export_timeout export timeout in milliseconds (integer)
4.5. Providers
---------------
Providers define resource attributes attached to all telemetry data. Each
provider has a user-chosen name.
resources key-value resource attributes (list)
Standard resource attribute keys include service.name, service.version,
service.instance.id and service.namespace.
4.6. Signals
-------------
Signals bind exporters, samplers, processors, readers and providers together
for each telemetry type. The supported signal names are "traces", "metrics"
and "logs".
scope_name instrumentation scope name (string)
exporters exporter name reference (string)
samplers sampler name reference (string, traces only)
processors processor name reference (string, traces/logs)
readers reader name reference (string, metrics only)
providers provider name reference (string)
min_severity minimum log severity level (string, logs only)
The "min_severity" option controls which log records are emitted. Only log
records whose severity is equal to or higher than the configured minimum are
passed to the exporter. The value is a severity name as listed under the
"log-record" keyword (e.g. "trace", "debug", "info", "warn", "error", "fatal").
If omitted, the logger accepts all severity levels.
5. HAProxy rule integration
----------------------------
Groups defined in the OTel configuration file can be triggered from HAProxy
TCP/HTTP rules using the 'otel-group' action keyword:
http-request otel-group <filter-id> <group> [condition]
http-response otel-group <filter-id> <group> [condition]
http-after-response otel-group <filter-id> <group> [condition]
tcp-request otel-group <filter-id> <group> [condition]
tcp-response otel-group <filter-id> <group> [condition]
This allows running specific groups of scopes based on ACL conditions defined
in the HAProxy configuration.
Example (from test/sa/haproxy.cfg):
acl acl-http-status-ok status 100:399
filter opentelemetry id otel-test-sa config sa/otel.cfg
# Run response scopes for successful responses
http-response otel-group otel-test-sa http_response_group if acl-http-status-ok
# Run after-response scopes for error responses
http-after-response otel-group otel-test-sa http_after_response_group if !acl-http-status-ok
6. Complete examples
---------------------
The test directory contains several complete example configurations. Each
subdirectory contains an OTel configuration file (otel.cfg), a YAML file
(otel.yml) and a HAProxy configuration file (haproxy.cfg).
6.1. Standalone example (sa)
------------------------------
The most comprehensive example. All possible events are used, with spans,
attributes, events, links, baggage, status, metrics and groups demonstrated.
--- test/sa/otel.cfg (excerpt) -----------------------------------------
[otel-test-sa]
otel-instrumentation otel-test-instrumentation
config sa/otel.yml
option dontlog-normal
option hard-errors
no option disabled
rate-limit 100.0
groups http_response_group
groups http_after_response_group
scopes on_stream_start
scopes on_stream_stop
scopes client_session_start
scopes frontend_tcp_request
...
scopes server_session_end
otel-group http_response_group
scopes http_response_1
scopes http_response_2
otel-scope http_response_1
span "HTTP response"
event "event_content" "hdr.content" res.hdr("content-type") str("; length: ") res.hdr("content-length") str(" bytes")
otel-scope on_stream_start
instrument udcnt_int "haproxy.sessions.active" desc "Active sessions" value int(1) unit "{session}"
span "HAProxy session" root
baggage "haproxy_id" var(sess.otel.uuid)
event "event_ip" "src" src str(":") src_port
acl acl-test-src-ip src 127.0.0.1
otel-event on-stream-start if acl-test-src-ip
otel-scope on_stream_stop
finish *
otel-event on-stream-stop
otel-scope client_session_start
span "Client session" parent "HAProxy session"
otel-event on-client-session-start
otel-scope frontend_http_request
span "Frontend HTTP request" parent "HTTP body request" link "HAProxy session"
attribute "http.method" method
attribute "http.url" url
attribute "http.version" str("HTTP/") req.ver
finish "HTTP body request"
otel-event on-frontend-http-request
otel-scope server_session_start
span "Server session" parent "HAProxy session"
link "HAProxy session" "Client session"
finish "Process sticking rules request"
otel-event on-server-session-start
otel-scope server_session_end
finish *
otel-event on-server-session-end
---------------------------------------------------------------------
6.2. Frontend / backend example (fe/be)
-----------------------------------------
Demonstrates distributed tracing across two cascaded HAProxy instances using
inject/extract to propagate the span context via HTTP headers.
The frontend HAProxy (test/fe) creates the root trace and injects context:
--- test/fe/otel.cfg (excerpt) -----------------------------------------
otel-scope backend_http_request
span "Backend HTTP request" parent "Backend TCP request"
finish "Backend TCP request"
span "HAProxy session"
inject "otel-ctx" use-headers
otel-event on-backend-http-request
---------------------------------------------------------------------
The backend HAProxy (test/be) extracts the context and continues the trace:
--- test/be/otel.cfg (excerpt) -----------------------------------------
otel-scope frontend_http_request
extract "otel-ctx" use-headers
span "HAProxy session" parent "otel-ctx" root
baggage "haproxy_id" var(sess.otel.uuid)
span "Client session" parent "HAProxy session"
span "Frontend HTTP request" parent "Client session"
attribute "http.method" method
attribute "http.url" url
attribute "http.version" str("HTTP/") req.ver
otel-event on-frontend-http-request
---------------------------------------------------------------------
6.3. Context propagation example (ctx)
----------------------------------------
Similar to 'sa', but spans are opened using extracted span contexts as parent
references instead of direct span names. This demonstrates the inject/extract
mechanism using HAProxy variables.
--- test/ctx/otel.cfg (excerpt) ----------------------------------------
otel-scope client_session_start_1
span "HAProxy session" root
inject "otel_ctx_1" use-headers use-vars
baggage "haproxy_id" var(sess.otel.uuid)
otel-event on-client-session-start
otel-scope client_session_start_2
extract "otel_ctx_1" use-vars
span "Client session" parent "otel_ctx_1"
inject "otel_ctx_2" use-headers use-vars
otel-event on-client-session-start
otel-scope frontend_tcp_request
extract "otel_ctx_2" use-vars
span "Frontend TCP request" parent "otel_ctx_2"
inject "otel_ctx_3" use-headers use-vars
otel-event on-frontend-tcp-request
otel-scope http_wait_request
extract "otel_ctx_3" use-vars
span "HTTP wait request" parent "otel_ctx_3"
finish "Frontend TCP request" "otel_ctx_3"
otel-event on-http-wait-request
---------------------------------------------------------------------
6.4. Comparison example (cmp)
-------------------------------
A configuration made for comparison purposes with other tracing implementations.
It uses a simplified span hierarchy without context propagation.
--- test/cmp/otel.cfg (excerpt) ----------------------------------------
otel-scope client_session_start
span "HAProxy session" root
baggage "haproxy_id" var(sess.otel.uuid)
span "Client session" parent "HAProxy session"
otel-event on-client-session-start
otel-scope http_response-error
span "HTTP response"
status "error" str("!acl-http-status-ok")
otel-event on-http-response if !acl-http-status-ok
otel-scope server_session_end
finish "HTTP response" "Server session"
otel-event on-http-response
otel-scope client_session_end
finish "*"
otel-event on-http-response
---------------------------------------------------------------------
6.5. Empty / minimal example (empty)
--------------------------------------
The minimal valid OTel configuration. The filter is initialized but no events
are triggered:
--- test/empty/otel.cfg -------------------------------------------------
otel-instrumentation otel-test-instrumentation
config empty/otel.yml
---------------------------------------------------------------------
This is useful for testing the OTel filter initialization behavior without any
actual telemetry processing.

715
addons/otel/README-func Normal file
View File

@ -0,0 +1,715 @@
OpenTelemetry filter -- function reference
==========================================================================
Functions are grouped by source file. Functions marked with [D] are only
compiled when DEBUG_OTEL is defined.
src/filter.c
----------------------------------------------------------------------
Filter lifecycle callbacks and helpers registered in flt_otel_ops.
flt_otel_mem_malloc
Allocator callback for the OTel C wrapper library. Uses the HAProxy
pool_head_otel_span_context pool.
flt_otel_mem_free
Deallocator callback for the OTel C wrapper library.
flt_otel_log_handler_cb
Diagnostic callback for the OTel C wrapper library. Counts SDK internal
diagnostic messages.
flt_otel_thread_id
Returns the current HAProxy thread ID (tid).
flt_otel_lib_init
Initializes the OTel C wrapper library: verifies the library version,
constructs the configuration path, calls otelc_init(), and creates the
tracer, meter and logger instances.
flt_otel_is_disabled
Checks whether the filter instance is disabled for the current stream.
Logs the event name when DEBUG_OTEL is enabled.
flt_otel_return_int
Error handler for callbacks returning int. In hard-error mode, disables
the filter; in soft-error mode, clears the error and returns OK.
flt_otel_return_void
Error handler for callbacks returning void. Same logic as
flt_otel_return_int but without a return value.
flt_otel_ops_init
Filter init callback (flt_ops.init). Called once per proxy to initialize
the OTel library via flt_otel_lib_init() and register CLI keywords.
flt_otel_ops_deinit
Filter deinit callback (flt_ops.deinit). Destroys the tracer, meter and
logger, frees the configuration, and calls otelc_deinit().
flt_otel_ops_check
Filter check callback (flt_ops.check). Validates the parsed
configuration: checks for duplicate filter IDs, resolves group/scope
placeholder references, verifies root span count, and sets analyzer bits.
flt_otel_ops_init_per_thread
Per-thread init callback (flt_ops.init_per_thread). Starts the OTel
tracer thread and enables HTX filtering.
flt_otel_ops_deinit_per_thread [D]
Per-thread deinit callback (flt_ops.deinit_per_thread).
flt_otel_ops_attach
Filter attach callback (flt_ops.attach). Called when a filter instance is
attached to a stream. Applies rate limiting, creates the runtime context,
and sets analyzer bits.
flt_otel_ops_stream_start
Stream start callback (flt_ops.stream_start). Fires the
on-stream-start event before any channel processing begins. The channel
argument is NULL. After the event, initializes the idle timer in the
runtime context from the precomputed minimum idle_timeout in the
instrumentation configuration.
flt_otel_ops_stream_set_backend
Stream set-backend callback (flt_ops.stream_set_backend). Fires the
on-backend-set event when a backend is assigned to the stream.
flt_otel_ops_stream_stop
Stream stop callback (flt_ops.stream_stop). Fires the
on-stream-stop event after all channel processing ends. The channel
argument is NULL.
flt_otel_ops_detach
Filter detach callback (flt_ops.detach). Frees the runtime context when
the filter is detached from a stream.
flt_otel_ops_check_timeouts
Timeout callback (flt_ops.check_timeouts). When the idle-timeout timer
has expired, fires the on-idle-timeout event and reschedules the timer
for the next interval. Sets the STRM_EVT_MSG pending event flag on the
stream.
flt_otel_ops_channel_start_analyze
Channel start-analyze callback. Registers analyzers on the channel and
runs the client/server session start event. Propagates the idle-timeout
expiry to the channel's analyse_exp so the stream task keeps waking.
flt_otel_ops_channel_pre_analyze
Channel pre-analyze callback. Maps the analyzer bit to an event index and
runs the corresponding event.
flt_otel_ops_channel_post_analyze
Channel post-analyze callback. Non-resumable; called once when a
filterable analyzer finishes.
flt_otel_ops_channel_end_analyze
Channel end-analyze callback. Runs the client/server session end event.
For the request channel, also fires the server-unavailable event if no
response was processed.
flt_otel_ops_http_headers
HTTP headers callback (flt_ops.http_headers). Fires
on-http-headers-request or on-http-headers-response depending on the
channel direction.
flt_otel_ops_http_payload [D]
HTTP payload callback (flt_ops.http_payload).
flt_otel_ops_http_end
HTTP end callback (flt_ops.http_end). Fires on-http-end-request or
on-http-end-response depending on the channel direction.
flt_otel_ops_http_reset [D]
HTTP reset callback (flt_ops.http_reset).
flt_otel_ops_http_reply
HTTP reply callback (flt_ops.http_reply). Fires the on-http-reply event
when HAProxy generates an internal reply.
flt_otel_ops_tcp_payload [D]
TCP payload callback (flt_ops.tcp_payload).
src/event.c
----------------------------------------------------------------------
Event dispatching, metrics recording and scope/span execution engine.
flt_otel_scope_run_instrument_record
Records a measurement for a synchronous metric instrument. Evaluates the
sample expression from the create-form instrument (instr_ref) and submits
the value to the meter via update_instrument_kv_n(), using per-scope
attributes from the update-form instrument (instr).
flt_otel_scope_run_instrument
Processes all metric instruments for a scope. Runs in two passes: the
first lazily creates create-form instruments via the meter, using
HA_ATOMIC_CAS to guarantee thread-safe one-time initialization; the second
iterates update-form instruments and records measurements via
flt_otel_scope_run_instrument_record(). Instruments whose index is still
negative (UNUSED or PENDING) are skipped.
flt_otel_scope_run_log_record
Emits log records for a scope. Iterates over the configured log-record
list, skipping entries whose severity is below the logger threshold.
Evaluates the body from sample fetch expressions or a log-format string,
optionally resolves a span reference against the runtime context, and
emits the record via the logger. A missing span is non-fatal -- the
record is emitted without span correlation.
flt_otel_scope_run_span
Executes a single span: creates the OTel span on first call, adds links,
baggage, attributes, events and status, then injects the context into HTTP
headers or HAProxy variables.
flt_otel_scope_run
Executes a complete scope: evaluates ACL conditions, extracts contexts,
iterates over configured spans (resolving links, evaluating sample
expressions), calls flt_otel_scope_run_span for each, processes metric
instruments via flt_otel_scope_run_instrument(), emits log records via
flt_otel_scope_run_log_record(), then marks and finishes completed spans.
flt_otel_event_run
Top-level event dispatcher. Called from filter callbacks, iterates over
all scopes matching the event index and calls flt_otel_scope_run() for
each.
src/scope.c
----------------------------------------------------------------------
Runtime context, span and context lifecycle management.
flt_otel_pools_info [D]
Logs the sizes of all registered HAProxy memory pools used by the OTel
filter.
flt_otel_runtime_context_init
Allocates and initializes the per-stream runtime context. Generates a
UUID and stores it in the sess.otel.uuid HAProxy variable.
flt_otel_runtime_context_free
Frees the runtime context: ends all active spans, destroys all extracted
contexts, and releases pool memory.
flt_otel_scope_span_init
Finds an existing scope span by name or creates a new one. Resolves the
parent reference (span or extracted context).
flt_otel_scope_span_free
Frees a scope span entry if its OTel span has been ended. Refuses to free
an active (non-NULL) span.
flt_otel_scope_context_init
Finds an existing scope context by name or creates a new one by extracting
the span context from a text map.
flt_otel_scope_context_free
Frees a scope context entry and destroys the underlying OTel span context.
flt_otel_scope_data_dump [D]
Dumps scope data contents (baggage, attributes, events, links, status) for
debugging.
flt_otel_scope_data_init
Zero-initializes a scope data structure and its event/link lists.
flt_otel_scope_data_free
Frees all scope data contents: key-value arrays, event entries, link
entries, and status description.
flt_otel_scope_finish_mark
Marks spans and contexts for finishing. Supports wildcard ("*"),
channel-specific ("req"/"res"), and named targets.
flt_otel_scope_finish_marked
Ends all spans and destroys all contexts that have been marked for
finishing by flt_otel_scope_finish_mark().
flt_otel_scope_free_unused
Removes scope spans with NULL OTel span and scope contexts with NULL OTel
context. Cleans up associated HTTP headers and variables.
src/parser.c
----------------------------------------------------------------------
Configuration file parsing for otel-instrumentation, otel-group and otel-scope
sections.
flt_otel_parse_strdup
Duplicates a string with error handling; optionally stores the string
length.
flt_otel_parse_keyword
Parses a single keyword argument: checks for duplicates and missing
values, then stores via flt_otel_parse_strdup().
flt_otel_parse_invalid_char
Validates characters in a name according to the specified type
(identifier, domain, context prefix, variable).
flt_otel_parse_cfg_check
Common validation for config keywords: looks up the keyword, checks
argument count and character validity, verifies that the parent section ID
is set.
flt_otel_parse_cfg_sample_expr
Parses a single HAProxy sample expression within a sample definition.
Calls sample_parse_expr().
flt_otel_parse_cfg_sample
Parses a complete sample definition (key plus one or more sample
expressions).
flt_otel_parse_cfg_str
Parses one or more string arguments into a conf_str list (used for the
"finish" keyword).
flt_otel_parse_cfg_file
Parses and validates a file path argument; checks that the file exists and
is readable.
flt_otel_parse_check_scope
Checks whether the current config parsing is within the correct HAProxy
configuration scope (cfg_scope filtering).
flt_otel_parse_cfg_instr
Section parser for the otel-instrumentation block. Handles keywords:
otel-instrumentation ID, log, config, groups, scopes, acl, rate-limit,
option, debug-level.
flt_otel_post_parse_cfg_instr
Post-parse callback for otel-instrumentation. Links the instrumentation
to the config and checks that a config file is specified.
flt_otel_parse_cfg_group
Section parser for the otel-group block. Handles keywords: otel-group ID,
scopes.
flt_otel_post_parse_cfg_group
Post-parse callback for otel-group. Checks that at least one scope is
defined.
flt_otel_parse_cfg_scope_ctx
Parses the context storage type argument ("use-headers" or "use-vars") for
inject/extract keywords.
flt_otel_parse_acl
Builds an ACL condition by trying multiple ACL lists in order
(scope-local, instrumentation, proxy).
flt_otel_parse_bounds
Parses a space-separated string of numbers into a dynamically allocated
array of doubles for histogram bucket boundaries. Sorts the values
internally.
flt_otel_parse_cfg_instrument
Parses the "instrument" keyword inside an otel-scope section. Supports
both "update" form (referencing an existing instrument) and "create" form
(defining a new metric instrument with type, name, optional aggregation
type, description, unit, value, and optional histogram bounds).
flt_otel_parse_cfg_scope
Section parser for the otel-scope block. Handles keywords: otel-scope ID,
span, link, attribute, event, baggage, status, inject, extract, finish,
instrument, log-record, acl, otel-event.
flt_otel_post_parse_cfg_scope
Post-parse callback for otel-scope. Checks that HTTP header injection is
only used on events that support it.
flt_otel_parse_cfg
Parses the OTel filter configuration file. Backs up current sections,
registers temporary otel-instrumentation/group/scope section parsers,
loads and parses the file, then restores the original sections.
flt_otel_parse
Main filter parser entry point, registered for the "otel" filter keyword.
Parses the filter ID and configuration file path from the HAProxy config
line.
src/conf.c
----------------------------------------------------------------------
Configuration structure allocation and deallocation. Most init/free pairs are
generated by the FLT_OTEL_CONF_FUNC_INIT and FLT_OTEL_CONF_FUNC_FREE macros.
flt_otel_conf_hdr_init
Allocates and initializes a conf_hdr structure.
flt_otel_conf_hdr_free
Frees a conf_hdr structure and removes it from its list.
flt_otel_conf_str_init
Allocates and initializes a conf_str structure.
flt_otel_conf_str_free
Frees a conf_str structure and removes it from its list.
flt_otel_conf_link_init
Allocates and initializes a conf_link structure (span link).
flt_otel_conf_link_free
Frees a conf_link structure and removes it from its list.
flt_otel_conf_ph_init
Allocates and initializes a conf_ph (placeholder) structure.
flt_otel_conf_ph_free
Frees a conf_ph structure and removes it from its list.
flt_otel_conf_sample_expr_init
Allocates and initializes a conf_sample_expr structure.
flt_otel_conf_sample_expr_free
Frees a conf_sample_expr structure and releases the parsed sample
expression.
flt_otel_conf_sample_init
Allocates and initializes a conf_sample structure.
flt_otel_conf_sample_init_ex
Extended sample initialization: sets the key, extra data (event name or
status code), concatenated value string, and expression count.
flt_otel_conf_sample_free
Frees a conf_sample structure including its value, extra data, and all
sample expressions.
flt_otel_conf_context_init
Allocates and initializes a conf_context structure.
flt_otel_conf_context_free
Frees a conf_context structure and removes it from its list.
flt_otel_conf_span_init
Allocates and initializes a conf_span structure with empty lists for
links, attributes, events, baggages and statuses.
flt_otel_conf_span_free
Frees a conf_span structure and all its child lists.
flt_otel_conf_instrument_init
Allocates and initializes a conf_instrument structure.
flt_otel_conf_instrument_free
Frees a conf_instrument structure and removes it from its list.
flt_otel_conf_log_record_init
Allocates and initializes a conf_log_record structure with an empty
samples list.
flt_otel_conf_log_record_free
Frees a conf_log_record structure: event_name, span, attributes and
samples list.
flt_otel_conf_scope_init
Allocates and initializes a conf_scope structure with empty lists for
ACLs, contexts, spans, spans_to_finish and instruments.
flt_otel_conf_scope_free
Frees a conf_scope structure, ACLs, condition, and all child lists.
flt_otel_conf_group_init
Allocates and initializes a conf_group structure with an empty placeholder
scope list.
flt_otel_conf_group_free
Frees a conf_group structure and its placeholder scope list.
flt_otel_conf_instr_init
Allocates and initializes a conf_instr structure. Sets the default rate
limit to 100%, initializes the proxy_log, and creates empty ACL and
placeholder lists.
flt_otel_conf_instr_free
Frees a conf_instr structure including ACLs, loggers, config path, and
placeholder lists.
flt_otel_conf_init
Allocates and initializes the top-level flt_otel_conf structure with empty
group and scope lists.
flt_otel_conf_free
Frees the top-level flt_otel_conf structure and all of its children
(instrumentation, groups, scopes).
src/cli.c
----------------------------------------------------------------------
HAProxy CLI command handlers for runtime filter management.
cmn_cli_set_msg
Sets the CLI appctx response message and state.
flt_otel_cli_parse_debug [D]
CLI handler for "otel debug [level]". Gets or sets the debug level.
flt_otel_cli_parse_disabled
CLI handler for "otel enable" and "otel disable".
flt_otel_cli_parse_option
CLI handler for "otel soft-errors" and "otel hard-errors".
flt_otel_cli_parse_logging
CLI handler for "otel logging [state]". Gets or sets the logging state
(off/on/nolognorm).
flt_otel_cli_parse_rate
CLI handler for "otel rate [value]". Gets or sets the rate limit
percentage.
flt_otel_cli_parse_status
CLI handler for "otel status". Displays filter configuration and runtime
state for all OTel filter instances.
flt_otel_cli_init
Registers the OTel CLI keywords with HAProxy.
src/otelc.c
----------------------------------------------------------------------
OpenTelemetry context propagation bridge (inject/extract) between HAProxy and
the OTel C wrapper library.
flt_otel_text_map_writer_set_cb
Writer callback for text map injection. Appends a key-value pair to the
text map.
flt_otel_http_headers_writer_set_cb
Writer callback for HTTP headers injection. Appends a key-value pair to
the text map.
flt_otel_inject_text_map
Injects span context into a text map carrier.
flt_otel_inject_http_headers
Injects span context into an HTTP headers carrier.
flt_otel_text_map_reader_foreach_key_cb
Reader callback for text map extraction. Iterates over all key-value
pairs in the text map.
flt_otel_http_headers_reader_foreach_key_cb
Reader callback for HTTP headers extraction. Iterates over all key-value
pairs in the text map.
flt_otel_extract_text_map
Extracts a span context from a text map carrier via the tracer.
flt_otel_extract_http_headers
Extracts a span context from an HTTP headers carrier via the tracer.
src/http.c
----------------------------------------------------------------------
HTTP header manipulation for context propagation.
flt_otel_http_headers_dump [D]
Dumps all HTTP headers from the channel's HTX buffer.
flt_otel_http_headers_get
Extracts HTTP headers matching a prefix into a text map. Used by the
"extract" keyword to read span context from incoming request headers.
flt_otel_http_header_set
Sets or removes an HTTP header. Combines prefix and name into the full
header name, removes all existing occurrences, then adds the new value
(if non-NULL).
flt_otel_http_headers_remove
Removes all HTTP headers matching a prefix. Wrapper around
flt_otel_http_header_set() with NULL name and value.
src/vars.c
----------------------------------------------------------------------
HAProxy variable integration for context propagation and storage. Only compiled
when USE_OTEL_VARS is defined.
flt_otel_vars_scope_dump [D]
Dumps all variables for a single HAProxy variable scope.
flt_otel_vars_dump [D]
Dumps all variables across all scopes (PROC, SESS, TXN, REQ/RES).
flt_otel_smp_init
Initializes a sample structure with stream ownership and optional string
data.
flt_otel_smp_add
Appends a context variable name to the binary sample data buffer used for
tracking registered context variables.
flt_otel_normalize_name
Normalizes a variable name: replaces dashes with 'D' and spaces with 'S',
converts to lowercase.
flt_otel_denormalize_name
Reverses the normalization applied by flt_otel_normalize_name(). Restores
dashes from 'D' and spaces from 'S'.
flt_otel_var_name
Constructs a full variable name from scope, prefix and name components,
separated by dots.
flt_otel_ctx_loop
Iterates over all context variable names stored in the binary sample data,
calling a callback for each.
flt_otel_ctx_set_cb
Callback for flt_otel_ctx_loop() that checks whether a context variable
name already exists.
flt_otel_ctx_set
Registers a context variable name in the binary tracking buffer if it is
not already present.
flt_otel_var_register
Registers a HAProxy variable via vars_check_arg() so it can be used at
runtime.
flt_otel_var_set
Sets a HAProxy variable value. For context-scope variables, also
registers the name in the context tracking buffer.
flt_otel_vars_unset_cb
Callback for flt_otel_ctx_loop() that unsets each context variable.
flt_otel_vars_unset
Unsets all context variables for a given prefix and removes the tracking
variable itself.
flt_otel_vars_get_scope
Resolves a scope name string ("proc", "sess", "txn", "req", "res") to the
corresponding HAProxy variable store.
flt_otel_vars_get_cb
Callback for flt_otel_ctx_loop() that reads each context variable value
and adds it to a text map.
flt_otel_vars_get
Reads all context variables for a prefix into a text map. Used by the
"extract" keyword with variable storage.
src/pool.c
----------------------------------------------------------------------
Memory pool and trash buffer helpers.
flt_otel_pool_alloc
Allocates memory from a HAProxy pool (if available) or from the heap.
Optionally zero-fills the allocated block.
flt_otel_pool_strndup
Duplicates a string using a HAProxy pool (if available) or the heap.
flt_otel_pool_free
Returns memory to a HAProxy pool or frees it from the heap.
flt_otel_trash_alloc
Allocates a trash buffer chunk, optionally zero-filled.
flt_otel_trash_free
Frees a trash buffer chunk.
src/util.c
----------------------------------------------------------------------
Utility and conversion functions.
flt_otel_args_dump [D]
Dumps configuration arguments array to stderr.
flt_otel_filters_dump [D]
Dumps all OTel filter instances across all proxies.
flt_otel_chn_label [D]
Returns "REQuest" or "RESponse" based on channel flags.
flt_otel_pr_mode [D]
Returns "HTTP" or "TCP" based on proxy mode.
flt_otel_stream_pos [D]
Returns "frontend" or "backend" based on stream flags.
flt_otel_type [D]
Returns "frontend" or "backend" based on filter flags.
flt_otel_analyzer [D]
Returns the analyzer name string for a given analyzer bit.
flt_otel_list_dump [D]
Returns a summary string for a list (empty, single, count).
flt_otel_args_count
Counts the number of valid (non-NULL) arguments in an args array, handling
gaps from blank arguments.
flt_otel_args_concat
Concatenates arguments starting from a given index into a single
space-separated string.
flt_otel_strtod
Parses a string to double with range validation.
flt_otel_strtoll
Parses a string to int64 with range validation.
flt_otel_sample_to_str
Converts sample data to its string representation. Handles bool, sint,
IPv4, IPv6, str, and HTTP method types.
flt_otel_sample_to_value
Converts sample data to an otelc_value. Preserves native types (bool,
int64) where possible; falls back to string.
flt_otel_sample_add_event
Adds a sample value as a span event attribute. Groups attributes by event
name; dynamically grows the attribute array.
flt_otel_sample_set_status
Sets the span status code and description from sample data.
flt_otel_sample_add_kv
Adds a sample value as a key-value attribute or baggage entry.
Dynamically grows the key-value array.
flt_otel_sample_add
Top-level sample evaluator. Processes all sample expressions for a
configured sample, converts results, and dispatches to the appropriate
handler (attribute, event, baggage, status).
src/group.c
----------------------------------------------------------------------
Group action support for http-response / http-after-response / tcp-request /
tcp-response rules.
flt_otel_group_action
Action callback (action_ptr) for the otel-group rule. Finds the filter
instance on the current stream and runs all scopes defined in the group.
flt_otel_group_check
Check callback (check_ptr) for the otel-group rule. Resolves filter ID
and group ID references against the proxy's filter configuration.
flt_otel_group_release
Release callback (release_ptr) for the otel-group rule.
flt_otel_group_parse
Parses the "otel-group" action keyword from HAProxy config rules.
Registered for tcp-request, tcp-response, http-request, http-response and
http-after-response action contexts.

1216
addons/otel/README-implementation Normal file
View File

File diff suppressed because it is too large Load Diff

101
addons/otel/README-misc Normal file
View File

@ -0,0 +1,101 @@
OpenTelemetry filter -- miscellaneous notes
==============================================================================
1 Parsing sample expressions in HAProxy
------------------------------------------------------------------------------
HAProxy provides two entry points for turning a configuration string into an
evaluable sample expression.
1.1 sample_parse_expr()
..............................................................................
Parses a bare sample-fetch name with an optional converter chain. The input is
the raw expression without any surrounding syntax.
Declared in: include/haproxy/sample.h
Defined in: src/sample.c
struct sample_expr *sample_parse_expr(char **str, int *idx, const char *file, int line, char **err_msg, struct arg_list *al, char **endptr);
The function reads from str[*idx] and advances *idx past the consumed tokens.
Configuration example (otel-scope instrument keyword):
instrument my_counter "name" desc req.hdr(host),lower ...
Here "req.hdr(host),lower" is a single configuration token that
sample_parse_expr() receives directly. It recognises the fetch "req.hdr(host)"
and the converter "lower" separated by a comma.
1.2 parse_logformat_string()
..............................................................................
Parses a log-format string that may contain literal text mixed with sample
expressions wrapped in %[...] delimiters.
Declared in: include/haproxy/log.h
Defined in: src/log.c
int parse_logformat_string(const char *fmt, struct proxy *curproxy, struct lf_expr *lf_expr, int options, int cap, char **err);
Configuration example (HAProxy log-format directive):
log-format "host=%[req.hdr(host),lower] status=%[status]"
The %[...] wrapper tells parse_logformat_string() where each embedded sample
expression begins and ends. The text outside the brackets ("host=", " status=")
is emitted as-is.
1.3 Which one to use
..............................................................................
Use sample_parse_expr() when the configuration token is a single, standalone
sample expression (no surrounding text). This is the case for the otel filter
keywords such as "attribute", "event", "baggage", "status", "value", and
similar.
Use parse_logformat_string() when the value is a free-form string that may mix
literal text with zero or more embedded expressions.
2 Signal keywords
------------------------------------------------------------------------------
The OTel filter configuration uses one keyword per signal to create or update
signal-specific objects. The keyword names follow the OpenTelemetry
specification's own terminology rather than using informal synonyms.
Signal Keyword Creates / updates
-------- ----------- ------------------------------------------
Tracing span A trace span.
Metrics instrument A metric instrument (counter, gauge, ...).
Logging log-record A log record.
The tracing keyword follows the same logic. A "trace" is the complete
end-to-end path of a request through a distributed system, composed of one or
more "spans". Each span represents a single unit of work within that trace.
The configuration operates at the span level: it creates individual spans, sets
their parent-child relationships, and attaches attributes and events. Using
"trace" as the keyword would be imprecise because one does not configure a trace
directly; one configures the spans that collectively form a trace.
The metrics keyword is analogous. In the OpenTelemetry data model the
terminology is layered: a "metric" is the aggregated output that the SDK
produces after processing recorded measurements, while an "instrument" is the
concrete object through which those measurements are recorded -- a counter,
histogram, gauge, or up-down counter. The configuration operates at the
instrument level: it creates an instrument of a specific type and records values
through it. Using "metric" as the keyword would be imprecise because one does
not configure a metric directly; one configures an instrument that yields
metrics.
The logging keyword follows the same pattern. A "log" is the broad signal
category, while a "log record" is a single discrete entry within that signal.
The configuration operates at the log-record level: it creates individual log
records with a severity, a body, and optional attributes and span context.
Using "log" as the keyword would be imprecise because one does not configure a
log stream directly; one configures the individual log records that comprise it.