mirrors/haproxy
1
0
Fork 0
mirror of https://git.haproxy.org/git/haproxy.git/ synced 2026-04-18 12:14:56 +02:00
Code Issues Projects Releases Wiki Activity

DOC: otel: test: added speed test guide and benchmark results

Browse Source 
The test directory gained a speed test guide (README-test-speed)
explaining how to run performance benchmarks at various rate-limit levels,
together with benchmark result files for the standalone, composite,
context-propagation, and frontend-backend test configurations.
This commit is contained in:
Miroslav Zagorac 2026-03-25 12:17:03 +01:00 committed by William Lallemand
parent 081a8fd5cf
commit bb2c512d29
6 changed files with 939 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

44
addons/otel/README
View File

@ -20,6 +20,7 @@ SUMMARY
4.3. "otel-scope" section
4.4. "otel-group" section
5. Examples
5.1. Benchmarking results
6. OTel CLI
7. Known bugs and limitations
@ -54,6 +55,12 @@ The OTel filter activation is done explicitly by specifying it in the HAProxy
configuration. If this is not done, the OTel filter in no way participates in
the work of HAProxy.
As for the impact on HAProxy speed, this is documented with test results located
in the test directory (see section 5.1). The speed of operation depends on the
way the filter is used and the complexity of the configuration. In typical
production use with a rate limit of 10% or less, the performance impact should
be negligible (see the 'rate-limit' keyword).
The OTel filter allows intensive use of ACLs, which can be defined anywhere in
the configuration. Thus, it is possible to use the filter only for those
connections that are of interest to us.
@ -1082,6 +1089,43 @@ one of the pre-configured scripts:
% ./run-fe-be.sh
5.1. Benchmarking results
--------------------------
To check the performance impact of the OTel filter, test configurations located
in the test directory have been benchmarked. The test results (with the names
README-speed-xxx, where xxx is the name of the configuration being tested) are
also in the test directory.
Testing was done with the wrk utility using 8 threads, 8 connections and a
5-minute test duration. Detailed results and methodology are documented in the
test/README-test-speed file.
Below is a summary of the 'sa' (standalone) configuration results, which uses
all possible events and demonstrates the worst-case scenario:
---------------------------------------------------------------
rate-limit req/s avg latency overhead
---------------------------------------------------------------
100.0% 38,202 213.08 us 21.6%
50.0% 42,777 190.49 us 12.2%
25.0% 45,302 180.46 us 7.0%
10.0% 46,879 174.69 us 3.7%
2.5% 47,993 170.58 us 1.4%
disabled 48,788 167.74 us ~0
off 48,697 168.00 us baseline
---------------------------------------------------------------
The 'off' baseline is measured with the 'filter opentelemetry' directive
commented out, so the filter is not loaded at all. The 'disabled' level has
the filter loaded but disabled via 'option disabled', so the initialization
overhead is included but no events are processed.
As the table shows, with the rate limit set to 25% the overhead is about 7%.
At 10% the overhead drops to 3.7%. In typical production use with a rate limit
of 10% or less, the performance impact should be negligible.
6. OTel CLI
------------

144
addons/otel/test/README-speed-cmp Normal file
View File

@ -0,0 +1,144 @@
--- rate-limit 100.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 182.58us 129.11us 16.19ms 98.11%
Req/Sec 5.63k 240.83 6.29k 69.74%
Latency Distribution
50% 169.00us
75% 183.00us
90% 209.00us
99% 367.00us
13438310 requests in 5.00m, 3.24GB read
Requests/sec: 44779.51
Transfer/sec: 11.06MB
----------------------------------------------------------------------
--- rate-limit 75.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 180.10us 122.51us 14.57ms 98.00%
Req/Sec 5.70k 253.08 6.28k 70.41%
Latency Distribution
50% 169.00us
75% 184.00us
90% 206.00us
99% 362.00us
13613023 requests in 5.00m, 3.28GB read
Requests/sec: 45361.63
Transfer/sec: 11.20MB
----------------------------------------------------------------------
--- rate-limit 50.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 176.59us 125.34us 15.58ms 98.02%
Req/Sec 5.81k 230.84 6.42k 72.14%
Latency Distribution
50% 166.00us
75% 182.00us
90% 202.00us
99% 361.00us
13888448 requests in 5.00m, 3.35GB read
Requests/sec: 46279.45
Transfer/sec: 11.43MB
----------------------------------------------------------------------
--- rate-limit 25.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 173.57us 118.35us 13.12ms 97.91%
Req/Sec 5.91k 257.69 6.46k 66.83%
Latency Distribution
50% 162.00us
75% 178.00us
90% 199.00us
99% 362.00us
14122906 requests in 5.00m, 3.41GB read
Requests/sec: 47060.71
Transfer/sec: 11.62MB
----------------------------------------------------------------------
--- rate-limit 10.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 170.85us 112.24us 10.72ms 97.84%
Req/Sec 6.00k 269.81 8.10k 69.46%
Latency Distribution
50% 159.00us
75% 172.00us
90% 194.00us
99% 361.00us
14342642 requests in 5.00m, 3.46GB read
Requests/sec: 47792.96
Transfer/sec: 11.80MB
----------------------------------------------------------------------
--- rate-limit 2.5 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 169.11us 127.52us 16.90ms 98.06%
Req/Sec 6.08k 261.57 6.55k 67.30%
Latency Distribution
50% 158.00us
75% 168.00us
90% 186.00us
99% 367.00us
14527714 requests in 5.00m, 3.50GB read
Requests/sec: 48409.62
Transfer/sec: 11.96MB
----------------------------------------------------------------------
--- rate-limit 0.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 168.09us 108.96us 9.07ms 97.81%
Req/Sec 6.10k 284.22 6.55k 70.65%
Latency Distribution
50% 157.00us
75% 167.00us
90% 184.00us
99% 362.00us
14580762 requests in 5.00m, 3.52GB read
Requests/sec: 48586.42
Transfer/sec: 12.00MB
----------------------------------------------------------------------
--- rate-limit disabled --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 168.57us 118.05us 13.51ms 97.94%
Req/Sec 6.09k 251.47 6.99k 67.33%
Latency Distribution
50% 158.00us
75% 167.00us
90% 184.00us
99% 363.00us
14557824 requests in 5.00m, 3.51GB read
Requests/sec: 48509.96
Transfer/sec: 11.98MB
----------------------------------------------------------------------
--- rate-limit off --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 168.64us 120.15us 14.49ms 98.01%
Req/Sec 6.09k 267.94 6.57k 66.19%
Latency Distribution
50% 158.00us
75% 167.00us
90% 184.00us
99% 361.00us
14551312 requests in 5.00m, 3.51GB read
Requests/sec: 48488.23
Transfer/sec: 11.98MB
----------------------------------------------------------------------

144
addons/otel/test/README-speed-ctx Normal file
View File

@ -0,0 +1,144 @@
--- rate-limit 100.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 270.35us 136.92us 16.02ms 97.98%
Req/Sec 3.77k 217.57 5.33k 67.74%
Latency Distribution
50% 264.00us
75% 287.00us
90% 309.00us
99% 494.00us
9012538 requests in 5.00m, 2.17GB read
Requests/sec: 30031.85
Transfer/sec: 7.42MB
----------------------------------------------------------------------
--- rate-limit 75.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 242.56us 131.47us 9.82ms 94.77%
Req/Sec 4.21k 218.42 5.61k 68.12%
Latency Distribution
50% 246.00us
75% 279.00us
90% 308.00us
99% 464.00us
10050409 requests in 5.00m, 2.42GB read
Requests/sec: 33490.26
Transfer/sec: 8.27MB
----------------------------------------------------------------------
--- rate-limit 50.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 215.92us 130.82us 9.93ms 96.81%
Req/Sec 4.73k 243.84 7.13k 67.13%
Latency Distribution
50% 208.00us
75% 264.00us
90% 300.00us
99% 439.00us
11307386 requests in 5.00m, 2.73GB read
Requests/sec: 37678.82
Transfer/sec: 9.31MB
----------------------------------------------------------------------
--- rate-limit 25.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 192.36us 132.47us 13.75ms 96.79%
Req/Sec 5.33k 260.46 6.17k 66.30%
Latency Distribution
50% 166.00us
75% 227.00us
90% 280.00us
99% 407.00us
12734770 requests in 5.00m, 3.07GB read
Requests/sec: 42448.91
Transfer/sec: 10.48MB
----------------------------------------------------------------------
--- rate-limit 10.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 180.08us 127.35us 13.34ms 97.06%
Req/Sec 5.71k 272.98 6.40k 67.94%
Latency Distribution
50% 161.00us
75% 183.00us
90% 250.00us
99% 386.00us
13641901 requests in 5.00m, 3.29GB read
Requests/sec: 45457.92
Transfer/sec: 11.23MB
----------------------------------------------------------------------
--- rate-limit 2.5 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 171.64us 107.08us 5.69ms 96.57%
Req/Sec 5.97k 289.99 6.55k 68.53%
Latency Distribution
50% 159.00us
75% 171.00us
90% 195.00us
99% 372.00us
14268464 requests in 5.00m, 3.44GB read
Requests/sec: 47545.77
Transfer/sec: 11.74MB
----------------------------------------------------------------------
--- rate-limit 0.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 168.86us 104.53us 5.75ms 97.73%
Req/Sec 6.07k 282.19 6.59k 67.47%
Latency Distribution
50% 158.00us
75% 168.00us
90% 186.00us
99% 361.00us
14498699 requests in 5.00m, 3.50GB read
Requests/sec: 48312.96
Transfer/sec: 11.93MB
----------------------------------------------------------------------
--- rate-limit disabled --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 168.36us 129.52us 17.68ms 98.13%
Req/Sec 6.11k 263.70 6.83k 70.42%
Latency Distribution
50% 157.00us
75% 167.00us
90% 183.00us
99% 363.00us
14590953 requests in 5.00m, 3.52GB read
Requests/sec: 48620.36
Transfer/sec: 12.01MB
----------------------------------------------------------------------
--- rate-limit off --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 168.73us 120.51us 15.03ms 98.02%
Req/Sec 6.09k 270.88 6.55k 68.99%
Latency Distribution
50% 158.00us
75% 167.00us
90% 185.00us
99% 360.00us
14538507 requests in 5.00m, 3.51GB read
Requests/sec: 48445.53
Transfer/sec: 11.97MB
----------------------------------------------------------------------

144
addons/otel/test/README-speed-fe-be Normal file
View File

@ -0,0 +1,144 @@
--- rate-limit 100.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 238.75us 129.45us 15.59ms 98.45%
Req/Sec 4.27k 75.22 5.28k 77.45%
Latency Distribution
50% 228.00us
75% 243.00us
90% 262.00us
99% 410.00us
10206938 requests in 5.00m, 2.46GB read
Requests/sec: 34011.80
Transfer/sec: 8.40MB
----------------------------------------------------------------------
--- rate-limit 75.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 230.08us 201.50us 32.12ms 99.25%
Req/Sec 4.46k 83.43 5.36k 75.61%
Latency Distribution
50% 222.00us
75% 241.00us
90% 261.00us
99% 401.00us
10641998 requests in 5.00m, 2.57GB read
Requests/sec: 35461.59
Transfer/sec: 8.76MB
----------------------------------------------------------------------
--- rate-limit 50.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 222.33us 242.52us 35.13ms 99.43%
Req/Sec 4.62k 99.86 5.78k 79.00%
Latency Distribution
50% 211.00us
75% 237.00us
90% 259.00us
99% 400.00us
11046951 requests in 5.00m, 2.66GB read
Requests/sec: 36810.91
Transfer/sec: 9.09MB
----------------------------------------------------------------------
--- rate-limit 25.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 210.95us 117.20us 10.57ms 97.85%
Req/Sec 4.84k 101.85 6.04k 68.10%
Latency Distribution
50% 198.00us
75% 222.00us
90% 252.00us
99% 394.00us
11551741 requests in 5.00m, 2.79GB read
Requests/sec: 38493.03
Transfer/sec: 9.51MB
----------------------------------------------------------------------
--- rate-limit 10.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 206.15us 209.34us 31.73ms 99.27%
Req/Sec 4.99k 112.62 5.36k 71.21%
Latency Distribution
50% 193.00us
75% 210.00us
90% 237.00us
99% 387.00us
11924489 requests in 5.00m, 2.88GB read
Requests/sec: 39735.09
Transfer/sec: 9.81MB
----------------------------------------------------------------------
--- rate-limit 2.5 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 202.35us 180.56us 27.28ms 99.10%
Req/Sec 5.08k 145.06 8.27k 71.24%
Latency Distribution
50% 191.00us
75% 205.00us
90% 223.00us
99% 374.00us
12131047 requests in 5.00m, 2.93GB read
Requests/sec: 40423.43
Transfer/sec: 9.98MB
----------------------------------------------------------------------
--- rate-limit 0.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 200.88us 223.19us 32.70ms 99.44%
Req/Sec 5.13k 151.03 6.55k 69.46%
Latency Distribution
50% 190.00us
75% 203.00us
90% 218.00us
99% 367.00us
12256706 requests in 5.00m, 2.96GB read
Requests/sec: 40842.16
Transfer/sec: 10.09MB
----------------------------------------------------------------------
--- rate-limit disabled --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 194.91us 222.55us 33.80ms 99.47%
Req/Sec 5.29k 167.52 5.95k 68.32%
Latency Distribution
50% 184.00us
75% 197.00us
90% 214.00us
99% 353.00us
12633928 requests in 5.00m, 3.05GB read
Requests/sec: 42112.54
Transfer/sec: 10.40MB
----------------------------------------------------------------------
--- rate-limit off --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 194.37us 166.76us 28.75ms 99.09%
Req/Sec 5.28k 160.02 5.86k 68.02%
Latency Distribution
50% 184.00us
75% 197.00us
90% 214.00us
99% 355.00us
12622896 requests in 5.00m, 3.04GB read
Requests/sec: 42062.31
Transfer/sec: 10.39MB
----------------------------------------------------------------------

144
addons/otel/test/README-speed-sa Normal file
View File

@ -0,0 +1,144 @@
--- rate-limit 100.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 213.08us 136.99us 17.58ms 98.10%
Req/Sec 4.80k 251.04 5.97k 68.01%
Latency Distribution
50% 203.00us
75% 223.00us
90% 245.00us
99% 405.00us
11464278 requests in 5.00m, 2.77GB read
Requests/sec: 38201.61
Transfer/sec: 9.44MB
----------------------------------------------------------------------
--- rate-limit 75.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 202.18us 121.42us 12.04ms 97.72%
Req/Sec 5.05k 248.48 5.85k 65.69%
Latency Distribution
50% 194.00us
75% 219.00us
90% 245.00us
99% 393.00us
12071015 requests in 5.00m, 2.91GB read
Requests/sec: 40223.31
Transfer/sec: 9.94MB
----------------------------------------------------------------------
--- rate-limit 50.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 190.49us 117.32us 7.33ms 97.62%
Req/Sec 5.37k 265.60 6.98k 65.98%
Latency Distribution
50% 181.00us
75% 208.00us
90% 237.00us
99% 383.00us
12837427 requests in 5.00m, 3.10GB read
Requests/sec: 42777.17
Transfer/sec: 10.57MB
----------------------------------------------------------------------
--- rate-limit 25.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 180.46us 123.40us 13.20ms 97.80%
Req/Sec 5.69k 242.93 7.75k 68.66%
Latency Distribution
50% 165.00us
75% 194.00us
90% 223.00us
99% 375.00us
13595213 requests in 5.00m, 3.28GB read
Requests/sec: 45302.34
Transfer/sec: 11.19MB
----------------------------------------------------------------------
--- rate-limit 10.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 174.69us 129.17us 16.50ms 97.93%
Req/Sec 5.89k 260.40 7.03k 69.57%
Latency Distribution
50% 160.00us
75% 178.00us
90% 210.00us
99% 374.00us
14068388 requests in 5.00m, 3.39GB read
Requests/sec: 46879.07
Transfer/sec: 11.58MB
----------------------------------------------------------------------
--- rate-limit 2.5 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 170.58us 116.89us 11.49ms 97.74%
Req/Sec 6.03k 249.35 6.54k 67.44%
Latency Distribution
50% 158.00us
75% 170.00us
90% 192.00us
99% 375.00us
14402604 requests in 5.00m, 3.47GB read
Requests/sec: 47992.71
Transfer/sec: 11.85MB
----------------------------------------------------------------------
--- rate-limit 0.0 --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 167.96us 114.36us 10.99ms 97.81%
Req/Sec 6.12k 266.16 6.57k 70.70%
Latency Distribution
50% 157.00us
75% 166.00us
90% 183.00us
99% 370.00us
14622790 requests in 5.00m, 3.53GB read
Requests/sec: 48726.40
Transfer/sec: 12.04MB
----------------------------------------------------------------------
--- rate-limit disabled --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 167.74us 114.11us 11.10ms 97.82%
Req/Sec 6.13k 251.71 6.57k 69.59%
Latency Distribution
50% 157.00us
75% 166.00us
90% 182.00us
99% 368.00us
14641307 requests in 5.00m, 3.53GB read
Requests/sec: 48788.18
Transfer/sec: 12.05MB
----------------------------------------------------------------------
--- rate-limit off --------------------------------------------------
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 168.00us 112.83us 11.07ms 97.79%
Req/Sec 6.12k 264.12 7.23k 68.95%
Latency Distribution
50% 157.00us
75% 166.00us
90% 183.00us
99% 369.00us
14613970 requests in 5.00m, 3.53GB read
Requests/sec: 48697.01
Transfer/sec: 12.03MB
----------------------------------------------------------------------

319
addons/otel/test/README-test-speed Normal file
View File

@ -0,0 +1,319 @@
-----------------------------------------
HAProxy OTEL filter speed test guide
Version 1.0
( Last update: 2026-04-09 )
-----------------------------------------
Author : Miroslav Zagorac
Contact : mzagorac at haproxy dot com
SUMMARY
--------
1. Overview
2. Prerequisites
3. Running the test
4. Test parameters
5. Rate-limit levels
6. Test configurations
7. Results
7.1. Standalone (sa)
7.2. Comparison (cmp)
7.3. Context propagation (ctx)
7.4. Frontend / backend (fe-be)
8. Summary
1. Overview
------------
The test-speed.sh script measures the performance impact of the OTEL filter on
HAProxy at various rate-limit settings. For each test configuration, the script
iterates through a series of rate-limit values -- from full tracing (100%) down
to the filter being completely removed -- measuring throughput and latency at
each level.
The script uses template files (haproxy.cfg.in and otel.cfg.in) from each test
directory to generate the actual configuration files. A sed substitution
adjusts the rate-limit value (or disables/removes the filter) before each run.
2. Prerequisites
-----------------
The following tools must be installed and available in PATH:
- thttpd : a lightweight HTTP server used as the backend origin server.
It serves a small static HTML file (index.html) on port 8000.
- wrk : an HTTP benchmarking tool that generates the test load.
See https://github.com/wg/wrk
3. Running the test
--------------------
The test is executed from the test directory. It can be run for all
configurations at once or for a single configuration.
To run all configurations:
% ./test-speed.sh all
This produces result files in the _logs directory:
_logs/README-speed-fe-be
_logs/README-speed-sa
_logs/README-speed-cmp
_logs/README-speed-ctx
To run a single configuration:
% ./test-speed.sh <cfg> [<dir>]
Where <cfg> corresponds to a run-<cfg>.sh script and <dir> is the configuration
directory (defaults to <cfg>). For example:
% ./test-speed.sh sa
% ./test-speed.sh fe-be fe
% ./test-speed.sh cmp
% ./test-speed.sh ctx
4. Test parameters
-------------------
The wrk benchmarking tool is invoked with the following parameters:
-t8 8 threads
-c8 8 concurrent connections
-d300 5-minute test duration (300 seconds)
--latency latency distribution reporting
Each rate-limit level is tested sequentially. Between runs, HAProxy is stopped
via SIGUSR1 and restarted with the next rate-limit configuration. A 10-second
pause separates consecutive runs.
The backend origin server (thttpd) serves a small static HTML page (index.html,
approximately 50 bytes) on port 8000. HAProxy listens on port 10080 and proxies
requests to the origin.
5. Rate-limit levels
---------------------
The script tests nine rate-limit levels in the following order:
100.0 - the filter processes every stream (worst case)
75.0 - the filter processes 75% of streams
50.0 - the filter processes 50% of streams
25.0 - the filter processes 25% of streams
10.0 - the filter processes 10% of streams
2.5 - the filter processes 2.5% of streams
0.0 - the filter is loaded and attached to every stream but never
processes any telemetry (the rate-limit check always fails);
this measures the per-stream attach and detach overhead
disabled - the filter is loaded but disabled via 'option disabled'; it is not
attached to streams at all; this measures the cost of loading and
initializing the filter library without any per-stream work
off - the 'filter opentelemetry' and 'otel-group' directives are
commented out of haproxy.cfg; the filter is not loaded and has zero
presence in the processing path; this is the absolute baseline
In the result tables, the 'overhead' column is the throughput loss relative to
the 'off' baseline, expressed as a percentage:
overhead = (req/s_off - req/s_test) / req/s_off * 100
6. Test configurations
-----------------------
Four OTEL filter configurations are tested. They differ in complexity and in
the features they exercise:
sa - Standalone. Uses all possible HAProxy filter events with spans,
attributes, events, links, baggage, status, metrics and groups.
This is the most comprehensive single-instance configuration and
represents the worst-case scenario.
cmp - Comparison. A simplified configuration made for comparison with
other tracing implementations. It uses a reduced span hierarchy
without context propagation, groups or metrics. This is closer to
a typical production deployment.
ctx - Context propagation. Similar to 'sa' in scope coverage, but spans
are opened using extracted span contexts (inject/extract via HAProxy
variables) as parent references instead of direct span names. This
adds the overhead of context serialization, variable storage and
deserialization on every scope execution.
fe-be - Frontend / backend. Two cascaded HAProxy instances: the frontend
(fe) creates the root trace and injects span context into HTTP
headers; the backend (be) extracts the context and continues the
trace. This configuration measures the combined overhead of two
OTEL filter instances plus the inter-process context propagation
cost.
Note: the rate-limit is varied only on the frontend. The backend
always runs with its default configuration (filter enabled,
hard-errors on). The backend configuration is only modified for
the 'disabled' and 'off' levels.
7. Results
-----------
The tables below summarize the benchmarking results. The 'req/s' column shows
the sustained request rate reported by wrk, 'avg latency' is the average
response time, and 'overhead' is the throughput loss relative to the 'off'
baseline.
7.1. Standalone (sa)
---------------------
---------------------------------------------------------------
rate-limit req/s avg latency overhead
---------------------------------------------------------------
100.0% 38,202 213.08 us 21.6%
75.0% 40,223 202.18 us 17.4%
50.0% 42,777 190.49 us 12.2%
25.0% 45,302 180.46 us 7.0%
10.0% 46,879 174.69 us 3.7%
2.5% 47,993 170.58 us 1.4%
0.0% 48,726 167.96 us ~0
disabled 48,788 167.74 us ~0
off 48,697 168.00 us baseline
---------------------------------------------------------------
With all possible events active, the sa configuration at 100% rate-limit shows
a 22% throughput reduction. At 10% rate-limit the overhead drops to 3.7%, and
at 2.5% it is barely measurable at 1.4%. The 'disabled' and '0.0' levels show
no measurable overhead above the baseline.
7.2. Comparison (cmp)
----------------------
---------------------------------------------------------------
rate-limit req/s avg latency overhead
---------------------------------------------------------------
100.0% 44,780 182.58 us 7.6%
75.0% 45,362 180.10 us 6.4%
50.0% 46,279 176.59 us 4.6%
25.0% 47,061 173.57 us 2.9%
10.0% 47,793 170.85 us 1.4%
2.5% 48,410 169.11 us 0.2%
0.0% 48,586 168.09 us ~0
disabled 48,510 168.57 us ~0
off 48,488 168.64 us baseline
---------------------------------------------------------------
The simplified cmp configuration shows significantly lower overhead than sa at
every rate-limit level. At 100% rate-limit the overhead is 7.6%, at 10% it is
1.4%, and below 2.5% it becomes indistinguishable from the baseline.
7.3. Context propagation (ctx)
-------------------------------
---------------------------------------------------------------
rate-limit req/s avg latency overhead
---------------------------------------------------------------
100.0% 30,032 270.35 us 38.0%
75.0% 33,490 242.56 us 30.9%
50.0% 37,679 215.92 us 22.2%
25.0% 42,449 192.36 us 12.4%
10.0% 45,458 180.08 us 6.2%
2.5% 47,546 171.64 us 1.9%
0.0% 48,313 168.86 us 0.3%
disabled 48,620 168.36 us ~0
off 48,446 168.73 us baseline
---------------------------------------------------------------
The ctx configuration is the most expensive due to the inject/extract cycle on
every scope execution. At 100% rate-limit the overhead reaches 38%. However,
the cost scales linearly with the rate-limit: at 10% the overhead is 6.2%, and
at 2.5% it drops to 1.9%. The filter attachment overhead (0.0% vs off) is
negligible at 0.3%.
7.4. Frontend / backend (fe-be)
--------------------------------
---------------------------------------------------------------
rate-limit req/s avg latency overhead
---------------------------------------------------------------
100.0% 34,012 238.75 us 19.1%
75.0% 35,462 230.08 us 15.7%
50.0% 36,811 222.33 us 12.5%
25.0% 38,493 210.95 us 8.5%
10.0% 39,735 206.15 us 5.5%
2.5% 40,423 202.35 us 3.9%
0.0% 40,842 200.88 us 2.9%
disabled 42,113 194.91 us ~0
off 42,062 194.37 us baseline
---------------------------------------------------------------
The fe-be configuration involves two HAProxy instances in series, so the
absolute baseline (off) is already lower at 42,062 req/s due to the extra
network hop. The rate-limit is varied only on the frontend; the backend
always has the filter loaded with hard-errors enabled.
This explains the 2.9% overhead at rate-limit 0.0: even though the frontend
never traces, the backend filter still attaches to every stream, attempts to
extract context from the HTTP headers, fails (because the frontend did not
inject any context), and the hard-errors option stops further processing.
This per-stream attach/extract/error cycle accounts for the residual cost.
When both instances have the filter disabled (disabled level), the overhead
is within measurement noise, consistent with the single-instance
configurations.
8. Summary
-----------
The table below shows the overhead for each configuration at selected rate-limit
levels:
---------------------------------------------------
rate-limit sa cmp ctx fe-be
---------------------------------------------------
100.0% 21.6% 7.6% 38.0% 19.1%
25.0% 7.0% 2.9% 12.4% 8.5%
10.0% 3.7% 1.4% 6.2% 5.5%
2.5% 1.4% 0.2% 1.9% 3.9%
---------------------------------------------------
Key observations:
- The overhead scales approximately linearly with the rate-limit value.
Reducing the rate-limit from 100% to 10% eliminates the vast majority
of the cost in all configurations.
- The cmp configuration, which uses a reduced span hierarchy typical of
production deployments, adds only 1.4% overhead at a 10% rate-limit.
- The sa configuration, which exercises all possible events, stays at about
7% overhead at a 25% rate-limit and below 4% at 10%.
- The ctx configuration is the most expensive due to the inject/extract
context propagation on every scope. It is designed as a stress test for
the propagation mechanism rather than a practical production template.
- The fe-be configuration carries a higher fixed cost because two HAProxy
instances are involved and the backend filter processes context extraction
regardless of the frontend rate-limit setting.
- Loading the filter but disabling it via 'option disabled' adds no measurable
overhead in any configuration.
- The filter attachment cost without any telemetry processing (rate-limit 0.0)
is 0.3% or less for single-instance configurations (sa, cmp, ctx).
- In typical production use with a rate-limit of 10% or less, the performance
impact of the OTEL filter should be negligible for single-instance deployments.