mirrors/haproxy
1
0
Fork 0
mirror of https://git.haproxy.org/git/haproxy.git/ synced 2025-08-10 00:57:02 +02:00
Code Issues Projects Releases Wiki Activity
14711bdc9a
haproxy/src
History
Willy Tarreau d114f4a68f MEDIUM: checks: spread the checks load over random threads 
The CPU usage pattern was found to be high (5%) on a machine with
48 threads and only 100 servers checked every second That was
supposed to be only 100 connections per second, which should be very
cheap. It was figured that due to the check tasks unbinding from any
thread when going back to sleep, they're queued into the shared queue.

Not only this requires to manipulate the global queue lock, but in
addition it means that all threads have to check the global queue
before going to sleep (hence take a lock again) to figure how long
to sleep, and that they would all sleep only for the shortest amount
of time to the next check, one would pick it and all other ones would
go down to sleep waiting for the next check.

That's perfectly visible in time-to-first-byte measurements. A quick
test consisting in retrieving the stats page in CSV over a 48-thread
process checking 200 servers every 2 seconds shows the following tail:

  percentile   ttfb(ms)
  99.98        2.43
  99.985       5.72
  99.99       32.96
  99.995     82.176
  99.996     82.944
  99.9965    83.328
  99.997      83.84
  99.9975    84.288
  99.998      85.12
  99.9985    86.592
  99.999         88
  99.9995    89.728
  99.9999   100.352

One solution could consist in forcefully binding checks to threads at
boot time, but that's annoying, will cause trouble for dynamic servers
and may cause some skew in the load depending on some server patterns.

Instead here we take a different approach. A check remains bound to its
thread for as long as possible, but upon every wakeup, the thread's load
is compared with another random thread's load. If it's found that that
other thread's load is less than half of the current one's, the task is
bounced to that thread. In order to prevent that new thread from doing
the same, we set a flag "CHK_ST_SLEEPING" that indicates that it just
woke up and we're bouncing the task only on this condition.

Tests have shown that the initial load was very unfair before, with a few
checks threads having a load of 15-20 and the vast majority having zero.
With this modification, after two "inter" delays, the load is either zero
or one everywhere when checks start. The same test shows a CPU usage that
significantly drops, between 0.5 and 1%. The same latency tail measurement
is much better, roughly 10 times smaller:

  percentile   ttfb(ms)
  99.98        1.647
  99.985       1.773
  99.99        4.912
  99.995        8.76
  99.996        8.88
  99.9965      8.944
  99.997       9.016
  99.9975      9.104
  99.998       9.224
  99.9985      9.416
  99.999         9.8
  99.9995      10.04
  99.9999     10.432

In fact one difference here is that many threads work while in the past
they were waking up and going down to sleep after having perturbated the
shared lock. Thus it is anticipated that this will scale way smoother
than before. Under strace it's clearly visible that all threads are
sleeping for the time it takes to relaunch a check, there's no more
thundering herd wakeups.

However it is also possible that in some rare cases such as very short
check intervals smaller than a scheduler's timeslice (such as 4ms),
some users might have benefited from the work being concentrated on
less threads and would instead observe a small increase of apparent
CPU usage due to more total threads waking up even if that's for less
work each and less total work. That's visible with 200 servers at 4ms
where show activity shows that a few threads were overloaded and others
doing nothing. It's not a problem, though as in practice checks are not
supposed to eat much CPU and to wake up fast enough to represent a
significant load anyway, and the main issue they could have been
causing (aside the global lock) is an increase last-percentile latency.
2022-10-12 21:49:30 +02:00
..
acl.c MINOR: acl: alphanumerically sort the ACL dump 2022-03-30 11:49:59 +02:00
action.c BUG/MINOR: rules: Fix check_capture() function to use the right rule arguments 2022-04-25 15:28:21 +02:00
activity.c MINOR: activity/cli: support sorting task profiling by total CPU time 2022-09-08 16:38:10 +02:00
applet.c BUG/MINOR: applet: make the call_rate only count the no-progress calls 2022-08-23 20:19:11 +02:00
arg.c MINOR: arg: Be able to forbid unresolved args when building an argument list 2021-09-30 16:37:05 +02:00
auth.c
backend.c BUG/MINOR: backend: only enforce turn-around state when not redispatching 2022-10-03 15:04:55 +02:00
base64.c
cache.c CLEANUP: applet: rename appctx_cs() to appctx_sc() 2022-05-27 19:33:35 +02:00
calltrace.c BUILD: calltrace: fix wrong include when building with TRACE=1 2022-04-19 08:23:30 +02:00
cbuf.c CLEANUP: pool/tree-wide: remove suffix "_pool" from certain pool names 2022-06-23 11:49:09 +02:00
cfgcond.c MINOR: ssl: Add ssllib_name_startswith precondition 2021-10-13 11:28:08 +02:00
cfgdiag.c
cfgparse-global.c BUG/MINOR: config: insufficient syntax check of the global "maxconn" value 2022-10-03 14:30:08 +02:00
cfgparse-listen.c MEDIUM: config: remove deprecated "bind-process" directives from frontends 2022-07-15 19:41:26 +02:00
cfgparse-quic.c CLEANUP: Reapply strcmp.cocci 2022-10-10 15:49:09 +02:00
cfgparse-ssl.c MINOR: listener/ssl: set the SSL xprt layer only once the whole config is known 2022-05-20 18:41:55 +02:00
cfgparse-tcp.c CLEANUP: tree-wide: remove 25 occurrences of unneeded fcntl.h 2022-04-26 10:59:48 +02:00
cfgparse-unix.c CLEANUP: tree-wide: remove 25 occurrences of unneeded fcntl.h 2022-04-26 10:59:48 +02:00
cfgparse.c CLEANUP: quic: fix headers 2022-10-03 16:25:17 +02:00
channel.c CLEANUP: conn_stream: tree-wide rename to stconn (stream connector) 2022-05-27 19:33:34 +02:00
check.c MEDIUM: checks: spread the checks load over random threads 2022-10-12 21:49:30 +02:00
chunk.c MINOR: chunk: inline alloc_trash_chunk() 2022-08-17 10:45:22 +02:00
cli.c MINOR: cli: correct commentary and replace 'set global-key' name 2022-09-29 10:53:15 +02:00
clock.c MINOR: clock: do not update the global date too often 2022-09-21 09:06:28 +02:00
compression.c CLEANUP: compression: move the default setting of maxzlibmem to defaults 2022-04-25 19:42:43 +02:00
connection.c BUG/MAJOR: conn-idle: fix hash indexing issues on idle conns 2022-10-03 12:06:36 +02:00
cpuset.c MEDIUM: cpu-map: replace the process number with the thread group number 2022-07-15 19:43:10 +02:00
debug.c MINOR: anon: store the anonymizing key in the CLI's appctx 2022-09-17 11:27:09 +02:00
dgram.c
dict.c
dns.c BUG/MEDIUM: dns: Properly initialize new DNS session 2022-08-03 10:30:07 +02:00
dynbuf.c REORG: thread/sched: move the last dynamic thread_info to thread_ctx 2021-10-08 17:22:26 +02:00
eb32sctree.c
eb32tree.c
eb64tree.c
ebimtree.c
ebistree.c
ebmbtree.c
ebpttree.c
ebsttree.c
ebtree.c
errors.c CLEANUP: ring: pass the ring watch flags to ring_attach_cli(), not in ctx.cli 2022-05-06 18:13:36 +02:00
ev_epoll.c MINOR: pollers: only update the local date during busy polling 2022-09-21 09:06:28 +02:00
ev_evports.c MINOR: pollers: only update the local date during busy polling 2022-09-21 09:06:28 +02:00
ev_kqueue.c MINOR: pollers: only update the local date during busy polling 2022-09-21 09:06:28 +02:00
ev_poll.c BUG/MINOR: signals/poller: ensure wakeup from signals 2022-09-09 11:15:22 +02:00
ev_select.c MEDIUM: fd/poller: turn update_mask to group-local IDs 2022-07-15 20:16:30 +02:00
extcheck.c MINOR: checks: use the lighter PRNG for spread checks 2022-10-12 21:49:30 +02:00
fcgi-app.c MEDIUM: http-ana: Add a proxy option to restrict chars in request header names 2022-05-16 16:00:26 +02:00
fcgi.c CLEANUP: Reapply ist.cocci with --include-headers-for-types --recursive-includes 2022-03-21 08:30:47 +01:00
fd.c MINOR: fd: add a new function to only raise RLIMIT_NOFILE 2022-10-04 08:38:47 +02:00
filters.c CLEANUP: tree-wide: Remove any ref to stream-interfaces 2022-04-13 15:10:16 +02:00
fix.c
flt_bwlim.c CLEANUP: Reapply strcmp.cocci 2022-10-10 15:49:09 +02:00
flt_http_comp.c BUG/MEDIUM: compression: Don't forget to update htx_sl and http_msg flags 2022-04-15 16:22:33 +02:00
flt_spoe.c BUG/MEDIUM: spoe: Properly update streams waiting for a ACK in async mode 2022-08-29 09:57:33 +02:00
flt_trace.c CLEANUP: Apply ist.cocci 2021-11-08 12:08:26 +01:00
freq_ctr.c MINOR: freq_ctr: use the thread's local time whenever possible 2022-10-12 14:19:05 +02:00
frontend.c REORG: rename cs_utils.h to sc_strm.h 2022-05-27 19:33:35 +02:00
h1_htx.c BUG/MEDIUM: http: Properly reject non-HTTP/1.x protocols 2022-05-24 15:38:05 +02:00
h1.c BUILD: h1: silence an initiialized warning with gcc-4.7 and -Os 2022-10-04 08:02:03 +02:00
h2.c BUG/MINOR: h2: properly set the direction flag on HTX response 2022-09-02 11:19:07 +02:00
h3_stats.c MINOR: h3: Add a statistics module for h3 2022-05-30 09:59:26 +02:00
h3.c CLEANUP: quic: create a dedicated quic_conn module 2022-10-03 16:25:17 +02:00
haproxy.c MINOR: init: do not try to shrink existing RLIMIT_NOFIlE 2022-10-04 08:38:47 +02:00
hash.c
hlua_fcn.c MEDIUM: stick-table: switch the table lock to rwlock 2022-10-12 14:19:05 +02:00
hlua.c MINOR: hlua: removing ambiguous lua_pushvalue with 0 index 2022-10-12 09:22:05 +02:00
hpack-dec.c BUILD/DEBUG: hpack: use unsigned int in printf format in debug code 2022-04-12 08:40:38 +02:00
hpack-enc.c
hpack-huff.c OPTIM: hpack-huff: reduce the cache footprint of the huffman decoder 2022-09-20 07:41:58 +02:00
hpack-tbl.c BUILD/DEBUG: hpack-tbl: fix format string in standalone debug code 2022-04-12 08:30:08 +02:00
hq_interop.c MINOR: mux-quic: refactor snd_buf 2022-09-20 15:35:29 +02:00
http_acl.c
http_act.c BUG/MINOR: http-act: initialize http fmt head earlier 2022-09-02 19:24:12 +02:00
http_ana.c BUG/MEDIUM: http-ana: fix crash or wrong header deletion by http-restrict-req-hdr-names 2022-08-17 15:52:17 +02:00
http_client.c MINOR: httpclient: enabled the use of SNI presets 2022-09-19 14:39:28 +02:00
http_conv.c BUG/MEDIUM: http-conv: Fix url_enc() to not crush const samples 2022-04-08 10:12:59 +02:00
http_fetch.c BUILD: http_fetch: silence an uninitiialized warning with gcc-4/5/6 at -Os 2022-10-04 09:18:34 +02:00
http_htx.c REORG: h2: extract cookies concat function in http_htx 2022-08-18 16:13:33 +02:00
http_rules.c BUILD: http: remove the two unused constructors in rules and ana 2022-04-25 19:26:26 +02:00
http.c MINOR: http: Add function to detect default port 2022-07-06 17:54:03 +02:00
htx.c BUG/MEDIUM: htx: Fix a possible null derefs in htx_xfer_blks() 2022-02-28 17:16:55 +01:00
init.c MINOR: init: add the pre-check callback 2022-04-22 15:45:47 +02:00
jwt.c BUG/MINOR: jwt: Memory leak if same key is used in multiple jwt_verify calls 2022-02-15 20:08:20 +01:00
lb_chash.c
lb_fas.c
lb_fwlc.c BUG/MEDIUM: leastconn: fix rare possibility of divide by zero 2021-09-22 07:24:02 +02:00
lb_fwrr.c
lb_map.c
listener.c BUG/MINOR: listener: null pointer dereference suspected by coverity 2022-09-12 10:12:18 +02:00
log.c BUG/MINOR: log: improper behavior when escaping log data 2022-09-20 16:25:30 +02:00
lru.c BUILD/DEBUG: lru: fix printf format in debug code 2022-04-12 08:19:33 +02:00
mailers.c BUG/MEDIUM: mailers: Set the object type for check attached to an email alert 2022-06-08 15:28:38 +02:00
map.c CLEANUP: applet: rename appctx_cs() to appctx_sc() 2022-05-27 19:33:35 +02:00
mjson.c
mqtt.c MEDIUM: mqtt: support mqtt_is_valid and mqtt_field_value converters for MQTTv3.1 2022-03-22 09:25:52 +01:00
mux_fcgi.c REORG: mux-fcgi: Extract flags and enums into mux_fcgi-t.h 2022-10-12 17:10:37 +02:00
mux_h1.c BUG/MEDIUM: mux-h1: Handle abort with an incomplete message during parsing 2022-10-12 17:10:44 +02:00
mux_h2.c BUG/MAJOR: conn-idle: fix hash indexing issues on idle conns 2022-10-03 12:06:36 +02:00
mux_pt.c CLEANUP: mux-pt: rename the "endp" field to "sd" 2022-05-27 19:33:35 +02:00
mux_quic.c MINOR: mux-quic: check quic-conn return code on Tx 2022-10-05 11:08:32 +02:00
mworker-prog.c BUILD: mworker: include tools.h for platforms without unsetenv() 2022-01-28 19:04:02 +01:00
mworker.c MINOR: mworker: store and shows loading status 2022-09-24 15:44:42 +02:00
namespace.c
ncbuf.c CLEANUP: assorted typo fixes in the code and comments 2022-08-06 17:12:51 +02:00
pattern.c BUG/MEDIUM: pattern: only visit equivalent nodes when skipping versions 2022-08-01 11:59:46 +02:00
payload.c CLEANUP: check: rename all occurrences of stconn "cs" to "sc" 2022-05-27 19:33:35 +02:00
peers.c MEDIUM: stick-table: switch the table lock to rwlock 2022-10-12 14:19:05 +02:00
pipe.c
pool.c MINOR: pools/memprof: store and report the pool's name in each bin 2022-08-17 10:34:00 +02:00
proto_quic.c BUG/MINOR: quic: set IP_PKTINFO socket option for QUIC receivers only 2022-10-11 16:46:04 +02:00
proto_sockpair.c MINOR: sockpair: move send_fd_uxst() error message in caller 2022-07-25 16:11:11 +02:00
proto_tcp.c MINOR: protocol: replace ctrl_type with xprt_type and clarify it 2022-05-20 18:39:43 +02:00
proto_udp.c MINOR: protocol: replace ctrl_type with xprt_type and clarify it 2022-05-20 18:39:43 +02:00
proto_uxdg.c MINOR: protocol: replace ctrl_type with xprt_type and clarify it 2022-05-20 18:39:43 +02:00
proto_uxst.c MINOR: protocol: replace ctrl_type with xprt_type and clarify it 2022-05-20 18:39:43 +02:00
protocol.c MINOR: listener: small API change 2022-09-09 17:23:01 +02:00
proxy.c BUG/MINOR: server: make sure "show servers state" hides private bits 2022-10-12 21:45:39 +02:00
qmux_http.c BUG/MEDIUM: mux-quic: properly trim HTX buffer on snd_buf reset 2022-09-20 15:35:33 +02:00
qmux_trace.c CLEANUP: quic: fix headers 2022-10-03 16:25:17 +02:00
qpack-dec.c MINOR: qpack: report error on enc/dec stream close 2022-08-17 11:04:53 +02:00
qpack-enc.c BUG/MINOR: qpack: support bigger prefix-integer encoding 2022-05-30 14:30:05 +02:00
qpack-tbl.c CLEANUP: tree-wide: remove a few rare non-ASCII chars 2022-03-04 08:58:32 +01:00
queue.c BUG/MEDIUM: queue/threads: limit the number of entries dequeued at once 2022-07-30 10:00:59 +02:00
quic_cc_cubic.c BUG/MEDIUM: quic: Floating point exception in cubic_root() 2022-08-03 14:27:20 +02:00
quic_cc_newreno.c CLEANUP: quic: create a dedicated quic_conn module 2022-10-03 16:25:17 +02:00
quic_cc.c CLEANUP: quic: fix headers 2022-10-03 16:25:17 +02:00
quic_conn.c MEDIUM: quic: retrieve frontend destination address 2022-10-10 11:48:27 +02:00
quic_frame.c CLEANUP: quic: create a dedicated quic_conn module 2022-10-03 16:25:17 +02:00
quic_loss.c CLEANUP: quic: create a dedicated quic_conn module 2022-10-03 16:25:17 +02:00
quic_sock.c MEDIUM: quic: retrieve frontend destination address 2022-10-10 11:48:27 +02:00
quic_stats.c CLEANUP: quic: fix headers 2022-10-03 16:25:17 +02:00
quic_stream.c CLEANUP: quic: create a dedicated quic_conn module 2022-10-03 16:25:17 +02:00
quic_tls.c CLEANUP: quic: create a dedicated quic_conn module 2022-10-03 16:25:17 +02:00
quic_tp.c CLEANUP: quic: create a dedicated quic_conn module 2022-10-03 16:25:17 +02:00
raw_sock.c MINOR: raw-sock: don't try to send if an error was already reported 2022-08-29 18:45:27 +02:00
regex.c BUG/MINOR: regex: Properly handle PCRE2 lib compiled without JIT support 2022-09-01 19:34:46 +02:00
resolvers.c BUG/MEDIUM: resolvers: Remove aborted resolutions from query_ids tree 2022-09-27 11:18:17 +02:00
ring.c BUG/MINOR: ring: fix the size check in ring_make_from_area() 2022-09-27 14:31:37 +02:00
sample.c MINOR: tasks: do not keep cpu and latency times in struct task 2022-09-08 14:19:15 +02:00
server_state.c BUILD: server-state: avoid using not-so-portable isblank() 2022-01-28 19:04:02 +01:00
server.c BUG/MAJOR: conn-idle: fix hash indexing issues on idle conns 2022-10-03 12:06:36 +02:00
session.c BUG/MINOR: session: fix theoretical risk of memleak in session_accept_fd() 2022-03-11 07:25:11 +01:00
sha1.c
shctx.c MINOR: shctx: add a few BUG_ON() for consistency checks 2021-11-19 19:25:13 +01:00
signal.c BUG/MINOR: signals/poller: ensure wakeup from signals 2022-09-09 11:15:22 +02:00
sink.c BUG/MEDIUM: sink: bad init sequence on tcp sink from a ring. 2022-09-13 17:03:30 +02:00
slz.c IMPORT: slz: use the correct CRC32 instruction when running in 32-bit mode 2021-12-06 09:14:20 +01:00
sock_inet.c BUG/MINOR: quic: set IP_PKTINFO socket option for QUIC receivers only 2022-10-11 16:46:04 +02:00
sock_unix.c MEDIUM: fd: add the tgid to the fd and pass it to fd_insert() 2022-07-15 19:58:06 +02:00
sock.c MINOR: sockpair: move send_fd_uxst() error message in caller 2022-07-25 16:11:11 +02:00
ssl_ckch.c BUG/MINOR: ssl: leak of ckch_inst_link in ckch_inst_free() v2 2022-08-31 15:24:01 +02:00
ssl_crtlist.c BUG/MEDIUM: ssl/cli: crash when crt inserted into a crt-list 2022-06-20 17:27:49 +02:00
ssl_sample.c CLEANUP: check: rename all occurrences of stconn "cs" to "sc" 2022-05-27 19:33:35 +02:00
ssl_sock.c CLEANUP: quic: create a dedicated quic_conn module 2022-10-03 16:25:17 +02:00
ssl_utils.c CLEANUP: Add missing header to ssl_utils.c 2022-05-17 11:40:33 +02:00
stats.c BUG/MINOR: stats: fixing stat shows disabled frontend status as 'OPEN' 2022-09-09 17:24:22 +02:00
stconn.c BUG/MEDIUM: stconn: Reset SE descriptor when we fail to create a stream 2022-09-27 11:18:11 +02:00
stick_table.c BUG/MINOR: stick-table: fix build with DEBUG_THREAD 2022-10-12 16:54:59 +02:00
stream.c BUG/MAJOR: stick-tables: do not try to index a server name for applets 2022-10-12 14:19:05 +02:00
task.c MEDIUM: tasks/activity: combine the called function with the caller 2022-09-08 16:21:22 +02:00
tcp_act.c REORG: rename cs_utils.h to sc_strm.h 2022-05-27 19:33:35 +02:00
tcp_rules.c BUG/MINOR: tcp-rules: Make action call final on read error and delay expiration 2022-06-13 08:04:10 +02:00
tcp_sample.c MINOR: Revert part of clarifying samples support per os commit 2022-09-03 06:11:08 +02:00
tcpcheck.c BUG/MINOR: checks: update pgsql regex on auth packet 2022-10-03 15:31:22 +02:00
thread.c MINOR: threads: report the number of thread groups in build options 2022-08-06 16:45:26 +02:00
time.c REORG: time: move time-keeping code and variables to clock.c 2021-10-08 17:22:26 +02:00
tools.c BUG/MEDIUM: config: count line arguments without dereferencing the output 2022-10-03 09:24:26 +02:00
trace.c BUG/MINOR: trace: Test server existence for health-checks to get proxy 2022-06-08 15:28:38 +02:00
uri_auth.c
uri_normalizer.c
vars.c CLEANUP: vars: move the per-process variables initialization to vars.c 2022-02-23 17:11:33 +01:00
version.c
wdt.c MINOR: wdt: do not rely on threads_to_dump anymore 2022-07-01 19:26:35 +02:00
xprt_handshake.c CLEANUP: pool/tree-wide: remove suffix "_pool" from certain pool names 2022-06-23 11:49:09 +02:00
xprt_quic.c MINOR: quic: limit usage of ssl_sock_ctx in favor of quic_conn 2022-10-05 11:08:32 +02:00