mirrors/haproxy
1
0
Fork 0
mirror of https://git.haproxy.org/git/haproxy.git/ synced 2025-08-11 17:46:57 +02:00
Code Issues Projects Releases Wiki Activity
14,354 Commits 18 Branches 394 Tags 183 MiB
fa68d2641b
Commit Graph

1519 Commits

Author SHA1 Message Date
Emeric Brun
94aab06e24 MEDIUM: log: support tcp or stream addresses on log lines. 
An explicit stream address prefix such as "tcp6@" "tcp4@"
"stream+ipv6@" "stream+ipv4@" or "stream+unix@" will
allocate an implicit ring buffer with a forward server
targeting the given address.

This is usefull to simply send logs to a log server in tcp
and It doesn't need to declare a ring section in configuration.
 2021-04-07 09:18:34 +02:00
Emeric Brun
ce325c4360 MINOR: server/bind: add support of new prefixes for addresses. 
Since the internal function str2sa_range is used to addresses
for different objects ('server', 'bind' but also 'log' or
'nameserver') we notice that some combinations are missing.

"ip@" is introduced to authorize the prefix "dgram+ip@" or
"stream+ip@" which dectects automatically IP version but
specify dgram or stream.

"tcp@" was introduced and is an alias for "stream+ip@".
"tcp6" and "tcp4" are now aliases for "stream+ipv6@" and
"stream+ipv4@".

"uxst@" and "uxdg@" are now aliases for "stream+unix@" and
"dgram+unix@".

This patch also adds a complete section in documentation to
describe adresses and their prefixes.
 2021-04-07 09:18:32 +02:00
Christopher Faulet
8043e831d1 MINOR: acl: Add HTTP_2.0 predefined macro 
HTTP_2.0 predefined macro returns true for HTTP/2 requests. HTTP/2 doen't
convey a version information, so this macro may seem a bit strange. But for
compatiblity reasons, internally, the "HTTP/2.0" version is set. Thus, it is
handy to rely on it to differenciate HTTP/1 and HTTP/2 requests.
 2021-04-06 07:31:51 +02:00
Christopher Faulet
779184e35e MINOR: No longer rely on deprecated sample fetches for predefined ACLs 
Some predefined ACLs were still based on deprecated sample fetches, like
req_proto_http or req_ver. Now, they use non-deprecated sample fetches. In
addition, the usage lines in the configuration manual have been updated to
be more explicit.
 2021-04-05 17:21:05 +02:00
Willy Tarreau
59fa1d1414 [RELEASE] Released version 2.4-dev15 
Released version 2.4-dev15 with the following main changes :
    - BUG/MINOR: payload: Wait for more data if buffer is empty in payload/payload_lv
    - BUG/MINOR: stats: Apply proper styles in HTML status page.
    - BUG/MEDIUM: time: make sure to always initialize the global tick
    - BUG/MINOR: tcp: fix silent-drop workaround for IPv6
    - BUILD: tcp: use IPPROTO_IPV6 instead of SOL_IPV6 on FreeBSD/MacOS
    - CLEANUP: socket: replace SOL_IP/IPV6/TCP with IPPROTO_IP/IPV6/TCP
    - BUG/MINOR: http_fetch: make hdr_ip() resistant to empty fields
    - BUG/MINOR: mux-h2: Don't emit log twice if an error occurred on the preface
    - MINOR: stream: Don't trigger errors on destructive HTTP upgrades
    - MINOR: frontend: Create HTTP txn for HTX streams
    - MINOR: stream: Be sure to set HTTP analysers when creating an HTX stream
    - BUG/MINOR: stream: Properly handle TCP>H1>H2 upgrades in http_wait_for_request
    - BUG/MINOR: config: Add warning for http-after-response rules in TCP mode
    - MINOR: muxes: Add a flag to notify a mux does not support any upgrade
    - MINOR: mux-h1: Don't perform implicit HTTP/2 upgrade if not supported by mux
    - MINOR: mux-pt: Don't perform implicit HTTP upgrade if not supported by mux
    - MEDIUM: mux-h1: Expose h1 in the list of supported mux protocols
    - MEDIUM: mux-pt: Expose passthrough in the list of supported mux protocols
    - MINOR: muxes: Show muxes flags when the mux list is displayed
    - DOC: config: Improve documentation about proto/check-proto keywords
    - MINOR: stream: Use stream type instead of proxy mode when appropriate
    - MINOR: filters/http-ana: Decide to filter HTTP headers in HTTP analysers
    - MINOR: http-ana: Simplify creation/destruction of HTTP transactions
    - MINOR: stream: Handle stream HTTP upgrade in a dedicated function
    - MEDIUM: Add tcp-request switch-mode action to perform HTTP upgrade
    - MINOR: config/proxy: Don't warn for HTTP rules in TCP if 'switch-mode http' set
    - MINOR: config/proxy: Warn if a TCP proxy without backend is upgradable to HTTP
    - DOC: config: Add documentation about TCP to HTTP upgrades
    - REGTESTS: Add script to tests TCP to HTTP upgrades
    - BUG/MINOR: payload/htx: Ingore L6 sample fetches for HTX streams/checks
    - MINOR: htx: Make internal.strm.is_htx an internal sample fetch
    - MINOR: action: Use a generic function to check validity of an action rule list
    - MINOR: payload/config: Warn if a L6 sample fetch is used from an HTTP proxy
    - MEDIUM: http-rules: Add wait-for-body action on request and response side
    - REGTESTS: Add script to tests the wait-for-body HTTP action
    - BUG/MINOR: http-fetch: Fix test on message state to capture the version
    - CLEANUP: vars: always pre-initialize smp in vars_parse_cli_get_var()
    - MINOR: global: define diagnostic mode of execution
    - MINOR: cfgparse: diag for multiple nbthread statements
    - MINOR: server: diag for 0 weight server
    - MINOR: diag: create cfgdiag module
    - MINOR: diag: diag if servers use the same cookie value
    - MINOR: config: diag if global section after non-global
    - TESTS: slightly reorganize the code in the tests/ directory
    - TESTS: move tests/*.cfg to tests/config
    - REGTESTS: ssl: "set ssl cert" and multi-certificates bundle
    - REGTESTS: ssl: mark set_ssl_cert_bundle.vtc as broken
    - CONTRIB: halog: fix issue with array of type char
    - CONTRIB: tcploop: add a shutr command
    - CONTRIB: debug: add the show-fd-to-flags script
    - CONTRIB: debug: split poll from flags
    - CONTRIB: move some dev-specific tools to dev/
    - BUILD: makefile: always build the flags utility
    - DEV: flags: replace the unneeded makefile with a README
    - BUILD: makefile: integrate the hpack tools
    - CONTRIB: merge ip6range with iprange
    - CONTRIB: move some admin-related sub-projects to admin/
    - CONTRIB: move halog to admin/
    - ADMIN: halog: automatically enable USE_MEMCHR on the right glibc version
    - BUILD: makefile: build halog with the correct flags
    - BUILD: makefile: add a "USE_PROMEX" variable to ease building prometheus-exporter
    - CONTRIB: move prometheus-exporter to addons/promex
    - DOC: add a few words about USE_* and the addons directory
    - CONTRIB: move 51Degrees to addons/51degrees
    - CONTRIB: move src/da.c and contrib/deviceatlas to addons/deviceatlas
    - CONTRIB: move src/wurfl.c and contrib/wurfl to addons/wurfl
    - CONTRIB: move contrib/opentracing to addons/ot
    - BUG/MINOR: opentracing: initialization after establishing daemon mode
    - DOC: clarify that compression works for HTTP/2
 2021-04-02 19:16:32 +02:00
Julien Pivotto
ff80c82877 DOC: clarify that compression works for HTTP/2 
This patch clarifies that compression also works with HTTP/2. I have
picked the wording "HTTP/1.1 or above" because it is already used
elsewhere in the documentation.

I have tested that compression indeed works in HTTP/2.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2021-04-02 19:15:25 +02:00
Willy Tarreau
a63d1a0863 CONTRIB: move contrib/opentracing to addons/ot 
This one is the last optional module to build with haproxy, so let's move
it to addons/. It was renamed to "ot" as it was the only one whose USE_*
option did not match the directory name, now this is consistent.

Few changes were required, only the Makefile, and doc were adjusted, as
the directory was already self-contained and relocatable.
 2021-04-02 17:48:42 +02:00
Christopher Faulet
021a8e4d7b MEDIUM: http-rules: Add wait-for-body action on request and response side 
Historically, an option was added to wait for the request payload (option
http-buffer-request). This option has 2 drawbacks. First, it is an ON/OFF
option for the whole proxy. It cannot be enabled on demand depending on the
message. Then, as its name suggests, it only works on the request side. The
only option to wait for the response payload was to write a dedicated
filter. While it is an acceptable solution for complex applications, it is a
bit overkill to simply match strings in the body.

To make everyone happy, this patch adds a dedicated HTTP action to wait for
the message payload, for the request or the response depending it is used in
an http-request or an http-response ruleset. The time to wait is
configurable and, optionally, the minimum payload size to have before stop
to wait.

Both the http action and the old http analyzer rely on the same internal
function.
 2021-04-01 16:27:40 +02:00
Christopher Faulet
a434a00864 BUG/MINOR: payload/htx: Ingore L6 sample fetches for HTX streams/checks 
Use a L6 sample fetch on an HTX streams or a HTX health-check is meaningless
because data are not raw but structured. So now, these sample fetches fail
when called from an HTTP proxy. In addition, a warning has been added in the
configuration manual, at the begining of the L6 sample fetches section.

Note that req.len and res.len samples return the HTX data size instead of
failing. It is not accurate because it does not reflect the buffer size nor
the raw data length. But we keep it for backward compatibility purpose.
However it remains a bit strange to use it on an HTTP proxy.

This patch may be backported to all versions supporting the HTX, i.e as far
as 2.0. But the part about the health-checks is only valid for the 2.2 and
upper.
 2021-04-01 15:31:55 +02:00
Christopher Faulet
4d37e53dfc DOC: config: Add documentation about TCP to HTTP upgrades 
This patch adds explanation about chaining a TCP frontend to an HTTP
backend. It also explain how the HTTP upgrades work in this context. A note
has also been added in "Fetching HTTP samples" section to warning about HTTP
content processing in TCP.
 2021-04-01 13:24:40 +02:00
Christopher Faulet
ae863c62e3 MEDIUM: Add tcp-request switch-mode action to perform HTTP upgrade 
It is now possible to perform HTTP upgrades on a TCP stream from the
frontend side. To do so, a tcp-request content rule must be defined with the
switch-mode action, specifying the mode (for now, only http is supported)
and optionnaly the proto (h1 or h2).

This way it could be possible to set HTTP directives on a TCP frontend which
will only be evaluated if an upgrade is performed. This new way to perform
HTTP upgrades should replace progressively the old way, consisting to route
the request to an HTTP backend. And it should be also a good start to remove
all HTTP processing from tcp-request content rules.

This action is terminal, it stops the ruleset evaluation. It is only
available on proxy with the frontend capability.

The configuration manual has been updated accordingly.
 2021-04-01 13:17:19 +02:00
Christopher Faulet
982e17dcf5 DOC: config: Improve documentation about proto/check-proto keywords 
This patch adds a description about information provided by "haproxy -vv"
command regarding the available protocols. The description is adapted
depending the context (bind line, server line or health-check).
 2021-04-01 11:06:48 +02:00
Willy Tarreau
af6d88b359 [RELEASE] Released version 2.4-dev14 
Released version 2.4-dev14 with the following main changes :
    - MEDIUM: quic: Fix build.
    - MEDIUM: quic: Fix build.
    - CI: codespell: whitelist "Dragan Dosen"
    - CLEANUP: assorted typo fixes in the code and comments
    - CI: github actions: update LibreSSL to 3.2.5
    - REGTESTS: revert workaround for a crash with recent libressl on http-reuse sni
    - CLEANUP: mark defproxy as const on parse tune.fail-alloc
    - REGTESTS: remove unneeded experimental-mode in cli add server test
    - REGTESTS: wait for proper return of enable server in cli add server test
    - MINOR: compression: use pool_alloc(), not pool_alloc_dirty()
    - MINOR: spoe: use pool_alloc(), not pool_alloc_dirty()
    - MINOR: fcgi-app: use pool_alloc(), not pool_alloc_dirty()
    - MINOR: cache: use pool_alloc(), not pool_alloc_dirty()
    - MINOR: ssl: use pool_alloc(), not pool_alloc_dirty()
    - MINOR: opentracing: use pool_alloc(), not pool_alloc_dirty()
    - MINOR: dynbuf: make b_alloc() always check if the buffer is allocated
    - CLEANUP: compression: do not test for buffer before calling b_alloc()
    - CLEANUP: l7-retries: do not test the buffer before calling b_alloc()
    - MINOR: channel: simplify the channel's buffer allocation
    - MEDIUM: dynbuf: remove last usages of b_alloc_margin()
    - CLEANUP: dynbuf: remove b_alloc_margin()
    - CLEANUP: dynbuf: remove the unused b_alloc_fast() function
    - CLEANUP: pools: remove the unused pool_get_first() function
    - MINOR: pools: make the pool allocator support a few flags
    - MINOR: pools: add pool_zalloc() to return a zeroed area
    - CLEANUP: connection: use pool_zalloc() in conn_alloc_hash_node()
    - CLEANUP: filters: use pool_zalloc() in flt_stream_add_filter()
    - CLEANUP: spoe: use pool_zalloc() instead of pool_alloc+memset
    - CLEANUP: frontend: use pool_zalloc() in frontend_accept()
    - CLEANUP: mailers: use pool_zalloc() in enqueue_one_email_alert()
    - CLEANUP: resolvers: use pool_zalloc() in resolv_link_resolution()
    - CLEANUP: ssl: use pool_zalloc() in ssl_init_keylog()
    - CLEANUP: tcpcheck: use pool_zalloc() instead of pool_alloc+memset
    - CLEANUP: quic: use pool_zalloc() instead of pool_alloc+memset
    - MINOR: time: also provide a global, monotonic global_now_ms timer
    - BUG/MEDIUM: freq_ctr/threads: use the global_now_ms variable
    - MINOR: tools: introduce new option PA_O_DEFAULT_DGRAM on str2sa_range.
    - BUILD: tools: fix build error with new PA_O_DEFAULT_DGRAM
    - BUG/MINOR: ssl: Prevent disk access when using "add ssl crt-list"
    - CLEANUP: ssl: remove unused definitions
    - BUILD: ssl: guard ecdh functions with SSL_CTX_set_tmp_ecdh macro
    - MINOR: lua: Slightly improve function dumping the lua traceback
    - BUG/MEDIUM: debug/lua: Use internal hlua function to dump the lua traceback
    - BUG/MEDIUM: lua: Always init the lua stack before referencing the context
    - MINOR: fd: make fd_clr_running() return the remaining running mask
    - MINOR: fd: remove the unneeded running bit from fd_insert()
    - BUG/MEDIUM: fd: do not wait on FD removal in fd_delete()
    - CLEANUP: fd: remove unused fd_set_running_excl()
    - CLEANUP: fd: slightly simplify up _fd_delete_orphan()
    - BUG/MEDIUM: fd: Take the fd_mig_lock when closing if no DWCAS is available.
    - BUG/MEDIUM: release lock on idle conn killing on reached pool high count
    - BUG/MEDIUM: thread: Fix a deadlock if an isolated thread is marked as harmless
    - MINOR: tools: make url2ipv4 return the exact number of bytes parsed
    - BUG/MINOR: http_fetch: make hdr_ip() reject trailing characters
    - BUG/MEDIUM: mux-h1: make h1_shutw_conn() idempotent
    - BUG/MINOR: ssl: Fix update of default certificate
    - BUG/MINOR: ssl: Prevent removal of crt-list line if the instance is a default one
    - BUILD: ssl: introduce fine guard for ssl random extraction functions
    - REORG: global: move initcall register code in a dedicated file
    - REORG: global: move free acl/action in their related source files
    - REORG: split proxy allocation functions
    - MINOR: proxy: implement a free_proxy function
    - MINOR: proxy: define cap PR_CAP_LUA
    - MINOR: lua: properly allocate the lua Socket proxy
    - MINOR: lua: properly allocate the lua Socket servers
    - MINOR: vars: make get_vars() allow the session to be null
    - MINOR: vars: make the var() sample fetch keyword depend on nothing
    - CLEANUP: sample: remove duplicate "stopping" sample fetch keyword
    - MINOR: sample: make smp_resolve_args() return an allocate error message
    - MINOR: sample: add a new SMP_SRC_CONST sample capability
    - MINOR: sample: mark the truly constant sample fetch keywords as such
    - MINOR: sample: add a new CFG_PARSER context for samples
    - MINOR: action: add a new ACT_F_CFG_PARSER origin designation
    - MEDIUM: vars: add support for a "set-var" global directive
    - REGTESTS: add a basic reg-test for some "set-var" commands
    - MINOR: sample: add a new CLI_PARSER context for samples
    - MINOR: action: add a new ACT_F_CLI_PARSER origin designation
    - MINOR: vars/cli: add a "get var" CLI command to retrieve global variables
    - MEDIUM: cli: add a new experimental "set var" command
    - MINOR: compat: add short aliases for a few very commonly used types
    - BUILD: ssl: use EVP_CIPH_GCM_MODE macro instead of HA_OPENSSL_VERSION
    - MEDIUM: backend: use a trylock to grab a connection on high FD counts as well
 2021-03-27 09:42:09 +01:00
Willy Tarreau
13d2ba2a82 MEDIUM: vars: add support for a "set-var" global directive 
While we do support process-wide variables ("proc.<name>"), there was
no way to preset them from the configuration. This was particularly
limiting their usefulness since configs involving them always had to
first check if the variable was set prior to performing an operation.

This patch adds a new "set-var" directive in the global section that
supports setting the proc.<name> variables from an expression, like
other set-var actions do. The syntax however follows what is already
being done for setenv, which consists in having one argument for the
variable name and another one for the expression.

Only "constant" expressions are allowed here, such as "int", "str"
etc, combined with arithmetic or string converters, and variable
lookups. A few extra sample fetch keywords like "date", "rand" and
"uuid" are also part of the constant expressions and may make sense
to allow to create a random key or differentiate processes.

The way it was done consists in parsing a dummy rule an executing the
expression in the CFG_PARSE context, then releasing the expression.
This is safe because the sample that variables store does not hold a
back pointer to expression that created them.
 2021-03-26 16:34:53 +01:00
Willy Tarreau
7b0e00d943 BUG/MINOR: http_fetch: make hdr_ip() reject trailing characters 
The hdr_ip() sample fetch function will try to extract IP addresses
from a header field. These IP addresses are parsed using url2ipv4()
and if it fails it will fall back to inet_pton(AF_INET6), otherwise
will fail.

There is a small problem there which is that if a field starts with
an IP address and is immediately followed by some garbage, the IP
address part is still returned. This is a problem with fields such
as x-forwarded-for because it prevents detection of accidental
corruption or bug along the chain. For example, the following string:

   x-forwarded-for: 1.2.3.4; 5.6.7.8

or this one:

   x-forwarded-for: 1.2.3.4O    ( the last one being the letter 'O')

would still return "1.2.3.4" despite the trailing characters. This is
bad because it will silently cover broken code running on intermediary
proxies and may even in some cases allow haproxy to pass improperly
formatted headers after they were apparently validated, for example,
if someone extracts the address from this field to place it into
another one.

This issue would only affect the IPv4 parser, because the IPv6 parser
already uses inet_pton() which fails at the first invalid character and
rejects trailing port numbers.

In strict compliance with RFC7239, let's make sure that if there are any
characters left in the string, the parsing fails and makes hdr_ip()
return nothing. However, a special case has to be handled to support
IPv4 addresses followed by a colon and a valid port number, because till
now the parser used to implicitly accept them and it appears that this
practice, though rare, does exist at least in Azure:
   https://docs.microsoft.com/en-us/azure/application-gateway/how-application-gateway-works

This issue has always been there so the fix may be backported to all
versions. It will need the following commit in order to work as expected:

    MINOR: tools: make url2ipv4 return the exact number of bytes parsed

Many thanks to https://twitter.com/melardev and the BitMEX Security Team
for their detailed report.
 2021-03-25 15:30:06 +01:00
Willy Tarreau
09cc669afb [RELEASE] Released version 2.4-dev13 
Released version 2.4-dev13 with the following main changes :
    - BUG/MEDIUM: cli: fix "help" crashing since recent spelling fixes
    - BUG/MINOR: cfgparse: use the GLOBAL not LISTEN keywords list for spell checking
    - MINOR: tools: improve word fingerprinting by counting presence
    - MINOR: tools: do not sum squares of differences for word fingerprints
    - MINOR: cli: improve fuzzy matching to work on all remaining words at once
    - MINOR: cli: sort the suggestions by order of relevance
    - MINOR: cli: limit spelling suggestions to 5
    - MINOR: cfgparse/proxy: also support spelling fixes on options
    - BUG/MINOR: resolvers: Add missing case-insensitive comparisons of DNS hostnames
    - MINOR: time: export the global_now variable
    - BUG/MINOR: freq_ctr/threads: make use of the last updated global time
    - MINOR: freq_ctr/threads: relax when failing to update a sliding window value
    - MINOR/BUG: mworker/cli: do not use the unix_bind prefix for the master CLI socket
    - MINOR: mworker/cli: alert the user if we enabled a master CLI but not the master-worker mode
    - MINOR: cli: implement experimental-mode
    - REORG: server: add a free server function
    - MINOR: cfgparse: always alloc idle conns task
    - REORG: server: move keywords in srv_kws
    - MINOR: server: remove fastinter from mistyped kw list
    - REORG: server: split parse_server
    - REORG: server: move alert traces in parse_server
    - REORG: server: rename internal functions from parse_server
    - REORG: server: attach servers in parse_server
    - REORG: server: use flags for parse_server
    - MINOR: server: prepare parsing for dynamic servers
    - MINOR: stats: export function to allocate extra proxy counters
    - MEDIUM: server: implement 'add server' cli command
    - REGTESTS: implement test for 'add server' cli
    - MINOR: server: enable standard options for dynamic servers
    - MINOR: server: support keyword proto in 'add server' cli
    - BUG/MINOR: protocol: add missing support of dgram unix socket.
    - CLEANUP: Fix a typo in fix_is_valid description
    - MINOR: raw_sock: Add a close method.
    - MEDIUM: connections: Introduce a new XPRT method, start().
    - MEDIUM: connections: Implement a start() method for xprt_handshake.
    - MEDIUM: connections: Implement a start() method in ssl_sock.
    - MINOR: muxes: garbage collect the reset() method.
    - CLEANUP: tcp-rules: Fix a typo in error messages about expect-netscaler-cip
    - MEDIUM: lua: Use a per-thread counter to track some non-reentrant parts of lua
    - BUG/MEDIUM: debug/lua: Don't dump the lua stack if not dumpable
 2021-03-19 17:16:18 +01:00
Christopher Faulet
ed4bef76bd CLEANUP: Fix a typo in fix_is_valid description 
MsgType tag was misspelled.
 2021-03-18 20:31:04 +01:00
Willy Tarreau
acdd47db9d [RELEASE] Released version 2.4-dev12 
Released version 2.4-dev12 with the following main changes :
    - CLEANUP: connection: Use `VAR_ARRAY` in `struct tlv` definition
    - CLEANUP: connection: Remove useless test for NULL before calling `pool_free()`
    - CLEANUP: connection: Use istptr / istlen for proxy_unique_id
    - MINOR: connection: Use a `struct ist` to store proxy_authority
    - CLEANUP: connection: Consistently use `struct ist` to process all TLV types
    - BUILD: task: fix build at -O0 with threads disabled
    - BUILD: bug: refine HA_LINK_ERROR() to only be used on gcc and derivatives
    - CLEANUP: config: make the cfg_keyword parsers take a const for the defproxy
    - BUILD: connection: do not use VAR_ARRAY in struct tlv
    - BUG/MEDIUM: session: NULL dereference possible when accessing the listener
    - MINOR: build: force CC to set a return code when probing options
    - CLEANUP: stream: rename a few remaining occurrences of "stream *sess"
    - BUG/MEDIUM: resolvers: handle huge responses over tcp servers.
    - CLEANUP: config: also address the cfg_keyword API change in the compression code
    - BUG/MEDIUM: ssl: properly remove the TASK_HEAVY flag at end of handshake
    - BUG/MINOR: sample: Rename SenderComID/TargetComID to SenderCompID/TargetCompID
    - MINOR: task: give the scheduler a bit more flexibility in the runqueue size
    - OPTIM: task: automatically adjust the default runqueue-depth to the threads
    - BUG/MINOR: connection: Missing QUIC initialization
    - BUG/MEDIUM: stick-tables: fix ref counter in table entry using multiple http tracksc.
    - BUILD: atomic/arm64: force the register pairs to use in __ha_cas_dw()
    - BUG/MEDIUM: filters: Set CF_FL_ANALYZE on channels when filters are attached
    - BUG/MINOR: tcpcheck: Update .health threshold of agent inside an agent-check
    - BUG/MINOR: proxy/session: Be sure to have a listener to increment its counters
    - BUG/MINOR: tcpcheck: Fix double free on error path when parsing tcp/http-check
    - BUG/MINOR: server-state: properly handle the case where the base is not set
    - BUG/MINOR: server-state: use the argument, not the global state
    - CLEANUP: tcp-rules: add missing actions in the tcp-request error message
    - CLEANUP: vars: make the error message clearer on missing arguments for set-var
    - CLEANUP: http-rules: remove the unexpected comma before the list of action keywords
    - CLEANUP: actions: the keyword must always be const from the rule
    - MINOR: tools: add simple word fingerprinting to find similar-looking words
    - MINOR: cfgparse: add cfg_find_best_match() to suggest an existing word
    - MINOR: cfgparse: suggest correct spelling for unknown words in proxy sections
    - MINOR: cfgparse: suggest correct spelling for unknown words in global section
    - MINOR: cfgparse/server: try to fix spelling mistakes on server lines
    - MINOR: cfgparse/bind: suggest correct spelling for unknown bind keywords
    - MINOR: actions: add a function to suggest an action ressembling a given word
    - MINOR: http-rules: suggest approaching action names on mismatch
    - MINOR: tcp-rules: suggest approaching action names on mismatch
    - BUG/MINOR: cfgparse/server: increment the extra keyword counter one at a time
    - Revert "BUG/MINOR: resolvers: Only renew TTL for SRV records with an additional record"
    - BUG/MINOR: resolvers: Consider server to have no IP on DNS resolution error
    - BUG/MINOR: resolvers: Reset server address on DNS error only on status change
    - BUG/MINOR: resolvers: Unlink DNS resolution to set RMAINT on SRV resolution
    - BUG/MEDIUM: resolvers: Don't set an address-less server as UP
    - BUG/MEDIUM: resolvers: Fix the loop looking for an existing ADD item
    - MINOR: resolvers: new function find_srvrq_answer_record()
    - BUG/MINOR; resolvers: Ignore DNS resolution for expired SRV item
    - BUG/MEDIUM: resolvers: Trigger a DNS resolution if an ADD item is obsolete
    - MINOR: resolvers: Use a function to remove answers attached to a resolution
    - MINOR: resolvers: Purge answer items when a SRV resolution triggers an error
    - MINOR: resolvers: Add function to change the srv status based on SRV resolution
    - MINOR: resolvers: Directly call srvrq_update_srv_state() when possible
    - BUG/MEDIUM: resolvers: Don't release resolution from a requester callbacks
    - BUG/MEDIUM: resolvers: Skip DNS resolution at startup if SRV resolution is set
    - MINOR: resolvers: Use milliseconds for cached items in resolver responses
    - MINOR: resolvers: Don't try to match immediatly renewed ADD items
    - CLEANUP: resolvers: Use ha_free() in srvrq_resolution_error_cb()
    - CLEANUP: resolvers: Perform unsafe loop on requester list when possible
    - BUG/MINOR: cli: make sure "help", "prompt", "quit" are enabled at master level
    - CLEANUP: cli: fix misleading comment and better indent the access level flags
    - MINOR: cli: set the ACCESS_MASTER* bits on the master bind_conf
    - MINOR: cli: test the appctx level for master access instead of comparing pointers
    - MINOR: cli: print the error message in the parser function itself
    - MINOR: cli: filter the list of commands to the matching part
    - MEDIUM: cli: apply spelling fixes for known commands before listing them
    - MINOR: tools: add the ability to update a word fingerprint
    - MINOR: cli: apply the fuzzy matching on the whole command instead of words
    - CLEANUP: cli: rename MAX_STATS_ARGS to MAX_CLI_ARGS
    - CLEANUP: cli: rename the last few "stats_" to "cli_"
    - CLEANUP: task: make sure tasklet handlers always indicate their statuses
    - CLEANUP: assorted typo fixes in the code and comments
 2021-03-13 11:48:28 +01:00
Willy Tarreau
060a761248 OPTIM: task: automatically adjust the default runqueue-depth to the threads 
The recent default runqueue size reduction appeared to have significantly
lowered performance on low-thread count configs. Testing various values
runqueue values on different workloads under thread counts ranging from
1 to 64, it appeared that lower values are more optimal for high thread
counts and conversely. It could even be drawn that the optimal value for
various workloads sits around 280/sqrt(nbthread), and probably has to do
with both the L3 cache usage and how to optimally interlace the threads'
activity to minimize contention. This is much easier to optimally
configure, so let's do this by default now.
 2021-03-10 11:15:34 +01:00
Daniel Corbett
befef70e23 BUG/MINOR: sample: Rename SenderComID/TargetComID to SenderCompID/TargetCompID 
The recently introduced Financial Information eXchange (FIX)
converters have some hard coded tags based on the specification that
were misspelled. Specifically, SenderComID and TargetComID should
be SenderCompID and TargetCompID according to the specification [1][2].

This patch updates all references, which includes the converters
themselves, the regression test, and the documentation.

[1] https://fiximate.fixtrading.org/en/FIX.5.0SP2_EP264/tag49.html
[2] https://fiximate.fixtrading.org/en/FIX.5.0SP2_EP264/tag56.html
 2021-03-10 10:44:20 +01:00
Emeric Brun
4c75195f5b BUG/MEDIUM: resolvers: handle huge responses over tcp servers. 
Parameter "accepted_payload_size" is currently considered regardless
the used nameserver is using TCP or UDP. It remains mandatory to annouce
such capability to support e-dns, so a value have to be announced also
in TCP. Maximum DNS message size in TCP is limited by protocol to 65535
and so for UDP (65507) if system supports such UDP messages. But
the maximum value for this option was arbitrary forced to 8192.

This patch change this maximum to 65535 to allow user to set bigger value
for UDP if its system supports. It also sets accepted_payload_size
in TCP allowing to retrieve huge responses if the configuration uses
TCP nameservers.

The request announcing the accepted_payload_size capability is currently
built at resolvers level and is common to all used nameservers of the
section regardess transport protocol used. A further patch should be
made to at least specify a different payload size depending of the
transport, and perhaps could be forced to 65535 in case of TCP and
maximum would be forced back to 65507 matching UDP max.

This patch is appliable since 2.4 version
 2021-03-09 15:44:46 +01:00
Willy Tarreau
7bbc6c9ac3 [RELEASE] Released version 2.4-dev11 
Released version 2.4-dev11 with the following main changes :
    - CI: codespell: skip Makefile for spell check
    - CLEANUP: assorted typo fixes in the code and comments
    - BUG/MINOR: tcp-act: Don't forget to set the original port for IPv4 set-dst rule
    - BUG/MINOR: connection: Use the client's dst family for adressless servers
    - BUG/MEDIUM: spoe: Kill applets if there are pending connections and nbthread > 1
    - CLEANUP: Use ist2(const void*, size_t) whenever possible
    - CLEANUP: Use IST_NULL whenever possible
    - BUILD: proxy: Missing header inclusion for quic_transport_params_init()
    - BUILD: quic: Implicit conversion between SSL related enums.
    - DOC: spoe: Add a note about fragmentation support in HAProxy
    - MINOR: contrib: add support for heartbeat control messages.
    - MINOR: contrib: Enhance peers dissector heuristic.
    - BUG/MINOR: mux-h2: Fix typo in scheme adjustment
    - CLEANUP: Reapply the ist2() replacement patch
    - CLEANUP: Use istadv(const struct ist, const size_t) whenever possible
    - CLEANUP: Use isttest(const struct ist) whenever possible
    - Revert "CI: Pin VTest to a known good commit"
    - CLEANUP: backend: fix a wrong comment
    - BUG/MINOR: backend: free allocated bind_addr if reuse conn
    - MINOR: backend: handle reuse for conns with no server as target
    - REGTESTS: test http-reuse if no server target
    - BUG/MINOR: hlua: Don't strip last non-LWS char in hlua_pushstrippedstring()
    - BUG/MINOR: server-state: Don't load server-state file for disabled backends
    - CLEANUP: dns: Use DISGUISE() on a never-failing ring_attach() call
    - CLEANUP: dns: Remove useless test on ns->dgram in dns_connect_nameserver()
    - DOC: fix originalto except clause on destination address
    - CLEANUP: Use the ist() macro whenever possible
    - CLEANUP: Replace for loop with only a condition by while
    - REORG: atomic: reimplement pl_cpu_relax() from atomic-ops.h
    - BUG/MINOR: mt-list: always perform a cpu_relax call on failure
    - MINOR: atomic: add armv8.1-a atomics variant for cas-dw
    - MINOR: atomic: implement a more efficient arm64 __ha_cas_dw() using pairs
    - BUG/MINOR: ssl: don't truncate the file descriptor to 16 bits in debug mode
    - MEDIUM: pools: add CONFIG_HAP_NO_GLOBAL_POOLS and CONFIG_HAP_GLOBAL_POOLS
    - MINOR: pools: double the local pool cache size to 1 MB
    - MINOR: stream: use ABORT_NOW() and not abort() in stream_dump_and_crash()
    - CLEANUP: stream: explain why we queue the stream at the head of the server list
    - MEDIUM: backend: use a trylock when trying to grab an idle connection
    - REORG: tools: promote the debug PRNG to more general use as a statistical one
    - OPTIM: lb-random: use a cheaper PRNG to pick a server
    - MINOR: task: stop abusing the nice field to detect a tasklet
    - MINOR: task: move the nice field to the struct task only
    - MEDIUM: task: extend the state field to 32 bits
    - MINOR: task: add an application specific flag to the state: TASK_F_USR1
    - MEDIUM: muxes: mark idle conns tasklets with TASK_F_USR1
    - MINOR: xprt: add new xprt_set_idle and xprt_set_used methods
    - MEDIUM: ssl: implement xprt_set_used and xprt_set_idle to relax context checks
    - MINOR: server: don't read curr_used_conns multiple times
    - CLEANUP: global: reorder some fields to respect cache lines
    - CLEANUP: sockpair: silence a coverity check about fcntl()
    - CLEANUP: lua: set a dummy file name and line number on the dummy servers
    - MINOR: server: add a global list of all known servers
    - MINOR: cfgparse: finish to set up servers outside of the proxy setup loop
    - MINOR: server: allocate a per-thread struct for the per-thread connections stuff
    - MINOR: server: move actconns to the per-thread structure
    - CLEANUP: server: reorder some fields in the server struct to respect cache lines
    - MINOR: backend: add a BUG_ON if conn mux NULL in connect_server
    - BUG/MINOR: backend: fix condition for reuse on mode HTTP
    - BUILD: Fix build when using clang without optimizing.
    - CLEANUP: assorted typo fixes in the code and comments
 2021-03-05 21:24:23 +01:00
Amaury Denoyelle
f8b4292560 DOC: fix originalto except clause on destination address 
Fix the description of the except clause of the originalto option. The
destination address and not the source is compared with the except range
address to prevent the addition of the X-Original-To header.

This can be backported in every releases.
 2021-03-04 18:58:29 +01:00
Ilya Shipitsin
0de36adb5c CLEANUP: assorted typo fixes in the code and comments 
This is 18th iteration of typo fixes
 2021-02-27 09:01:43 +01:00
Willy Tarreau
8ab65c201a [RELEASE] Released version 2.4-dev10 
Released version 2.4-dev10 with the following main changes :
    - BUILD: SSL: introduce fine guard for RAND_keep_random_devices_open
    - MINOR: Configure the `cpp` userdiff driver for *.[ch] in .gitattributes
    - BUG/MINOR: ssl/cli: potential null pointer dereference in "set ssl cert"
    - BUG/MINOR: sample: secure convs that accept base64 string and var name as args
    - BUG/MEDIUM: vars: make functions vars_get_by_{name,desc} thread-safe
    - CLEANUP: vars: make smp_fetch_var() to reuse vars_get_by_desc()
    - DOC: muxes: add a diagram of the exchanges between muxes and outer world
    - BUG/MEDIUM: proxy: use thread-safe stream killing on hard-stop
    - BUG/MEDIUM: cli/shutdown sessions: make it thread-safe
    - BUG/MINOR: proxy: wake up all threads when sending the hard-stop signal
    - MINOR: stream: add an "epoch" to figure which streams appeared when
    - MINOR: cli/streams: make "show sess" dump all streams till the new epoch
    - MINOR: streams: use one list per stream instead of a global one
    - MEDIUM: streams: do not use the streams lock anymore
    - BUILD: dns: avoid a build warning when threads are disabled (dss unused)
    - MEDIUM: task: remove the tasks_run_queue counter and have one per thread
    - MINOR: tasks: do not maintain the rqueue_size counter anymore
    - CLEANUP: tasks: use a less confusing name for task_list_size
    - CLEANUP: task: move the tree root detection from __task_wakeup() to task_wakeup()
    - MINOR: task: limit the remote thread wakeup to the global runqueue only
    - MINOR: task: move the allocated tasks counter to the per-thread struct
    - CLEANUP: task: split the large tasklet_wakeup_on() function in two
    - BUG/MINOR: fd: properly wait for !running_mask in fd_set_running_excl()
    - BUG/MINOR: resolvers: Fix condition to release received ARs if not assigned
    - BUG/MINOR: resolvers: Only renew TTL for SRV records with an additional record
    - BUG/MINOR: resolvers: new callback to properly handle SRV record errors
    - BUG/MEDIUM: resolvers: Reset server address and port for obselete SRV records
    - BUG/MEDIUM: resolvers: Reset address for unresolved servers
    - DOC: Update the module list in MAINTAINERS file
    - MINOR: htx: Add function to reserve the max possible size for an HTX DATA block
    - DOC: Update the HTX API documentation
    - DOC: Update the filters guide
    - BUG/MEDIUM: contrib/prometheus-exporter: fix segfault in listener name dump
    - MINOR: task: split the counts of local and global tasks picked
    - MINOR: task: do not use __task_unlink_rq() from process_runnable_tasks()
    - MINOR: task: don't decrement then increment the local run queue
    - CLEANUP: task: re-merge __task_unlink_rq() with task_unlink_rq()
    - MINOR: task: make grq_total atomic to move it outside of the grq_lock
    - MINOR: tasks: also compute the tasklet latency when DEBUG_TASK is set
    - MINOR: task: make tasklet wakeup latency measurements more accurate
    - MINOR: server: Be more strict on the server-state line parsing
    - MINOR: server: Only fill one array when parsing a server-state line
    - MEDIUM: server: Refactor apply_server_state() to make it more readable
    - CLEANUP: server: Rename state_line node to node instead of name_name
    - CLEANUP: server: Rename state_line structure into server_state_line
    - CLEANUP: server: Use a local eb-tree to store lines of the global server-state file
    - MINOR: server: Be more strict when reading the version of a server-state file
    - MEDIUM: server: Store parsed params of a server-state line in the tree
    - MINOR: server: Remove cached line from global server-state tree when found
    - MINOR: server: Move loading state of servers in a dedicated function
    - MEDIUM: server: Use a tree to store local server-state lines
    - MINOR: server: Parse and store server-state lines in a dedicated function
    - MEDIUM: server: Don't load server-state file if a line is corrupted
    - REORG: server: Export and rename some functions updating server info
    - REORG: server-state: Move functions to deal with server-state in its own file
    - MINOR: server-state: Don't load server-state file for serverless proxies
    - CLEANUP: muxes: Remove useless if condition in show_fd function
    - BUG/MINOR: stats: fix compare of no-maint url suffix
    - MINOR: task: limit the number of subsequent heavy tasks with flag TASK_HEAVY
    - MINOR: ssl: mark the SSL handshake tasklet as heavy
    - CLEANUP: server: rename srv_cleanup_{idle,toremove}_connections()
    - BUG/MINOR: ssl: potential null pointer dereference in ckchs_dup()
    - MINOR: task: add one extra tasklet class: TL_HEAVY
    - MINOR: task: place the heavy elements in TL_HEAVY
    - MINOR: task: only limit TL_HEAVY tasks but not others
    - BUG/MINOR: http-ana: Only consider dst address to process originalto option
    - MINOR: tools: Add net_addr structure describing a network addess
    - MINOR: tools: Add function to compare an address to a network address
    - MEDIUM: http-ana: Add IPv6 support for forwardfor and orignialto options
    - CLEANUP: hlua: Use net_addr structure internally to parse and compare addresses
    - REGTESTS: Add script to test except param for fowardedfor/originalto options
    - DOC: scheduler: add a diagram showing the different queues and their usages
    - CLEANUP: tree-wide: replace free(x);x=NULL with ha_free(&x)
    - CLEANUP: config: replace a few free() with ha_free()
    - CLEANUP: vars: always zero the pointers after a free()
    - CLEANUP: ssl: remove a useless "if" before freeing an error message
    - CLEANUP: ssl: make ssl_sock_free_srv_ctx() zero the pointers after free
    - CLEANUP: ssl: use realloc() instead of free()+malloc()
 2021-02-26 22:49:10 +01:00
Christopher Faulet
5d1def623a MEDIUM: http-ana: Add IPv6 support for forwardfor and orignialto options 
A network may be specified to avoid header addition for "forwardfor" and
"orignialto" option via the "except" parameter. However, only IPv4
networks/addresses are supported. This patch adds the support of IPv6.

To do so, the net_addr structure is used to store the parameter value in the
proxy structure. And ipcmp2net() function is used to perform the comparison.

This patch should fix the issue #1145. It depends on the following commit:

  * c6ce0ab MINOR: tools: Add function to compare an address to a network address
  * 5587287 MINOR: tools: Add net_addr structure describing a network addess
 2021-02-26 13:52:48 +01:00
Willy Tarreau
31dd393da0 [RELEASE] Released version 2.4-dev9 
Released version 2.4-dev9 with the following main changes :
    - BUG/MINOR: server: Remove RMAINT from admin state when loading server state
    - CLEANUP: check: fix get_check_status_info declaration
    - CLEANUP: contrib/prometheus-exporter: align for with srv status case
    - MEDIUM: stats: allow to select one field in `stats_fill_li_stats`
    - MINOR: stats: add helper to get status string
    - MEDIUM: contrib/prometheus-exporter: add listen stats
    - BUG/MINOR: dns: add test on result getting value from buffer into ring.
    - BUG/MINOR: dns: dns_connect_server must return -1 unsupported nameserver's type
    - BUG/MINOR: dns: missing test writing in output channel in session handler
    - BUG/MINOR: dns: fix ring attach control on dns_session_new
    - BUG/MEDIUM: dns: fix multiple double close on fd in dns.c
    - BUG/MAJOR: connection: prevent double free if conn selected for removal
    - BUG/MINOR: session: atomically increment the tracked sessions counter
    - REGTESTS: fix http_reuse_conn_hash proxy test
    - BUG/MINOR: backend: do not call smp_make_safe for sni conn hash
    - MINOR: connection: remove pointers for prehash in conn_hash_params
    - BUG/MINOR: checks: properly handle wrapping time in __health_adjust()
    - BUG/MEDIUM: checks: don't needlessly take the server lock in health_adjust()
    - DEBUG: thread: add 5 extra lock labels for statistics and debugging
    - OPTIM: server: switch the actconn list to an mt-list
    - Revert "MINOR: threads: change lock_t to an unsigned int"
    - MINOR: lb/api: let callers of take_conn/drop_conn tell if they have the lock
    - OPTIM: lb-first: do not take the server lock on take_conn/drop_conn
    - OPTIM: lb-leastconn: do not take the server lock on take_conn/drop_conn
    - OPTIM: lb-leastconn: do not unlink the server if it did not change
    - MINOR: tasks: add DEBUG_TASK to report caller info in a task
    - MINOR: tasks/debug: add some extra controls of use-after-free in DEBUG_TASK
    - BUG/MINOR: sample: Always consider zero size string samples as unsafe
    - MINOR: cli: add missing agent commands for set server
    - BUILD/MEDIUM: da Adding pcre2 support.
    - BUILD: ssl: introduce fine guard for OpenSSL specific SCTL functions
    - REGTESTS: reorder reuse conn proxy protocol test
    - DOC: explain the relation between pool-low-conn and tune.idle-pool.shared
    - MINOR: tasks: refine the default run queue depth
    - MINOR: listener: refine the default MAX_ACCEPT from 64 to 4
    - MINOR: mux_h2: do not try to remove front conn from idle trees
    - REGTESTS: workaround for a crash with recent libressl on http-reuse sni
    - BUG/MEDIUM: lists: Avoid an infinite loop in MT_LIST_TRY_ADDQ().
    - MINOR: connection: allocate dynamically hash node for backend conns
    - DOC: DeviceAtlas documentation typo fix.
    - BUG/MEDIUM: spoe: Resolve the sink if a SPOE logs in a ring buffer
    - BUG/MINOR: http-rules: Always replace the response status on a return action
    - BUG/MINOR: server: Init params before parsing a new server-state line
    - BUG/MINOR: server: Be sure to cut the last parsed field of a server-state line
    - MEDIUM: server: Don't introduce a new server-state file version
    - DOC: contrib/prometheus-exporter: remove htx reference
    - REGTESTS: contrib/prometheus-exporter: test NaN values
    - REGTESTS: contrib/prometheus-exporter: test well known labels
    - CI: github actions: switch to stable LibreSSL release
    - BUG/MINOR: server: Fix test on number of fields allowed in a server-state line
    - MINOR: dynbuf: make the buffer wait queue per thread
    - MINOR: dynbuf: use regular lists instead of mt_lists for buffer_wait
    - MINOR: dynbuf: pass offer_buffers() the number of buffers instead of a threshold
    - MINOR: sched: have one runqueue ticks counter per thread
 2021-02-20 13:30:31 +01:00
Willy Tarreau
66161326fd MINOR: listener: refine the default MAX_ACCEPT from 64 to 4 
The maximum number of connections accepted at once by a thread for a single
listener used to default to 64 divided by the number of processes but the
tasklet-based model is much more scalable and benefits from smaller values.
Experimentation has shown that 4 gives the highest accept rate for all
thread values, and that 3 and 5 come very close, as shown below (HTTP/1
connections forwarded per second at multi-accept 4 and 64):

 ac\thr|    1     2    4     8     16
 ------+------------------------------
      4|   80k  106k  168k  270k  336k
     64|   63k   89k  145k  230k  274k

Some tests were also conducted on SSL and absolutely no change was observed.

The value was placed into a define because it used to be spread all over the
code.

It might be useful at some point to backport this to 2.3 and 2.2 to help
those who observed some performance regressions from 1.6.
 2021-02-19 16:02:04 +01:00
Willy Tarreau
4327d0ac00 MINOR: tasks: refine the default run queue depth 
Since a lot of internal callbacks were turned to tasklets, the runqueue
depth had not been readjusted from the default 200 which was initially
used to favor batched processing. But nowadays it appears too large
already based on the following tests conducted on a 8c16t machine with
a simple config involving "balance leastconn" and one server. The setup
always involved the two threads of a same CPU core except for 1 thread,
and the client was running over 1000 concurrent H1 connections. The
number of requests per second is reported for each (runqueue-depth,
nbthread) couple:

 rq\thr|    1     2     4     8    16
 ------+------------------------------
     32|  120k  159k  276k  477k  698k
     40|  122k  160k  276k  478k  722k
     48|  121k  159k  274k  482k  720k
     64|  121k  160k  274k  469k  710k
    200|  114k  150k  247k  415k  613k  <-- default

It's possible to save up to about 18% performance by lowering the
default value to 40. One possible explanation to this is that checking
I/Os more frequently allows to flush buffers faster and to smooth the
I/O wait time over multiple operations instead of alternating phases
of processing, waiting for locks and waiting for new I/Os.

The total round trip time also fell from 1.62ms to 1.40ms on average,
among which at least 0.5ms is attributed to the testing tools since
this is the minimum attainable on the loopback.

After some observation it would be nice to backport this to 2.3 and
2.2 which observe similar improvements, since some users have already
observed some perf regressions between 1.6 and 2.2.
 2021-02-19 16:01:55 +01:00
Willy Tarreau
0784db8566 DOC: explain the relation between pool-low-conn and tune.idle-pool.shared 
Disabling idle-pool sharing can result in awful performance in presence
of a not so high number of threads, because the number of available idle
connections will be shared among threads, resulting in most of them
abandonning their connections after a request is done if there are already
enough total available. This is a case where pool-low-conn ought to be
used to preserve a number of connections for each thread, but this relation
isn't obvious as is. Let's add mentions about this with both keywords.
 2021-02-19 11:49:04 +01:00
Willy Tarreau
dc626ecdd2 [RELEASE] Released version 2.4-dev8 
Released version 2.4-dev8 with the following main changes :
    - BUILD: ssl: fix typo in HAVE_SSL_CTX_ADD_SERVER_CUSTOM_EXT macro
    - BUILD: ssl: guard SSL_CTX_add_server_custom_ext with special macro
    - BUG/MINOR: mux-h1: Don't emit extra CRLF for empty chunked messages
    - MINOR: contrib/prometheus-exporter: use stats desc when possible followup
    - MEDIUM: contrib/prometheus-exporter: export base stick table stats
    - CLEANUP: assorted typo fixes in the code and comments
    - CLEANUP: check: fix some typo in comments
    - CLEANUP: tools: typo in `strl2irc` mention
    - BUILD: ssl: guard SSL_CTX_set_msg_callback with SSL_CTRL_SET_MSG_CALLBACK macro
    - MEDIUM: ssl: add a rwlock for SSL server session cache
    - BUG/MINOR: intops: fix mul32hi()'s off-by-one
    - BUG/MINOR: freq_ctr: fix a wrong delay calculation in next_event_delay()
    - MINOR: stick-tables/counters: add http_fail_cnt and http_fail_rate data types
    - MINOR: ssl: add SSL_SERVER_LOCK label in threads.h
    - BUG/MINOR: mux-h1: Don't increment HTTP error counter for 408/500/501 errors
    - BUG/MINOR: http-ana: Don't increment HTTP error counter on internal errors
    - BUG/MEDIUM: mux-h1: Always set CS_FL_EOI for response in MSG_DONE state
    - BUG/MINOR: mux-h1: Fix data skipping for bodyless responses
    - BUG/MINOR: mux-h1: Don't blindly skip EOT block for non-chunked messages
    - BUG/MEDIUM: mux-h2: Add EOT block when EOM flag is set on an empty HTX message
    - MINOR: mux-h1: Be sure EOM flag is set when processing end of outgoing message
    - REGTESTS: Add a script to test payload skipping for bodyless HTTP responses
    - BUG/MINOR: server: re-align state file fields number
    - CLEANUP: muxes: Remove useless calls to b_realign_if_empty()
    - BUG/MINOR: tools: Fix a memory leak on error path in parse_dotted_uints()
    - CLEANUP: remove unused variable assigned found by Coverity
    - CLEANUP: queue: Remove useless tests on p or pp in pendconn_process_next_strm()
    - BUG/MINOR: backend: hold correctly lock when killing idle conn
    - MEDIUM: connection: protect idle conn lists with locks
    - MEDIUM: connection: replace idle conn lists by eb trees
    - MINOR: backend: search conn in idle/safe trees after available
    - MINOR: backend: search conn in idle tree after safe on always reuse
    - MINOR: connection: prepare hash calcul for server conns
    - MINOR: connection: use the srv pointer for the srv conn hash
    - MINOR: backend: compare conn hash for session conn reuse
    - MINOR: connection: use sni as parameter for srv conn hash
    - MINOR: reg-tests: test http-reuse with sni
    - MINOR: backend: rewrite alloc of stream target address
    - MINOR: connection: use dst addr as parameter for srv conn hash
    - MINOR: reg-test: test http-reuse with specific dst addr
    - MINOR: backend: rewrite alloc of connection src address
    - MINOR: connection: use src addr as parameter for srv conn hash
    - MINOR: connection: use proxy protocol as parameter for srv conn hash
    - MINOR: reg-tests: test http-reuse with proxy protocol
    - MINOR: doc: update http reuse for new eligilible connections
    - BUG/MINOR: backend: fix compilation without ssl
    - REGTESTS: adjust http_reuse_conn_hash requirements
    - REGTESTS: deactivate a failed test on CI in http_reuse_conn_hash
    - REGTESTS: fix sni used in http_reuse_conn_hash for libressl 3.3.0
    - CI: cirrus: update FreeBSD image to 12.2
    - MEDIUM: cli: add check-addr command
    - MEDIUM: cli: add agent-port command
    - MEDIUM: server: add server-states version 2
    - MEDIUM: server: support {check,agent}_addr, agent_port in server state
    - MINOR: server: enhance error precision when applying server state
    - BUG/MINOR: server: Fix server-state-file-name directive
    - CLEANUP: deinit: release global and per-proxy server-state variables on deinit
    - BUG/MEDIUM: config: don't pick unset values from last defaults section
    - BUG/MINOR: stats: revert the change on ST_CONVDONE
    - BUG/MINOR: cfgparse: do not mention "addr:port" as supported on proxy lines
    - BUG/MINOR: http-htx: defpx must be a const in proxy_dup_default_conf_errors()
    - BUG/MINOR: tcpheck: the source list must be a const in dup_tcpcheck_var()
    - BUILD: proxy: add missing compression-t.h to proxy-t.h
    - REORG: move init_default_instance() to proxy.c and pass it the defproxy pointer
    - REORG: proxy: centralize the proxy allocation code into alloc_new_proxy()
    - MEDIUM: proxy: only take defaults when a default proxy is passed.
    - MINOR: proxy: move the defproxy freeing code to proxy.c
    - MINOR: proxy: always properly reset the just freed default instance pointers
    - BUG/MINOR: extcheck: proxy_parse_extcheck() must take a const for the defproxy
    - BUG/MINOR: tcpcheck: proxy_parse_*check*() must take a const for the defproxy
    - BUG/MINOR: server: parse_server() must take a const for the defproxy
    - MINOR: cfgparse: move defproxy to cfgparse-listen as a static
    - MINOR: proxy: add a new capability PR_CAP_DEF
    - MINOR: cfgparse: check PR_CAP_DEF instead of comparing poiner against defproxy
    - MINOR: cfgparse: use a pointer to the current default proxy
    - MINOR: proxy: also store the name for a defaults section
    - MINOR: proxy: support storing defaults sections into their own tree
    - MEDIUM: proxy: store the default proxies in a tree by name
    - MEDIUM: cfgparse: allow a proxy to designate the defaults section to use
    - MINOR: http: add baseq sample fetch
    - CLEANUP: tcpcheck: Remove a useless test on port variable
    - BUG/MINOR: server: Don't call fopen() with server-state filepath set to NULL
    - CLEANUP: server: Remove useless "filepath" variable in apply_server_state()
    - MINOR: peers/cli: do not dump the peers dictionaries by default on "show peers"
    - MINOR: cfgparse: implement a simple if/elif/else/endif macro block handler
    - DOC: tune: explain the origin of block size for ssl.cachesize
    - MINOR: tcp: add support for defer-accept on FreeBSD.
    - MINOR: ring: adds new ring_init function.
    - CLEANUP: channel: fix comment in ci_putblk.
    - BUG/MINOR: dns: add missing sent counter and parent id to dns counters.
    - BUG/MINOR: resolvers: fix attribute packed struct for dns
    - MINOR: resolvers: renames some resolvers internal types and removes dns prefix
    - MINOR: resolvers: renames type dns_resolvers to resolvers.
    - MINOR: resolvers: renames some resolvers specific types to not use dns prefix
    - MINOR: resolvers: renames some dns prefixed types using resolv prefix.
    - MINOR: resolvers: renames resolvers DNS_RESP_* errcodes RSLV_RESP_*
    - MINOR: resolvers: renames resolvers DNS_UPD_* returncodes to RSLV_UPD_*
    - MINOR: resolvers: rework prototype suffixes to split resolving and dns.
    - MEDIUM: resolvers: move resolvers section parsing from cfgparse.c to dns.c
    - MINOR: resolvers: replace nameserver's resolver ref by generic parent pointer
    - MINOR: resolvers: rework dns stats prototype because specific to resolvers
    - MEDIUM: resolvers: split resolving and dns message exchange layers.
    - MEDIUM: resolvers/dns: split dns.c into dns.c and resolvers.c
    - MEDIUM: dns: adds code to support pipelined DNS requests over TCP.
    - MEDIUM: resolvers: add supports of TCP nameservers in resolvers.
 2021-02-13 10:17:27 +01:00
Emeric Brun
56fc5d9ebc MEDIUM: resolvers: add supports of TCP nameservers in resolvers. 
This patch introduce the new line "server" to set a TCP
nameserver in a "resolvers" section:

server <name> <address> [param*]
  Used to configure a DNS TCP or stream server. This supports for all
  "server" parameters found in 5.2 paragraph. Some of these parameters
  are irrelevant for DNS resolving. Note: currently 4 queries are pipelined
  on the same connections. A batch of idle connections are removed every
  5 seconds. "maxconn" can be configured to limit the amount of those
  concurrent connections and TLS should also usable if the server supports
. The current implementation limits to 4 pipelined

The name of the line in configuration is open to discussion
and could be changed before the next release.
 2021-02-13 10:03:46 +01:00
William Dauchy
9a4bbfe151 DOC: tune: explain the origin of block size for ssl.cachesize 
A user could eventually ask himself where those 200 bytes block size are
coming from. This patch tries to better explain the origin in case
people are curious or want to double check the reality.

Signed-off-by: William Dauchy <wdauchy@gmail.com>
 2021-02-13 09:00:28 +01:00
Willy Tarreau
4b10302fd8 MINOR: cfgparse: implement a simple if/elif/else/endif macro block handler 
Very often, especially since reg-tests, it would be desirable to be able
to conditionally comment out a config block, such as removing an SSL
binding when SSL is disabled, or enabling HTX only for certain versions,
etc.

This patch introduces a very simple nested block management which takes
".if", ".elif", ".else" and ".endif" directives to take or ignore a block.

For now the conditions are limited to empty string or "0" for false versus
a non-nul integer for true, which already suffices to test environment
variables. Still, it needs to be a bit more advanced with defines, versions
etc.

A set of ".notice", ".warning" and ".alert" statements are provided to
emit messages, often in order to provide advice about how to fix certain
conditions.
 2021-02-12 18:54:19 +01:00
Yves Lafon
b4d3708cb7 MINOR: http: add baseq sample fetch 
Symetrical to path/pathq, baseq returns the concatenation of
the Host header and the path including the query string.
 2021-02-12 16:38:50 +01:00
Willy Tarreau
7c0b4d861e MEDIUM: cfgparse: allow a proxy to designate the defaults section to use 
Now it becomes possible to specify "from foo" on a frontend/listen/backend
or even on a "defaults" line, to mention that defaults section "foo" needs
to be used to preset the proxy's settings.

When not set, the last section remains used. In case the designated name
is found at multiple places, it is rejected and an error indicates two
occurrences of the same name. Similarly, if the section name is found,
its name must only use valid characters. This allows multiple named
defaults section to continue to coexist without the risk that they will
cause trouble by accident.

When it comes to "defaults" relying on another defaults, what happens is
just that a new defaults section is created from the designated one. This
will make it possible for example to reuse some settings such as log-format
like below:

    defaults tcp-clear
        log stdout local0 info
        log-format "%ci:%cp/%b/%si:%sp %ST %ts %U/%B %{+Q}r"

    defaults tcp-ssl
        log stdout local0 info
        log-format "%ci:%cp/%b/%si:%sp %ST %ts %U/%B %{+Q}r ssl=%sslv"

    defaults http-clear from tcp-clear
        mode http

    defaults http-ssl from tcp-ssl
        mode http

    frontend fe1 from http-clear
        bind :8001

    frontend fe2 from http-ssl
        bind :8002

A small corner case remains in the error detection, if a second defaults
section appears with the same name after the point where it was used, and
nobody references it, the duplicate will not be detected. This could be
addressed by performing the syntactic checks in check_config_validity(),
and by postponing the freeing of the defaults, after tagging a defaults
section as explicitly looked up by another section. This doesn't seem
that important at the moment though.
 2021-02-12 16:23:46 +01:00
Christopher Faulet
583b6de68a BUG/MINOR: server: Fix server-state-file-name directive 
Since the beginning, this directive is documented to accept an optional file
name. But it should also be possible to use it without any argument to use
the backend name as file name. However, when no argument is provided, an
error is reported during the configuration parsing requesting an argument, a
file name or "use-backend-name". And This last special argument is not
documented.

So, to respect the documentation and to avoid configuration breakages, all
modes are now supported. If this directive is called with no argument or
with "use-backend-name", the backend name is use as file name for the
server-state file. Otherwise, the provided string is used.

In addition, we take care to release any previously allocated file name in
case this directive is defines multiple times in the same backend. And an
error is reported if more than one argument are defined. Finally, the
documentation is updated accordingly. Sections supporting this directive are
also mentioned.

This patch should be backported as far as 1.6.
 2021-02-12 16:04:52 +01:00
Amaury Denoyelle
d773a4ee23 MINOR: doc: update http reuse for new eligilible connections 
Update the doc to remove entries on http-reuse marking private
connection for specific source address or sni.
 2021-02-12 12:54:04 +01:00
Willy Tarreau
826f3ab5e6 MINOR: stick-tables/counters: add http_fail_cnt and http_fail_rate data types 
Historically we've been counting lots of client-triggered events in stick
tables to help detect misbehaving ones, but we've been missing the same on
the server side, and there's been repeated requests for being able to count
the server errors per URL in order to precisely monitor the quality of
service or even to avoid routing requests to certain dead services, which
is also called "circuit breaking" nowadays.

This commit introduces http_fail_cnt and http_fail_rate, which work like
http_err_cnt and http_err_rate in that they respectively count events and
their frequency, but they only consider server-side issues such as network
errors, unparsable and truncated responses, and 5xx status codes other
than 501 and 505 (since these ones are usually triggered by the client).
Note that retryable errors are purposely not accounted for, so that only
what the client really sees is considered.

With this it becomes very simple to put some protective measures in place
to perform a redirect or return an excuse page when the error rate goes
beyond a certain threshold for a given URL, and give more chances to the
server to recover from this condition. Typically it could look like this
to bypass a URL causing more than 10 requests per second:

  stick-table type string len 80 size 4k expire 1m store http_fail_rate(1m)
  http-request track-sc0 base       # track host+path, ignore query string
  http-request return status 503 content-type text/html \
      lf-file excuse.html if { sc0_http_fail_rate gt 10 }

A more advanced mechanism using gpt0 could even implement high/low rates
to disable/enable the service.

Reg-test converteers_ref_cnt_never_dec.vtc was updated to test it.
 2021-02-10 12:27:01 +01:00
Ilya Shipitsin
acf84595a7 CLEANUP: assorted typo fixes in the code and comments 
This is 17th iteration of typo fixes
 2021-02-08 10:49:08 +01:00
Willy Tarreau
5d46fbddb1 [RELEASE] Released version 2.4-dev7 
Released version 2.4-dev7 with the following main changes :
    - BUG/MINOR: stats: Continue to fill frontend stats on unimplemented metric
    - BUILD: ssl: guard Client Hello callbacks with HAVE_SSL_CLIENT_HELLO_CB macro instead of openssl version
    - BUG/MINOR: stats: Init the metric variable when frontend stats are filled
    - MINOR: contrib/prometheus-exporter: better output of Not-a-Number
    - CLEANUP: stats: improve field selection for frontend http fields
    - CLEANUP: assorted typo fixes in the code and comments
    - DOC: Improve documentation of the various hdr() fetches
    - MEDIUM: stats: allow to select one field in `stats_fill_be_stats`
    - MINOR: contrib/prometheus-exporter: use fill_be_stats for backend dump
    - MEDIUM: stats: allow to select one field in `stats_fill_sv_stats`
    - MINOR: contrib/prometheus-exporter: use fill_sv_stats for server dump
    - MINOR: abort() on my_unreachable() when DEBUG_USE_ABORT is set.
    - BUG/MEDIUM: filters/htx: Fix data forwarding when payload length is unknown
    - BUG/MINOR: config: fix leak on proxy.conn_src.bind_hdr_name
    - MINOR: reg-tests: add http-reuse test
    - CLEANUP: srv: fix comment for pool-max-conn
    - CLEANUP: backend: remove an obsolete comment on conn_backend_get
    - REORG: backend: simplify conn_backend_get
    - MINOR: ssl: Server ssl context prepare function refactoring
    - MINOR: ssl: Certificate chain loading refactorization
    - MEDIUM: ssl: Load client certificates in a ckch for backend servers
    - MEDIUM: ssl: Enable backend certificate hot update
    - MINOR: ssl: Remove client_crt member of the server's ssl context
    - CLEANUP: ssl/cli: rework free in cli_io_handler_commit_cert()
    - CLEANUP: ssl: remove SSL_CTX function parameter
    - CLEANUP: ssl: make load_srv_{ckchs,cert} match their bind counterpart
    - BUILD: Include stdlib.h in compiler.h if DEBUG_USE_ABORT is set
    - CI: Fix DEBUG_STRICT definition for Coverity
    - BUG/MINOR: stats: Remove a break preventing ST_F_QCUR to be set for servers
    - BUG/MINOR: stats: Add a break after filling ST_F_MODE field for servers
    - CLEANUP: ssl: remove dead code in ckch_inst_new_load_srv_store()
    - BUG/MINOR: ssl: init tmp chunk correctly in ssl_sock_load_sctl_from_file()
    - BUG/MEDIUM: session: only retrieve ready idle conn from session
    - BUG/MEDIUM: backend: never reuse a connection for tcp mode
    - REGTESTS: set_ssl_server_cert.vtc: remove the abort command
    - REGTESTS: set_ssl_server_cert.vtc: check the Sha1 Fingerprint
    - REGTESTS: set_ssl_server_cert.vtc: check the sha1 from the server
    - MEDIUM: stream-int: Take care of EOS if the SI wake callback function
    - MINOR: mux-h1: Try to wake up data layer first before calling its wake callback
    - MINOR: mux-h1: Wake up H1C after its creation if input buffer is not empty
    - MEDIUM: mux-h1: Add ST_READY state for the H1 connections
    - MINOR: stream: Add a function to validate TCP to H1 upgrades
    - MEDIUM: http-ana: Do nothing in wait-for-request analyzer if not htx
    - BUG/MEDIUM: stream: Don't immediatly ack the TCP to H1 upgrades
    - BUG/MAJOR: mux-h1: Properly handle TCP to H1 upgrades
    - MINOR: htx/http-ana: Save info about Upgrade option in the Connection header
    - MEDIUM: http-ana: Refuse invalid 101-switching-protocols responses
    - BUG/MINOR: h2/mux-h2: Reject 101 responses with a PROTOCOL_ERROR h2s error
    - MINOR: mux-h1/mux-fcgi: Don't set TUNNEL mode if payload length is unknown
    - MINOR: mux-h1: Split H1C_F_WAIT_OPPOSITE flag to separate input/output sides
    - MINOR: mux-h2: Add 2 flags to help to properly handle tunnel mode
    - MEDIUM: mux-h2: Block client data on server side waiting tunnel establishment
    - MEDIUM: mux-h2: Close streams when processing data for an aborted tunnel
    - MEDIUM: mux-h1: Properly handle tunnel establishments and aborts
    - BUG/MAJOR: mux-h1/mux-h2/htx: Fix HTTP tunnel management at the mux level
    - MINOR: htx: Rename HTX_FL_EOI flag into HTX_FL_EOM
    - REGTESTS: Don't run http_msg_full_on_eom script on the 2.4 anymore
    - MINOR: htx: Add a function to know if a block is the only one in a message
    - MAJOR: htx: Remove the EOM block type and use HTX_FL_EOM instead
    - MINOR: mux-h1: Add a flag on H1 streams with a response known to be bodyless
    - MEDIUM: mux-h1: Don't emit any payload for bodyless responses
    - MINOR: mux-h1: Don't emit C-L and T-E headers for 204 and 1xx responses
    - MINOR: mux-h1: Don't add Connection close/keep-alive header for 1xx messages
    - MINOR: h2/mux-h2: Add flags to notify the response is known to have no body
    - MEDIUM: mux-h2: Don't emit DATA frame for bodyless responses
    - MEDIUM: http-ana: Deal with L7 retries in HTTP analysers
    - MINOR: h1: reject websocket handshake if missing key
    - MEDIUM: h1: generate WebSocket key on response if needed
    - MINOR: mux_h2: define H2_SF_EXT_CONNECT_SENT stream flag
    - MEDIUM: h2: parse Extended CONNECT reponse to htx
    - MEDIUM: mux_h2: generate Extended CONNECT from htx upgrade
    - MEDIUM: h1: add a WebSocket key on handshake if needed
    - MEDIUM: mux_h2: generate Extended CONNECT response
    - MEDIUM: h2: parse Extended CONNECT request to htx
    - MEDIUM: h2: send connect protocol h2 settings
    - MINOR: vtc: add test for h1/h2 protocol upgrade translation
    - MINOR: vtc: add websocket test
    - REGTESTS: Fix required versions for several scripts
    - REGTEST: Don't use the websocket to validate http-check
    - MINOR: mux-h1/trace: add traces at level ERROR for all kind of errors
    - MINOR: mux-fcgi/trace: add traces at level ERROR for all kind of errors
    - MINOR: h1: Raise the chunk size limit up to (2^52 - 1)
    - BUG/MEDIUM: listener: do not accept connections faster than we can process them
    - REGTESTS: set_ssl_server_cert.vtc: set as broken
    - Revert "BUG/MEDIUM: listener: do not accept connections faster than we can process them"
    - BUG/MINOR: backend: check available list allocation for reuse
    - CI: Fix the coverity builds
    - DOC: management: fix "show resolvers" alphabetical ordering
    - MINOR: tools: add print_time_short() to print a condensed duration value
    - MINOR: activity: make profiling more manageable
    - MINOR: activity: declare a new structure to collect per-function activity
    - MEDIUM: tasks/activity: collect per-task statistics when profiling is enabled
    - MINOR: activity: also report collected tasks stats in "show profiling"
    - MINOR: activity: flush scheduler stats on "set profiling tasks on"
    - MINOR: activity: add a new "show tasks" command to list currently active tasks
    - MINOR: listener: export accept_queue_process
    - MINOR: session: export session_expire_embryonic()
    - MINOR: muxes: export the timeout and shutr task handlers
    - MINOR: checks: export a few functions that appear often in trace dumps
    - MINOR: peers: export process_peer_sync() to improve traces
    - MINOR: stick-tables: export process_table_expire()
    - MINOR: mux-h1: Remove first useless test on count in h1_process_output()
    - BUG/MINOR: stick-table: Always call smp_fetch_src() with a valid arg list
    - MINOR: http-fetch: Don't check if argument list is set in sample fetches
    - MINOR: http-conv: Don't check if argument list is set in sample converters
    - MINOR: sample: Don't check if argument list is set in sample fetches
    - MINOR: ssl-sample: Don't check if argument list is set in sample fetches
    - MINOR: mux-h2: Don't tests the start-line when sending HEADERS frame
    - MINOR: mux-h2: Slightly improve request HEADERS frames sending
    - MINOR: contrib/prometheus-exporter: declare states for objects
    - MAJOR: contrib/prometheus-exporter: move ftd/bkd/srv states to labels
    - MEDIUM: contrib/prometheus-exporter: Use dynamic labels instead of static ones
    - MINOR: listener: export manage_global_listener_queue()
    - BUG/MINOR: activity: take care of late wakeups in "show tasks"
    - REGTESTS: set_ssl_server_cert.vtc: remove SSL caching and set as working
    - REGTESTS: set_ssl_server_cert: cleanup the SSL caching option
    - MINOR: checks: Add function to get the result code corresponding to a status
    - MAJOR: contrib/prometheus-exporter: move health check status to labels
    - MINOR: contrib/prometheus-exporter: improve service status description field
    - MINOR: stats: improve pending connections description
    - MINOR: stats: improve max stats descriptions
    - MINOR: contrib/prometheus-exporter: use stats desc when possible
    - MINOR: contrib/prometheus-exporter: add uweight field
    - MINOR: contrib/prometheus-exporter: add recv logs_logs_total field
    - CLEANUP: contrib/prometheus-exporter: remove unused includes
    - CLEANUP: contrib/prometheus-exporter: align and reorder fields
    - CLEANUP: contrib/prometheus-exporter: remove description in README
    - DOC: contrib/prometheus-exporter: Add missing metrics in README
    - BUG/MINOR: contrib/prometheus-exporter: Add missing label for ST_F_HRSP_1XX
    - BUG/MINOR: contrib/prometheus-exporter: Restart labels dump at the right pos
    - BUG/MEDIUM: ssl/cli: abort ssl cert is freeing the old store
    - BUG/MEDIUM: ssl: check a connection's status before computing a handshake
    - BUG/MINOR: mux_h2: fix incorrect stat titles
    - MINOR: ssl/cli: flush the server session cache upon 'commit ssl cert'
    - BUG/MINOR: cli: fix set server addr/port coherency with health checks
    - MINOR: server: Don't set the check port during the update from a state file
    - MINOR: dns: Don't set the check port during a server dns resolution
    - MEDIUM: check: remove checkport checkaddr flag
    - MEDIUM: server: adding support for check_port in server state
    - BUG/MINOR: check: consitent way to set agentaddr
    - MEDIUM: check: align agentaddr and agentport behaviour
    - DOC: server: Add missing params in comment of the server state line parsing
    - BUG/MINOR: xxhash: make sure armv6 uses memcpy()
    - REGTESTS: mark http-check-send.vtc as 2.4-only
    - REGTESTS: mark sample_fetches/hashes.vtc as 2.4-only
    - BUG/MINOR: ssl: do not try to use early data if not configured
    - REGTESTS: unbreak http-check-send.vtc
    - MINOR: cli/show_fd: report local and report ports when known
    - BUILD: Makefile: move REGTESTST_TYPE default setting
    - BUG/MEDIUM: mux-h2: handle remaining read0 cases
    - CLEANUP: http-htx: Set buffer area to NULL instead of malloc(0)
    - BUG/MINOR: sock: Unclosed fd in case of connection allocation failure
    - BUG/MEDIUM: mux-h2: do not quit the demux loop before setting END_REACHED
 2021-02-05 15:17:33 +01:00
William Dauchy
4858fb2e18 MEDIUM: check: align agentaddr and agentport behaviour 
in the same manner of agentaddr, we now:
- permit to set agentport through `port` keyword, like it is the case
  for agentaddr through `addr`
- set the priority on `agent-port` keyword when used
- add a flag to be able to test when the value is set like for agentaddr

it makes the behaviour between `addr` and `port` more consistent.

Signed-off-by: William Dauchy <wdauchy@gmail.com>
 2021-02-04 14:00:38 +01:00
Christopher Faulet
d1ac2b90cd MAJOR: htx: Remove the EOM block type and use HTX_FL_EOM instead 
The EOM block may be removed. The HTX_FL_EOM flags is enough. Most of time,
to know if the end of the message is reached, we just need to have an empty
HTX message with HTX_FL_EOM flag set. It may also be detected when the last
block of a message with HTX_FL_EOM flag is manipulated.

Removing EOM blocks simplifies the HTX message filling. Indeed, there is no
more edge problems when the message ends but there is no more space to write
the EOM block. However, some part are more tricky. Especially the
compression filter or the FCGI mux. The compression filter must finish the
compression on the last DATA block. Before it was performed on the EOM
block, an extra DATA block with the checksum was added. Now, we must detect
the last DATA block to be sure to finish the compression. The FCGI mux on
its part must be sure to reserve the space for the empty STDIN record on the
last DATA block while this record was inserted on the EOM block.

The H2 multiplexer is probably the part that benefits the most from this
change. Indeed, it is now fairly easier to known when to set the ES flag.

The HTX documentaion has been updated accordingly.
 2021-01-28 16:37:14 +01:00
Tim Duesterhus
27c70ae23c DOC: Improve documentation of the various hdr() fetches 
GitHub issue #796 notes that many administrators miss the fact that the `hdr()`
fetch (without the `f`) splits the header value at commas. This is only
mentioned at the end of a long paragraph.

This patch attempts to improve the documentation by:
- Explaning the "comma issue" as early as possible.
- Adding newlines to split the explanation into distinct sections.
- Reducing duplication by making the `res` siblings refer to their `req`
  counterparts.

This patch may be backported as long as it applies cleanly. During the
refactoring I needed to adjust several explanations for consistency and not all
of them might be available in older branches.
 2021-01-26 09:22:43 +01:00
Willy Tarreau
24c41d55bd [RELEASE] Released version 2.4-dev6 
Released version 2.4-dev6 with the following main changes :
    - MINOR: converter: adding support for url_enc
    - BUILD: SSL: guard TLS13 ciphersuites with HAVE_SSL_CTX_SET_CIPHERSUITES
    - BUILD: ssl: guard EVP_PKEY_get_default_digest_nid with ASN1_PKEY_CTRL_DEFAULT_MD_NID
    - BUILD: ssl: guard openssl specific with SSL_READ_EARLY_DATA_SUCCESS
    - BUILD: Makefile: exclude broken tests by default
    - CLEANUP: cfgparse: replace "realloc" with "my_realloc2" to fix to memory leak on error
    - BUG/MINOR: hlua: Fix memory leak in hlua_alloc
    - MINOR: contrib/prometheus-exporter: export build_info
    - DOC: fix some spelling issues over multiple files
    - CLEANUP: Fix spelling errors in comments
    - SCRIPTS: announce-release: fix typo in help message
    - CI: github: add a few more words to the codespell ignore list
    - DOC: Add maintainers for the Prometheus exporter
    - BUG/MINOR: sample: fix concat() converter's corruption with non-string variables
    - BUG/MINOR: server: Memory leak of proxy.used_server_addr during deinit
    - CLEANUP: sample: remove uneeded check in json validation
    - MINOR: reg-tests: add a way to add service dependency
    - BUG/MINOR: sample: check alloc_trash_chunk return value in concat()
    - BUG/MINOR: reg-tests: fix service dependency script
    - MINOR: reg-tests: add base prometheus test
    - Revert "BUG/MINOR: dns: SRV records ignores duplicated AR records"
    - BUG/MINOR: sample: Memory leak of sample_expr structure in case of error
    - BUG/MINOR: check: Don't perform any check on servers defined in a frontend
    - BUG/MINOR: init: enforce strict-limits when using master-worker
    - MINOR: contrib/prometheus-exporter: avoid connection close header
    - MINOR: contrib/prometheus-exporter: use fill_info for process dump
    - BUG/MINOR: init: Use a dynamic buffer to set HAPROXY_CFGFILES env variable
    - MINOR: config: Add failifnotcap() to emit an alert on proxy capabilities
    - MINOR: server: Forbid server definitions in frontend sections
    - BUG/MINOR: threads: Fixes the number of possible cpus report for Mac.
    - CLEANUP: pattern: rename pat_ref_commit() to pat_ref_commit_elt()
    - MINOR: pattern: add the missing generation ID manipulation functions
    - MINOR: peers: Add traces for peer control messages.
    - BUG/MINOR: dns: SRV records ignores duplicated AR records (v2)
    - BUILD: peers: fix build warning about unused variable
    - BUG/MEDIUM: stats: add missing INF_BUILD_INFO definition
    - MINOR: cache: Do not store responses with an unknown encoding
    - BUG/MINOR: peers: Possible appctx pointer dereference.
    - MINOR: build: discard echoing in help target
    - MINOR: cache: Remove the `hash` part of the accept-encoding secondary key
    - CLEANUP: cache: Use proper data types in secondary_key_cmp()
    - CLEANUP: Rename accept_encoding_hash_cmp to accept_encoding_bitmap_cmp
    - BUG/MINOR: peers: Wrong "new_conn" value for "show peers" CLI command.
    - MINOR: contrib: Make the wireshark peers dissector compile for more distribs.
    - BUG/MINOR: mux_h2: missing space between "st" and ".flg" in the "show fd" helper
    - CLEANUP: tools: make resolve_sym_name() take a const pointer
    - CLEANUP: cli: make "show fd" use a const connection to access other fields
    - MINOR: cli: make "show fd" also report the xprt and xprt_ctx
    - MINOR: xprt: add a new show_fd() helper to complete some "show fd" dumps.
    - MINOR: ssl: provide a "show fd" helper to report important SSL information
    - MINOR: xprt/mux: export all *_io_cb functions so that "show fd" resolves them
    - MINOR: mux-h2: make the "show fd" helper also decode the h2s subscriber when known
    - MINOR: mux-h1: make the "show fd" helper also decode the h1s subscriber when known
    - MINOR: mux-fcgi: make the "show fd" helper also decode the fstrm subscriber when known
    - CI: Pin VTest to a known good commit
    - MINOR: cli: give the show_fd helpers the ability to report a suspicious entry
    - MINOR: cli/show_fd: report some easily detectable suspicious states
    - MINOR: ssl/show_fd: report some FDs as suspicious when possible
    - MINOR: mux-h2/show_fd: report as suspicious an entry with too many calls
    - MINOR: mux-h1/show_fd: report as suspicious an entry with too many calls
    - BUG/MINOR: mworker: define _GNU_SOURCE for strsignal()
    - BUG/MEDIUM: tcpcheck: Don't destroy connection in the wake callback context
    - BUG/MEDIUM: mux-h2: Xfer rxbuf to the upper layer when creating a front stream
    - MINOR: http: Add HTTP 501-not-implemented error message
    - MINOR: muxes: Add exit status for errors about not implemented features
    - MINOR: mux-h1: Be prepared to return 501-not-implemented error during parsing
    - MEDIUM: mux-h1: Return a 501-not-implemented for upgrade requests with a body
    - DOC: Remove space after comma in converter signature
    - DOC: Rename '<var name>' to '<var>' in converter signature
    - MINOR: stats: duplicate 3 fields in bytes in info
    - MINOR: stats: add new start time field
    - MINOR: contrib/prometheus-exporter: merge info description from stats
    - MEDIUM: stats: allow to select one field in `stats_fill_fe_stats`
    - MINOR: contrib/prometheus-exporter: use fill_fe_stats for frontend dump
    - MINOR: contrib/prometheus-exporter: Don't needlessly set empty label for metrics
    - MINOR: contrib/prometheus-exporter: Split the PROMEX_FL_STATS_METRIC flag
    - MINOR: contrib/prometheus-exporter: Add promex_metric struct defining a metric
    - MEDIUM: contrib/prometheus-exporter: Rework matrices defining Promex metrics
    - BUG/MINOR: stream: Don't update counters when TCP to H2 upgrades are performed
    - BUG/MEDIUM: mux-h2: fix read0 handling on partial frames
    - MINOR: debug: always export the my_backtrace function
    - MINOR: debug: extract the backtrace dumping code to its own function
    - MINOR: debug: create ha_backtrace_to_stderr() to dump an instant backtrace
    - MEDIUM: debug: now always print a backtrace on CRASH_NOW() and friends
    - MINOR: debug: let ha_dump_backtrace() dump a bit further for some callers
    - BUILD: debug: fix build warning by consuming the write() result
    - MINOR: lua: remove unused variable
    - BUILD/MINOR: lua: define _GNU_SOURCE for LLONG_MAX
 2021-01-22 16:19:46 +01:00
Tim Duesterhus
ef4e45ca55 DOC: Rename '<var name>' to '<var>' in converter signature 
The space appears to trip up the dconv parser and `<var>` is used for other
converters.
 2021-01-21 17:59:34 +01:00
Tim Duesterhus
a308209241 DOC: Remove space after comma in converter signature 
This space appears to trip up the dconv parser and is inconsistent with
other converts.
 2021-01-21 17:59:34 +01:00
Christopher Faulet
e095f31d36 MINOR: http: Add HTTP 501-not-implemented error message 
Add the support for the 501-not-implemented status code with the
corresponding default message. The documentation is updated accordingly
because it is now part of status codes HAProxy may emit via an errorfile or
a deny/return HTTP action.
 2021-01-21 15:21:12 +01:00
Remi Tricot-Le Breton
6ca89162dc MINOR: cache: Do not store responses with an unknown encoding 
If a server varies on the accept-encoding header and it sends a response
with an encoding we do not know (see parse_encoding_value function), we
will not store it. This will prevent unexpected errors caused by
cache collisions that could happen in accept_encoding_hash_cmp.
 2021-01-15 22:33:05 +01:00
Thayne McCombs
cdbcca9995 DOC: fix some spelling issues over multiple files 
This is from the output of codespell and may be backported.
 2021-01-08 14:53:47 +01:00