There was a mixup between read/send events and ability for an applet to
receive and send. The fix seems obvious by reading it. The call-rate must be
incremented when nothing was received from the applet while it was allowed
and nothing was sent to the applet while it was allowed.
This patch must be backported as far as 3.0.
The computed maxconn was only displayed in verbose or debug modes. This
is too bad because lots of users just don't know what they're starting
with and can be trapped when an environment changes. Let's use ha_notice()
instead of a conditional fprintf() so that it gets displayed right after
the other startup messages, hoping that users will get used to seeing it
and more easily spot anomalies. See github issue #3191 for more context.
The fix in commit 9481cef948 ("BUG/MEDIUM: connection: do not reinsert a
purgeable conn in idle list") is also needed for ssl_sock_io_cb() which
can also release an idle connection and must perform the same checks.
This fix must be backported to all stable versions containing the fix
above.
Let the --dry-run mode connect to the socket and compare the
certificates. It would exits the process just before trying to move
the previous certificate and replace it.
This allow to have the "[NOTICE] (1234) XXX is already up to date" message
with dry-run.
Connection struct defines an handle which can point to either a FD or a
quic_conn. On the latter case, CO_FL_FDLESS must be set. This is already
the case on frontend side.
This patch fixes QUIC backend support. Before setting connection handle
member to a quic_conn instance, ensure that CO_FL_FDLESS flag is set on
the connection.
Prior to this patch, crash can occur in "show sess all".
No need to backport.
quic_conn has a local_addr member which is used to store the connection
source address. On backend side, this member is initialized to NULL as
the address is not yet known prior to connect. With this patch,
quic_connect_server() is extended so that local_addr is updated after
connect() success.
Also, quic_sock_get_src() is completed for the backend side which now
returns local_addr member. This step is necessary to properly support
fetches bc_src/bc_src_port.
If a mux protocol is forced and an incompatible ALPN or NPN settings are
used, connection errors may be experienced. There is no check performed
during HAProxy startup and It is not necessarily obvious. So a note is added
to warn users about this usage.
Since the commit 5003ac7fe ("MEDIUM: config: set useful ALPN defaults for
HTTPS and QUIC"), the ALPN is set by default to "h2,http/1.1" for HTTPS
listeners. However, it is in conflict with the forced mux protocol, if
any. Indeed, with "proto" keyword, the mux can be forced. In that case, some
combinations with the default ALPN will triggers connections errors.
For instance, by setting "proto h2", it will not be possible to use the H1
multiplexer. So we must take care to not advertise it in the ALPN. Worse,
since the commit above, most modern HTTP clients will try to use the H2
because it is advertised in the ALPN. By setting "proto h1" on the bind line
will make all the traffic rejected in error.
To fix the issue, and thanks to previous commits, if it is defined, we are
now relying on the ALPN defined by the mux protocol by default. The H1
multiplexer (only the one that can be forced) defines it to "http/1.1" while
the H2 multiplexer defines it to "h2". So by default, if one or another of
these muxes is forced, and if no ALPN is set, the mux ALPN is used.
Other multiplexers are not defining any default ALPN for now, because it is
useless. In addition, only the listeners are concerned because there is no
default ALPN on the server side.Finally, there is no tests performed if the
ALPN is forced on the bind line. It is the user responsibility to properly
configure his listeners (at least for now).
This patch depends on:
* MINOR: config: Do proto detection for listeners before checks about ALPN
* MINOR: muxes: Support an optional ALPN string when defining mux protocols
The series must be backported as far as 2.8.
The verification of any forced mux protocol, via the "proto" keyword, for
listeners is now performed before any tests on the ALPN. It will be
mandatory to be able to force the default ALPN, if not forced on the bind
line.
This patch will be mandatory for the next fix.
When a multiplexer protocol is defined, it is now possible to specify the
ALPN it supports, in binary format. This info is optionnal. For now only the
h2 and the h1 multiplexers define an ALPN because this will be mandatory for
a fix. But this could be used in future for different purpose.
This patch will be mandatory for the next fix.
In assign_server_and_queue(), there's a rare case when the server was
full, so we created a pendconn, another server was considered but in the
meanwhile the pendconn was unqueued already, so we just left the
function. We did so, however, while still holding the queue lock, which
will ultimately lead to a deadlock, and ultimately the watchdog would
kill the process.
To fix that, just unlock the queue before leaving.
This should be backported to 3.2.
When configuring an acme section with the 'map' keyword, the user must
use an existing map. If the map doesn't exist, a log will be emitted
when trying to add the challenge to the map.
This patch change the behavior by checking at startup if the map exists,
so haproxy would warn and won't start with a non-existing map.
This must be backported in 3.2.
Contrary to TCP, QUIC does not SSL_free() its SSL * object when its ->close()
XPRT callback is called. This has as side effect to trigger some BUG_ON(!conn)
with <conn> the connection from TLS callbacks registered at configuration
parsing time, so after this <conn> have been released.
This is the case for instance with ssl_sock_srv_verifycbk() whose role is to
add some checks to the built-in server certificate verification process.
This patch prevents the pointer to <conn> dereferencing inside several callbacks
shared between TCP and QUIC.
Thank you to @InputOutputZ for its report in GH #3188.
As the QUIC backend feature arrived with the current 3.3 dev, no need to backport.
As shown in github issue #3191, the error message shown when FD limits
are exceeded is not very useful as-is, since the current hard limit is
not displayed, and no suggestion is made about what to change in the
config. Let's explain about maxconn/ulimit-n/fd-hard-limit, suggest
dropping them or setting them to a context-based value at roughly 49%
of the current limit minus the known used FDs for listeners and checks.
This allows common "large" hard limits to report mostly round maxconns.
Example:
[ALERT] (25330) : [haproxy.main()] Cannot raise FD limit to 4001020,
current limit is 1024 and hard limit is 4096. You may prefer to let
HAProxy adjust the limit by itself; for this, please just drop any
'maxconn' and 'ulimit-n' from the global section, and possibly add
'fd-hard-limit' lower than this hard limit. You may also force a new
'maxconn' value that is a bit lower than half of the hard limit minus
listeners and checks. This results in roughly 1500 here.
It's always a pain to guess the number of FDs that can be needed by
listeners, checks, threads, pollers etc. We have this estimate in
global.maxsock before calling set_global_maxconn(), but we lose it
the line after. Let's copy it into global.est_fd_usage and keep it.
This will be helpful to try to provide more accurate suggestions for
maxconn.
This quic_conn ->xrpt_ctx is passed to qc_send_ppkts(), the quic_conn is retrieved
from this context to be used inside this function and it is not used at all
by this function.
This patch simply directly passes the quic_conn to qc_send_ppkts(). This is only
what this function needs.
This patch completes the 0-rtt test to verify that early-data:1 is
properly emitted to the server in the relevant situations. We carefully
compare it with the expected values that are computed based on the TLS
version, the client and listener's support for 0-rtt and the resumption
status. A response header "x-early-data-test" is set to OK on success,
or KO on failure and the client tests this. The previous test is kept
as well. This was tested with quictls-1.1.1 and quictls-3.0.1 for TCP,
as well as aws-lc for QUIC.
These tests try all the combinations of {0,1}rtt <-> {0,1}rtt with
stateless and stateful tickets. They take into consideration the TLS
version to decide whether or not 0rtt should work. Since we cannot
use environment variables in the client, the tests are run in haproxy
itself where the frontends set a "x-early-rcvd-test" response header
that the client checks. At this stage, the test only verifies that
*some* early data were received.
Note that the tests are a bit complex because we need 4 listeners
for the various combinations of 0rtt/tickets, then we have to set
expectations based on the TLS version (1.2 vs 1.3), as well as the
session resumption status.
We have to set alpn on the server lines because currently our frontends
expect it for 0-rtt to work.
The dependency to halog build provokes problems when changing CFLAGS and
LDFLAGS, because you're suppose to have the same flags during the build
and the install if there's still some things to build.
We probably need to store the flags somewhere to reuse them at another
step, but we need to do it cleanly. In the meantime it's better not to
have this dependency.
On the frontend side, QUIC transfer can be performed either via a
connection owned FD or multiplex on the listener one. When a quic_conn
is freed and converted to quic_conn_closed instance, its FD if open is
closed and all exchanges are now multiplex via the listener FD.
This is different for the backend as connections only has the choice to
use their owned FD. Thus, special care care must be taken when freeing a
connection and converting it to a quic_conn_closed instance. In this
case, qc_release_fd() is delayed to the quic_conn_closed release.
Furthermore, when the FD is transferred, its iocb and owner fields are
updated to the new quic_conn_closed instance. Without it, a crash will
occur when accessing the freed quic_conn tasklet. A newly dedicated
handler quic_conn_closed_sock_fd_iocb is used to ensure access to
quic_conn_closed members only.
jobs is a global counter which serves to account activity through the
whole process. Soft-stop procedure will wait until this counter is
resetted to the nul value.
jobs is not used for backend connections. Thus, it is not incremented
when a QUIC backend connection is instantiated as expected. However,
decrement is performed on all sides during quic_conn_release(). This
causes the counter wrapping.
Fix this by decrementing jobs only for frontend connections. Without
this patch, soft stop procedure will hang indefinitely if QUIC backend
connections were in use.
Properly implement support for max-reuse server keyword. This is done by
adding a total count of streams seen for the whole connection. This
value is used in avail_streams callback.
When haproxy is compiled in -O0, the SSL_get_max_early_data() symbol is
used in the generated assembly, however -O2 seems to remove this symbol
when optimizing the code.
It happens because `if conn_is_back(conn)` and `if
(objt_listener(conn->target))` are opposed conditions, which mean we
never use the branch when objt_listener(conn->target) is true.
This patch removes the dead code. Bonus: SSL_get_max_early_data() is not
implemented in rustls, and that's the only thing preventing to start
with it.
This can be backported in every stable branches.
make install now have a dependency to install-admin which have a
dependency to admin/halog/halog.
halog links haproxy .o together with its own objects, but those objects
when built with ASAN must also be linked with ASAN or it won't be
possible to link the binary.
We don't need an ASAN-ready halog, so let's just do an install-bin
instead that will just install haproxy.
QUIC and TLS don't use the same tests because QUIC only supports
TLS 1.3 while SSL tests both TLS 1.2 and 1.3, which complicates
the tests scenarios.
This change extracts the core of the test into a single generic
ssl_reuse.vtci file and creates new high-level tests for TLSv1.2
over TCP, TLSv1.3 over TCP and TLSv1.3 over QUIC, which simply
include this file and set two variables. The test is now cleaner
and simpler.
When trying to use the P-256 curve in the acme configuration with
OpenSSL 3.x, the generation of the account was failing because OpenSSL
doesn't return a NIST or SECG curve name, but a ANSI X9.62 one.
Since the ANSI X9.62 curve names were not in the list, it couldn't match
anything supported.
This patch fixes the issue by adding both prime192v1 and prime256v1 name
in the struct curve array which is used during curve parsing.
Must be backported to 3.2.
Since the new master-worker model in 3.1, signals are registered in
step_init_3(). However, those signals were supposed to be registered
only for the worker or the standalone mode. It would call the wrong
callback in the master even during configuration parsing.
The patch set the signals handler to NULL for the master so it does
nothing until they really are registered.
Must be backported as far as 3.1.
Since haproxy 3.1, the master-worker mode changed to let the worker
parse the configuration instead of the master.
Previously, signals were blocked during configuration parsing and
unblocked before entering the polling loop of the master. This way it
was impossible to start a reload during the configuration parsing.
But with the new model, the polling loop is started in the master before
the configuration parsing is finished, and the signals are still
unblocked at this step. Meaning that it is possible to start a reload
while the configuration is parsing.
This patch reintroduce the behavior of blocking the signals during
configuration parsing adapted to the new model:
- Before the exec() of the reload, signals are blocked.
- When entering the polling loop, the SIGCHLD is unblocked because it is
required to get a failure during configuration parsing in the worker
- Once the configuration is parsed, upon success in _send_status() or
upon failure in run_master_in_recovery_mode() every signals are unblocked.
This patch must be backported as far as 3.1.
Move the char *msg variable declared in main() in a sub-block since
there's already multiple msg variable in other sub-blocks in this
function.
Also make it const.
The QUIC backend crashes when its peer does not support 0-RTT. In this case,
when the sessions are reused, no early-data level secrets are derived by
the TLS stack. This leads to crashes from qc_send_mux() which does not suppose
that both early-data level (qc->eel) and application level (qc->ael) cipher levels
could be non initialized.
To fix this:
- prevent qc_send_mux() to send data if these two encryption level are not
intialized. In this case it returns QUIC_TX_ERR_NONE;
- avoid waking up the MUX from XPRT ->start() callback if the MUX is ready
but without early-data level secrets to send them;
- ensure the MUX is woken up by qc_ssl_do_handshake() after handshake completion
if it is ready calling qc_notify_send()
Thank you to @InputOutputZ for having reported this issue in GH #3188.
No need to backport because QUIC backends is a current 3.3 development feature.
There was no mworker-max-reload value by default, it was set to INT_MAX
so this was impossible to reach.
The default value is now 50, which is still high, but no workers should
undergo that much reloads. Meaning that a worker will be killed with
SIGTERM if it reach this much reloads.
Remove <ipv4> argument from qc_new_conn(). This parameter is unnecessary
as it can be derived from the family type of the addresses also passed
as argument.
The objective of this patch is to streamline qc_new_conn() usage so that
it is similar for frontend and backend sides.
Previously, several parameters were set only for frontend connections.
These arguments are replaced by a single quic_rx_packet argument, which
represents the INITIAL packet triggering the connection allocation on
the server side. For a QUIC client endpoint, it remains NULL. This usage
is consider more explicit.
As a minor change, <target> is moved as the first argument of the
function. This is considered useful as this argument determines whether
the connection is a frontend or backend entry.
Along with these changes, qc_new_conn() documentation has been reworded
so that it is now up-to-date with the newest usage.
quic_conn has two fields named <dcid> and <scid>. It may cause confusion
as it is not obvious how these fields are related to the connection
direction. Try to improve this by extending the documentation of these
two fields.
When a new backend connection is instantiated, a CID is first randomly
generated. It will serve as the first DCID for incoming packets from the
server. Prior to this patch, if the generated CID caused a collision
with an other entries from another connection, an error is reported and
the connection cannot be allocated.
This patch improves this procedure by implementing retries when a
collision occurs. Now, at most three attemps will be performed before
giving up. This is the same procedure already performed for CIDs
instantiated after RETIRE_CONNECTION_ID frame parsing.
Along with this functional change, qc_new_conn() is refactored for
backend instantiation. The CID generation is extracted from it and the
value is passed as an argument. This is considered cleaner as the code
is more similar between frontend and backend sides.
quic_newcid_from_hash64 is an external callback. If defined, it serves
as a CID method generation, as an alternative to the default random
implementation.
This mechanism was not correctly implemented on the backend side.
Indeed, <hash64> quic_conn member is only setted for frontend
connections. The simplest solution would be to properly define it also
for backend ones. However, quic_newcid_from_hash64 derivation is really
only useful for the frontend side for now. Thus, this patch disables
using it on the backend side in favor of the default random generator.
To implement this, quic_cid_generate() is splitted in two functions, for
both methods of CIDs generation. This is the responsibility of the
caller to select the proper method. On backend side, only random
implementation is now used.
The shard keyword is already used by the peers and on the server lines. And
it is unrelated with the session keys distribution. So instead of talking
about shard for the session key hashing, we now use the term "bucket".
Released version 3.3-dev13 with the following main changes :
- BUG/MEDIUM: config: for word expansion, empty or non-existing are the same
- BUG/MINOR: quic: close connection on CID alloc failure
- MINOR: quic: adjust CID conn tree alloc in qc_new_conn()
- MINOR: quic: split CID alloc/generation function
- BUG/MEDIUM: quic: handle collision on CID generation
- MINOR: quic: extend traces on CID allocation
- MEDIUM/OPTIM: quic: alloc quic_conn after CID collision check
- MINOR: stats-proxy: ensure future-proof FN_AGE manipulation in me_generate_field()
- BUG/MEDIUM: stats-file: fix shm-stats-file preload not working anymore
- BUG/MINOR: do not account backend connections into maxconn
- BUG/MEDIUM: init: 'devnullfd' not properly closed for master
- BUG/MINOR: acme: more explicit error when BIO_new_file()
- BUG/MEDIUM: quic-be: do not launch the connection migration process
- MINOR: quic-be: Parse the NEW_TOKEN frame
- MEDIUM: quic-be: Parse, store and reuse tokens provided by NEW_TOKEN
- MINOR: quic-be: helper functions to save/restore transport params (0-RTT)
- MINOR: quic-be: helper quic_reuse_srv_params() function to reuse server params (0-RTT)
- MINOR: quic-be: Save the backend 0-RTT parameters
- MEDIUM: quic-be: modify ssl_sock_srv_try_reuse_sess() to reuse backend sessions (0-RTT)
- MINOR: quic-be: allow the preparation of 0-RTT packets
- MINOR: quic-be: Send post handshake frames from list of frames (0-RTT)
- MEDIUM: quic-be: qc_send_mux() adaptation for 0-RTT
- MINOR: quic-be: discard the 0-RTT keys
- MEDIUM: quic-be: enable the use of 0-RTT
- MINOR: quic-be: validate the 0-RTT transport parameters
- MINOR: quic-be: do not create the mux after handshake completion (for 0-RTT)
- MINOR: quic-be: avoid a useless I/O callback wakeup for 0-RTT sessions
- BUG/MEDIUM: acme: move from mt_list to a rwlock + ebmbtree
- BUG/MINOR: acme: can't override the default resolver
- MINOR: ssl/sample: expose ssl_*c_curve for AWS-LC
- MINOR: check: delay MUX init when SSL ALPN is used
- MINOR: cfgdiag: adjust diag on servers
- BUG/MINOR: check: only try connection reuse for http-check rulesets
- BUG/MINOR: check: fix reuse-pool if MUX inherited from server
- MINOR: check: clarify check-reuse-pool interaction with reuse policy
- DOC: configuration: add missing ssllib_name_startswith()
- DOC: configuration: add missing openssl_version predicates
- MINOR: cfgcond: add "awslc_api_atleast" and "awslc_api_before"
- REGTESTS: ssl: activate ssl_curve_name.vtc for AWS-LC
- BUILD: ech: fix clang warnings
- BUG/MEDIUM: stick-tables: Always return the good stksess from stktable_set_entry
- BUG/MINOR: stick-tables: Fix return value for __stksess_kill()
- CLEANUP: stick-tables: Don't needlessly compute shard number in stksess_free()
- MINOR: h1: h1_release() should return if it destroyed the connection
- BUG/MEDIUM: h1: prevent a crash on HTTP/2 upgrade
- MINOR: check: use auto SNI for QUIC checks
- MINOR: check: ensure QUIC checks configuration coherency
- CLEANUP: peers: remove an unneeded null check
- Revert "BUG/MEDIUM: connections: permit to permanently remove an idle conn"
- BUG/MEDIUM: connection: do not reinsert a purgeable conn in idle list
- DEBUG: extend DEBUG_STRESS to ease testing and turn on extra checks
- DEBUG: add BUG_ON_STRESS(): a BUG_ON() implemented only when DEBUG_STRESS > 0
- DEBUG: servers: add a few checks for stress-testing idle conns
- BUG/MINOR: check: fix QUIC check test when QUIC disabled
- BUG/MINOR: quic-be: missing version negotiation
- CLEANUP: quic: Missing succesful SSL handshake backend trace (OpenSSL 3.5)
- BUG/MINOR: quic-be: backend SSL session reuse fix (OpenSSL 3.5)
- REGTEST: quic: quic/ssl_reuse.vtc supports OpenSSL 3.5 QUIC API
This scripts is supported by OpenSSL 3.5 QUIC API since this previous commit:
BUG/MINOR: quic: backend SSL session reuse fix (HAVE_OPENSSL_QUIC)
Should be backported where this commit is backported.
This bug impacts only the QUIC backends when haproxy is compiled against
OpenSSL 3.5 with QUIC API(HAVE_OPENSSL_QUIC).
The QUIC clients could not reuse their SSL session because the TLS tickets
received from the servers could not be provided to the TLS stack. This should
be done when the stack calls ha_quic_ossl_crypto_recv_rcd()
(OSSL_FUNC_SSL_QUIC_TLS_CRYPTO_RECV_RCD callback).
According to OpenSSL team, an SSL_read() call must be done after the handshake
completion. It seems the correct location is at the same level as for
SSL_process_quic_post_handshake() for quictls.
Thank you to @mattcaswell, @Sashan and @vdukhovni for having helped in solving
this issue.
Must be backported to 3.1
This very minor issue impacts only the backend when compiled against OpenSSL 3.5
with QUIC API (HAVE_OPENSSL_QUIC).
The "SSL handshake OK" trace was not dumped by a TRACE() call. This was very
annoying when debugging.
Modify the concerned code section which is a bit ugly and simplify it.
The TRACE() call is done at a unique location for now on.
Should be backported to 3.2 to ease any further backport.
This bug impacts only the QUIC clients (or backends). The version negotiation
was not supported at all for them. This is an oversight.
Contrary to the QUIC server which choose the negotiated version after having
received the transport parameters (into ClientHello message) the client selects
the negotiated version from the first Initial packet version field. Indeed, the
server transport parameters are inside the ServerHello messages ciphered
into Handshake packets.
This non intrusive patch does not impact the QUIC server implementation.
It only selects the negotiated version from the first Initial packet
received from the server and consequently initializes the TLS cipher context.
Thank you to @InputOutputZ for having reporte this issue in GH #3178.
No need to backport because the QUIC backends support arrives with 3.3.
