As mentioned in previous commit, b_peek_ofs() performs a wrapping check
but is often called with ofs == 0 as a constant. We can detect this case
with __builtin_const_p() so it makes sense to use it. A test shows a size
reduction of about 320 bytes, which is not much, but it happens in hot code
paths, and each 16 bytes reduction indicates an eliminated conditional
branch.
Some clear winners are ci_getblk_nc() (-48 bytes), h2c_dec_hdrs (-141B),
h1_copy_msg_data (-124B), tcpcheck_spop_expect_hello (-80B),
h1_parse_msg_data (-44B). These ones will definitely benefit from doing
less conditional jumps.
The function is an exact copy of b_peek_varint() with ofs==0 and doing a
b_del() at the end. We can simply call that other one and delete the
contents. It turns out that the code is bigger with this change because
b_peek_varint() passes its offset to b_peek() which performs a wrapping
check. When ofs==0 the wrapping cannot happen, but there's no real way
to tell that to the compiler. Instead conditioning the if() in b_peek()
with (!__builtin_constant_p(ofs) || ofs) does the job, but it's not worth
it at the moment since we have no users of b_get_varint() for now. Let's
just stick to the simple normal code.
Some large functions were moved to buf.c by commit ac66df4e2 ("REORG:
buffers: move some of the heavy functions from buf.h to buf.c"). However,
as found by Amaury, haring doesn't build anymore. Upon close inspection,
b_getblk_nc() isn't that big since it's very much inlinable, and a part
of its apparently large size comes from the BUG_ON_HOT() that were
implemented. Regarding b_peek_varint(), it doesn't have any dependency
and is used only at 4 places in the DNS code, so its loop will not have
big impacts, and the rest around can be optimised away by the compiler
so it remains relevant to keep it inlined. Also it can serve as a base
to deduplicate the code in b_get_varint().
No backport needed.
The ARGT_ID argument type may now be used to set a custom resolve
function in order to help resolve the argument string value. If the
custom resolve function is not set, the behavior is the same as of
type ARGT_STR.
A useless BUG_ON() statement was let in a conditional block that already
checks that the condition cannot be met within the block. Remove the
useless BUG_ON()
"option forwarded" provides a convenient way to automatically insert
rfc7239 forwarded header to requests sent to servers.
On the other hand, manually crafting the header is quite complicated due
to specific formatting rules that must be followed as per rfc7239.
However, sometimes it may be necessary to craft the header manually, for
instance if it has to be conditional or based on parameters that "option
forwarded" doesn't provide. To ease this task, in this patch we implement
rfc7239_nn and rfc7239_np which are respectively meant to craft nodename:
nodeport values, specifically intended to manually build rfc7239 'for'
and 'by' header fields while ensuring rfc7239 compliancy.
Example:
# build RFC-compliant 7239 header:
http-request set-var-fmt(txn.forwarded) "for=\"%[ipv6(::1),rfc7239_nn]:%[str(8888),rfc7239_np]\";host=\"haproxy.org\";proto=http"
# check RFC-compliancy:
http-request set-var(txn.test) "var(txn.forwarded),debug(ok,stderr),rfc7239_is_valid,debug(ok,stderr)"
# stderr output:
# [debug] ok: type=str <for="[::1]:_8888";host="haproxy.org";proto=http>
# [debug] ok: type=bool <1>
See documentation for more info and examples.
This issue came with this commit:
f627b92 BUG/MEDIUM: quic: always validate sender address on 0-RTT
and could be easily reproduced with picoquic QUIC client with -Q option
which splits a big ClientHello TLS message into two Initial datagrams.
A second condition must be fulfilled to reprodue this issue: picoquic
must not send the token provided by haproxy (NEW_TOKEN). To do that,
haproxy must be patched to prevent it to send such tokens.
Under these conditions, if haproxy has enough time to reply to the first Initial
datagrams, when it receives the second Initial datagram it sends a Retry paquet.
Then the client ignores the Retry paquet as mentionned by RFC 9000:
17.2.5.2. Handling a Retry Packet
A client MUST accept and process at most one Retry packet for each connection
attempt. After the client has received and processed an Initial or Retry packet
from the server, it MUST discard any subsequent Retry packets that it receives.
On its side, haproxy has closed the connection. When it receives the second
Initial datagram, it open a new connection but with Initial packets it
cannot decrypt (wrong ODCID) leaving the client without response.
To fix this, as the aim of the token (NEW_TOKEN) sent by haproxy is to validate
the peer address, in place of closing the connection when no token was received
for a 0RTT connection, one leaves this validation to the handshake process.
Indeed, the peer adress is validated during the handshake when a valid handshake
packet is received by the listener. But as one does not want haproxy to process
0RTT data when no token was received, one does not accept the connection before
the successful handshake completion. In addition to this, the 0RTT packets
are not released after successful handshake completion when no token was received
to leave a chance to haproxy to process these 0RTT data in such case (see
quic_conn_io_cb()).
Must be backported as far as 2.9.
Tokens are sent when opening a connection, just after the handshake, to
be possibly reused by the peer for the next connection. They are used
to validate the peer address during the 0RTT connection openings.
But there is no reason to reserve this feature to 0RTT connections.
This patch modifies quic_build_post_handshake_frames() to do so.
This bug came with this commit:
f627b92 BUG/MEDIUM: quic: always validate sender address on 0-RTT
If an error happens in quic_build_post_handshake_frames() during the
code exexuted for th NEW_TOKEN frame allocation, some could leak because
of the wrong label used to interrupt this function asap.
Replace the "goto leave" by "goto err" to deallocated such frames to fix
this issue.
Must be backported as far as 2.9.
When a filter is registered on the data, it means it may change the payload
length by rewritting data. It means consumers of the message cannot trust the
expected length of payload as announced by the producer. The commit 8bd835b2d2
("MEDIUM: filters/htx: Don't rely on HTX extra field if payload is filtered")
was pushed to solve this issue. When the HTTP payload of a message is filtered,
the extra field is set to 0 to be sure it will never be used by error by any
consumer. However, it is not enough.
Indeed, the filters must be called before fowarding some data. They cannot be
by-passed. But if a consumer is unable to flush the HTX message, some outgoing
data can remain blocked in the channel's buffer. If some new data are then
pushed because there is some room in the channel's buffe, the producer will set
the HTX extra field. At this stage, if the consumer is unblocked and can send
again data, it is possible to call it to forward outgoing data blocked in the
channel's buffer before waking the stream up to filter new input data. It is the
purpose of the data fast-forwarding. In this case, the HTX extra field will be
seen by the consumer. It is unexpected and leads to undefined behavior.
One consequence of this bug is to perform a wrong chunking on compressed
messages, leading to processing errors at the end of the message, reported as
"ID--" in logs.
To fix the bug, a HTX flag is added to state the payload of the current HTX
message is altered. When this flag is set (HTX_FL_ALTERED_PAYLOAD), the HTX
extra field must not be trusted. And to keep things simple, when this flag is
set, the HTX extra field is automatically set to 0 when the HTX message is
loaded, in htxbuf() function.
It is probably the less intrusive way to fix the bug for now. But this part must
be reviewed to save meta-info of the HTX message outside of the message itself.
This commit should solve the issue #2741. It must be backported as far as 2.9.
In sc_notify() function, the consumer side of the SC is tested to verify if
we must perform a shutdown on the endpoint. To do so, no output data must be
present in the buffer and in the iobuf. However, there is a bug here, the
iobuf of the opposite SC is tested instead of the one of the current SC. So
a shutdown can be performed on the endpoint while there are still output
data in the iobuf that must be sent. Concretely, it can only be data blocked
in a pipe.
Because of this bug, data blocked in the pipe will be never sent. I've not
tested but I guess this may block the stream during the client or server
timeout.
This patch must be backported as far as 2.9.
If a parsing error is reported by the mux on the response payload, a proxy
error (PRXCOND) must be reported instead of a server abort (SRVCL). Because
of this bug, inavlid response may are reported as "SD--" or "SL--" in logs
instead of "PD--" or "PL--".
This patch must be backported to all stable versions.
When the stream is shut down, some tests are performed to know if the
connection must also be closed or not. There are trace messages for all
cases, except for the default one: Abort or close-mode. Thanks to this
patch, there is now a message too in this case.
Info about the SD iobuf are now dumped in trace messages when a stream send
event is processed. It is a useful information to debug zero-copy forwarding
issues.
When a send attempt is performed on the opposite side from sc_notify() and
all outgoing data are sent while a shut was scheduled, the SE is shut down
because we consider all data were sent and no more are expected. However,
here we must also be carefull to have sent all pending data in the
iobuf. Indeed, some spliced data may be blocked. In this case, if the SE is
shut down, these data may be lost.
This patch should fix the original bug reported in #2749. It must be
backported as far as 2.9.
The proxy must be created in mworker mode, but only in the worker, not in
the master. The current code creates the proxy in both processes.
The patch only checks that we are not in the master to start the
ocsp-update pre-check.
No backport needed.
Since commit fe75c1e12d ("MEDIUM: startup: remove
MODE_MWORKER_WAIT") the MODE_MWORKER_WAIT constant disappeared. The
initialization of the default resolvers section was conditionned by this
constant.
The section must be created in mworker mode, but only in the worker not in
the master. It was currently completely disabled in both the master and
the worker which could break configuration using it, as well as the
httpclient.
No backport needed.
Since commit fe75c1e12d ("MEDIUM: startup: remove
MODE_MWORKER_WAIT") the MODE_MWORKER_WAIT constant disappearded. The
initialization of the httpclient proxy was conditionned by this
constant.
The proxy must be created in mworker mode, but only in the worker not in
the master. It was currently completely disabled in both the master and
the worker provoking a NULL dereference upon httpclient usage.
No backport needed.
Latest patches on the mworker rework skipped the httpclient_proxy
creation by accident. This is not supposed to happen because haproxy is
supposed to stop when the proxy creation failed, but it shows a flaw in
the API.
When the httpclient_proxy or the proxy used in parameter of
httpclient_new_from_proxy() is NULL, it will be dereferenced and cause a
crash.
The patch only returns a NULL when doing an httpclient_new() if the
proxy is not available.
Must be backported as far as 2.7.
Released version 3.1-dev10 with the following main changes :
- BUG/MAJOR: mux-quic: do not crash on empty STREAM frame emission
- BUG/MINOR: stats: Fix the name for the total number of streams created
- MINOR: quic: strengthen qc_release_frm()
- MEDIUM: quic: decount acknowledged data for MUX txbuf window
- MINOR: quic: implement dedicated type for out-of-order stream ACK
- MEDIUM: quic: merge contiguous/overlapping buffered ack stream range
- MEDIUM: quic: decount out-of-order ACK data range for MUX txbuf window
- MINOR: log: add do_log() logging helper
- MINOR: log: add do_log_parse_act() helper func
- MINOR: action: add do-log action
- REGTESTS: add some tests for 'do-log' action
- BUG/MEDIUM: hlua: make hlua_ctx_renew() safe
- BUG/MEDIUM: hlua: properly handle sample func errors in hlua_run_sample_{fetch,conv}()
- BUG/MINOR: quic: fix discarding of already stored out-of-order ACK
- BUG/MEDIUM: quic: properly decount out-of-order ACK on stream release
- MINOR: ssl: disable server side default CRL check with WolfSSL
- MEDIUM: sink: implement sink_find_early()
- MINOR: trace: postresolve sink names
- MINOR: sample: postresolve sink names in debug() converter
- BUG/MEDIUM: mux-quic: ensure timeout server is active for short requests
- MINOR: cfgparse: simulate long configuration parsing with force-cfg-parser-pause
- BUILD: cache: silence an uninitialized warning at -Og with gcc-12.2
- BUG/MINOR: mux-h2/traces: present the correct buffer for trailers errors traces
- MINOR: mux-h2/traces: print the size of the DATA frames
- CLEANUP: muxes: remove useless inclusion of ebmbtree.h
- REORG: buffers: move some of the heavy functions from buf.h to buf.c
- MINOR: buffer: add a buffer list type with functions
- MINOR: mux-h2: split the amount of rx data from the amount to ack
- MINOR: mux-h2: create and initialize an rx offset per stream
- MEDIUM: mux-h2: start to update stream when sending WU
- MEDIUM: mux-h2: start to introduce the window size in the offset calculation
- MINOR: mux-h2: count within a connection, how many streams are receiving data
- MINOR: mux-h2: allocate the array of shared rx bufs in the h2c
- MINOR: mux-h2: add rxbuf head/tail/count management for h2s
- MINOR: mux-h2: move H2_CF_WAIT_IN_LIST flag away from the demux flags
- MINOR: mux-h2: simplify the exit code in h2_rcv_buf()
- MINOR: mux-h2: simplify the wake up code in h2_rcv_buf()
- MINOR: mux-h2: clear up H2_CF_DEM_DFULL and H2_CF_DEM_SHORT_READ ambiguity
- MAJOR: mux-h2: make streams use the connection's buffers
- MAJOR: mux-h2: permit a stream to allocate as many buffers as desired
- MAJOR: mux-h2: make the rxbuf allocation algorithm a bit smarter
- MINOR: mux-h2: add tune.h2.be.rxbuf and tune.h2.fe.rxbuf global settings
- MEDIUM: mux-h2: change the default initial window to 16kB
- DOC: design-thoughts: add diagrams illustrating an rx win groth
- MEDIUM: mux-h2: rework h2_restart_reading() to differentiate recv and demux
- OPTIM: mux-h2: make h2_send() report more accurate wake up conditions
- OPTIM: mux-h2: try to continue reading after demuxing when useful
- OPTIM: mux-h2: use tasklet_wakeup_after() in h2s_notify_recv()
- MINOR: mux-h2/traces: add missing flags and proxy ID in traces
- MINOR: mux-h2/traces: add buffer-related info to h2s and h2c
- CI: cirrus-ci: bump FreeBSD image to 14-1
- REGTESTS: fix a reload race in abns_socket.vtc
- MINOR: activity/memprofile: always return "other" bin on NULL return address
- MINOR: quic: notify connection layer on handshake completion
- BUG/MINOR: stream: unblock stream on wait-for-handshake completion
- BUG/MEDIUM: quic: support wait-for-handshake
- BUG/MEDIUM: server: server stuck in maintenance after FQDN change
- BUG/MEDIUM: queue: make sure never to queue when there's no more served conns
- DEBUG: mux-h2/flags: add H2_CF_DEM_RXBUF & H2_SF_EXPECT_RXDATA for the decoder
- REGTESTS: cli: add delay 0.1 before connect to cli
- MINOR: startup: add O_CLOEXEC flag to open /dev/null
- MEDIUM: startup: move daemonization fork in init
- MINOR: startup: refactor "daemonization" fork
- MEDIUM: startup: move PID handling in init()
- MAJOR: mworker: move master-worker fork in init()
- BUG/MINOR: mworker: fix memory leak due to master-worker fork
- REORG: mworker: set nbthread=1 for master after fork
- MINOR: init: check MODE_MWORKER before creating master CLI
- REORG: mworker: move mworker_create_master_cli in master 'case'
- MEDIUM: startup: call chroot() if needed in one place
- MEDIUM: startup: do set_identity() if needed in one place
- MINOR: startup: only worker gets capabilities from bin
- CLEANUP: haproxy: rm no longer used mworker_reexec_waitmode
- MINOR: startup: rename exit_on_waitmode_failure to exit_on_failure
- MINOR: defaults: update MASTER_MAXCONN description
- MEDIUM: startup: remove MODE_MWORKER_WAIT
- MINOR: global: add MODE_DISCOVERY flag
- MEDIUM: cfgparse: add KWF_DISCOVERY keyword flag
- MEDIUM: cfgparse: call some parsers only in MODE_DISCOVERY
- MEDIUM: cfgparse-global: parse only KWF_DISCOVERY keywords in MODE_DISCOVERY
- MEDIUM: cfgparse: parse only "global" section in MODE_DISCOVERY
- MEDIUM: startup: introduce load_cfg and read_cfg
- MINOR: cfgparse: fix *thread keywords sensitive to global section position
- MINOR: mworker/cli: rename mworker_cli_proxy_new_listener
- MINOR: mworker/cli: rename and clean mworker_cli_sockpair_new
- MINOR: mworker/cli: create master CLI sockpair before fork
- MINOR: mworker/cli: create MASTER proxy before mcli listeners
- MINOR: mworker: add and set state PROC_O_INIT for new worker
- MEDIUM: mworker/cli: close child and parent fds, setup listeners
- MINOR: mworker: mworker_catch_sigchld: use fd_delete instead of close
- MINOR: startup: rename and adapt reexec_on_failure
- MINOR: mworker: add support for case when new worker dies
- MINOR: mworker: simplify the code that sets PROC_O_LEAVING
- MINOR: mworker/cli: add _send_status to support state transition
- MEDIUM: startup: split sending oldpids_sig logic for standalone and mworker modes
- MINOR: startup: split init() into separate initialization routines
- MINOR: startup: split main: add step_init_3
- MINOR: startup: simplify check for calling sock_get_old_sockets
- MINOR: startup: encapsulate sock_get_old_sockets in a function
- MINOR: startup: add bind_listeners
- MINOR: startup: split main: add step_init_4
- MINOR: startup: encapsulate master's code in run_master
- MINOR: startup: add read_cfg_in_discovery_mode
- MINOR: mworker: adapt exit_on_failure for master recovery mode
- MEDIUM: mworker: add support of master recovery mode
- MINOR: startup: add set_verbosity
- MEDIUM: mworker: block reloads
- MINOR: mworker: slow load status delivery if worker is starting
- MINOR: mworker: readapt program support in mworker_catch_sigchld
- MINOR: mworker: deserialize process list before read_cfg_in_discovery_mode
- MINOR: mworker: parse program only in MODE_DISCOVERY
- MINOR: cfgparse: add support for program section
- MINOR: startup: reintroduce program support
- MINOR: mworker-prog: stop old programs in mworker_ext_launch_all
- MINOR: mworker: reintroduce systemd support
- MINOR: mworker: report explicitly when worker exits due to max reloads
- MINOR: cfgparse-global: parse *env keywords in MODE_DISCOVERY
- MINOR: startup: reintroduce *env keywords support
- MINOR: startup: close devnullfd, when daemon mode is applied
In case of daemon mode now daemonization fork happens in the early init stage
before parsing and applying the configuration, so we can't close
stdio/stderr/stdout immediately after forking. We keep it open until the most
of configuration, including chroot are applied in order to show alerts, if
there are some problems. To achieve this /dev/null is opened just before calling
chroot(), and after the chroot block it's used to close all standard outputs
and stdin. At this point we no longer need the fd of /dev/null, so we can close
it as well.
setenv/resetenv/presetenv/unsetenv keywords in the configuration modify the
process environment. In case of master-worker and programs we need to restore
the initial process environment before reload, as the configuration could
change in between and newly forked workers and programs should be launched
in the environment corresponded to this new configuration.
To achieve this we backup the initial process environment before the first
configuration read, when 'global' and 'program' sections are read. And then we
clean up master process environment and restore the initial one from the backup
in mworker_reexec().
setenv/resetenv/presetenv/unsetenv keywords should be parsed by master
process and by worker. As some other master parameters could be enabled in
conditional blocks (.if...endif). To achieve this let's tag '*env' keywords
with KWF_DISCOVERY flag.
It's convienient for testing and for usage to produce different warning
messages, when the former worker exits due to max reloads exceeded, and when it
was terminated by the master.
Let's reintroduce systemd support in the refactored master-worker mode.
As for now, the master-worker fork happens during early initialization steps and
then the master process receieves the "READY" status message from the newly
forked worker, that has successfully loaded. Let's propagate this "READY" status
message at this moment to the systemd from the master process context
(_send_status()). We use the master process to send messages to systemd,
because it is only the process, monitored by systemd.
In master recovery mode, we also need to send to the systemd the "READY"
message, but with the status "Reload failed". "READY" will signal to systemd,
that master process is still alive, because it doesn't exit in recovery mode
and it keeps the existed worker. Status "Reload failed" will signal to user,
that something wrong has happened with the configuration. Same message logic
was originally preserved for the case, when the worker fails to read its
configuration, see on_new_child_failure() for more details.
This patch is a part of series to reintroduce the program support in the new
master-worker architecture.
Now, after refactoring in master-worker mode it's the master process, who
stops workers forked before the reload. Current worker no longer sends USR1 or
TERM signals to the previous one after ports binding. This behaviour is kept
only for the standalone mode.
So, in case of programs, it's up to master process as well to stop programs,
which were launched before reload. Let's do this in mworker_ext_launch_all(),
just before starting the new programs.
This patch is a part of series to reintroduce the program support in the new
master-worker architecture.
Let's add here mworker_ext_launch_all() call before master-worker fork to
start external programs. We keep the order and the place of these two forks
(program and master-worker) the same as before the refactoring, in order to
avoid regressions.
This patch is a part of series to reintroduce the program support in the new
master-worker architecture.
Programs are launched by master, thus only the master process needs its
configuration. Therefore, program section parser should be called only in
discovery mode, when master parses its configuration.
Program section has a post section parser. It should be called only in
discovery mode as well.
This patch is a part of series to reintroduce the program support in the new
master-worker architecture.
Master process launches external programs, so it needs to read program
section. Thus, it should be parsed in MODE_DISCOVERY. Worker does not need
program settings, so let's check the runtime mode in cfg_parse_program. Worker
should always skip this section.
This patch is a part of series to reintroduce the program support in the new
master-worker architecture.
For the moment we keep the order of program and worker forks the same as before
the refactoring, as we need to be sure that this won't introduce regressions.
So, programs are forked before the new worker process.
Before the program's fork we already need deserialized processes list to find
the programs launched before reload and to stop them. Processes list saved
before the reload in HAPROXY_PROCESSES variable. It should be deserialized
before the first configuration read in discovery mode, because resetenv keyword
could be presented in the global section.
So, let's move mworker_env_to_proc_list() from mworker_create_master_cli() to
main(). We need to call it only after reload in master-worker mode, thus
HAPROXY_MWORKER_REEXEC and HAPROXY_PROCESSES should be still presented in the
re-executing process environment before the first configuration read.
This patch is a part of series to reintroduce the program support in the new
master-worker architecture.
We just only launch and stop external programs and there is no any
communication between the master process and the started program binary. So,
ipc_fd[0] and ipc_fd[1] are not used and kept as -1 for programs processes. Due
to this, no need for the exiting program process to call fd_delete on this
fds. Otherwise, this will trigger a BUG_ON.
With refactored master-worker architecture master and worker processes parse
its parts of the configuration. Worker could have a huge configuration, so it
will take some time to load. As now HAPROXY_LOAD_SUCCESS is set to 1 only
after receiving the status READY from the new worker
cli_io_handler_show_loadstatus() may exit very fast by showing load status 0,
and in such case and mcli socket will be closed.
This already breaks some regression tests and can confuse some APIs. So, let's
slow down the load status delivery. If in the process list there is still some
process, which is loading (PROC_O_INIT). appctx task will sleep in this case for
50ms and then return 0. cli_io_handler_show_loadstatus() is called in loop, so
with such pacing, there is a high chance that the next time, when we enter in
its scope all processes will have the state READY. Like this master CLI
connection socket won't be closed until the loading of the new worker is really
finished, thus the reload status and logs (Success=1/0) will be shown in
synchronious way.
When reloads arrive very often (sent by some APIs), newly forked workers
almost don't have a time to load completely and to send its READY status to
master, which allows then to stop the previous worker (launched before reload).
As a result, the number of workers increases very quickly, previous workers are
still alive and the memory consumption is very high.
To avoid such situations let's return in cli_parse_reload() reload status 0
with the text ""Another reload is still in progress", if there is still a
process with PROC_O_INIT flag in the processes list.
Let's encapsulate the logic to set verbosity modes (MODE_DEBUG and MODE_VERBOSE)
in a separate function set_verbosity(). This makes the code of main() more
readable and this allows to call set_verbosity() for master process in recovery
mode. So, in this mode, verbosity settings before the master re-execution will
be re-applied to master. set_verbosity() will be extended in future commits to
reduce the verbosiness of master in order not to dump pollers list and filters,
if it was started with -V or -d.
In this commit we add run_master_in_recovery_mode(), which groups all necessary
initialization steps, which master should perform to be able to enter in its
polling loop (run_master()), when it fails while parsing its new config.
As exit_on_failure() is now adapted for master recovery mode. Let's register
it as atexit handler, when master enters in this mode. And let's remove
atexit_flag variable for master, because we no longer use it.
We also slightly refactor here read_cfg_in_discovery_mode() in order to call
run_master_in_recovery_mode() for the case, described above. Warning messages
are mandatory before calling the run_master_in_recovery_mode() as this allows
to stop haproxy with error, if it was launched in zero-warning mode.
So, in recovery mode master does not launch any worker. It just performs its
necessary initialization routines and enters in its polling loop to continue
to monitor the existed worker process.
Master recovery mode replaces the former wait-mode with a difference, that
master in this case doesn't try to fork the new worker process. But it still
needs to enter to its polling loop in order to monitor the previous worker.
Master performs some initialization steps for this and it recreates its master
CLI. During its initialization steps, master could potentially fail again.
As we use for the moment for master init steps some common routines
(step_init_2() and step_init_3()), there is no way there to signal to user that
failure has happened for the master and in addition, in its recovery mode. So,
in such case exit_on_failure() can be still useful in order to print an
appropriate alert, as we can register this function as atexit handler for the
master.
Let's encapsulate here the code to load and to read the configuration at the
first time in MODE_DISCOVERY. This makes the code of main() more readable and
this adds the structure for adding necessary master initializations routines
to support master recovery mode.
Let's encapsulate master's code (steps which it does before entering in its
polling loop and deinitialization routines after) in a separate run_master()
function. This makes the code of main() more readable. In future we plan to put
in run_master() more master process related code, in order to clean completely
init_step_2(), init_step_3() and init_step_4().
Let's encapsulate here another part of main, after binding listeners sockets
and before calling the master's code in master-worker mode. This block
contains the code, which applies verbosity settings, checks limits and updates
the ready date. It will take some time to figure out, which of these parts are
really needed for the master, or which ones it could skip. So let's put all
these for the moment in step_init_4() and let's call it for all modes.
Let's encapsulate here the code, which tries to bind listeners for the new
process in a separate function. This will make the main() code more readable.
Master process, even if it has failed while reading its new configuration, has
to bind its master CLI sockets. So like this we will can call this function in
the master recovery mode.
Master CLI socket address and port for external connections (user, monitoring
tools) are provided for now only via the command line. So, master, even after
this failure can and must reestablish master CLI connections again.
Let's encapsulate here the code, that calls sock_get_old_sockets() to obtain
listeners sockets from the previous process into a separate function. This
will make the code of main() more readable and we can move this new function
(if we might need so) in future.
MODE_CHECK and MODE_CHECK_CONDITION are applied now very early in
step_init_1() and step_init_2() in order to check the configuration or to check
some condition provided via the command line. When these checks have
terminated, the main process exits. So, no longer need to verify these modes at
the moment, when the current process have already done its basic initialization
routines and is asking for listeners sockets from the previously started one.
The first part of main(), just after calling the former init() and before
trying to bind listeners, need to be also encapsulated into a separate
step_init_3() as it is. It contains important blocks to register signals, to
apply memory and nofile limits, etc. The order of these blocks should be also
preserved (especially the signals part).
For the moment step_init_3() must be also executed for all runtime modes.
This is the first commit in a series to add a support of the 5-th reload
use case, when the master process fails to read its new configuration. In this
case it just need to perform its initialization steps and keep the existed
worker.
To add the support for this last use case we need to split init() and main()
in a shorter steps in order to encapsulate necessary initialization routines
into separate functions.
Let's at first, make here progname as a global variable for haproxy.c, as it
will be used in error messages in the initialization functions. Then let's
split the init() into separate routines, which set and apply modes, write
process PID in a pidfile, etc.
The big part of the former init(), which called functions to allocate pools,
to initialize proxies, to calculate maxconn and to perform some post checks was
just encasulated as is, into step_init_2(). It will take some time to figure
out exactly which parts of this initialization block are really necessary for
the master process and which ones it could skip. So, for the moment
step_init_2() is called for all runtime modes.
Before refactoring the master-worker mode, in all runtime modes, when the new
process successfully parsed its configuration and bound to sockets, it sent
either SIGUSR1 or SIGTERM to the previous one in order to terminate it.
Let's keep this logic as is for the standalone mode. In addition, in standalone
mode we need to send the signal to old process before calling set_identity(),
because in set_identity() effective user or group may change. So, the order is
important here.
In case of master-worker mode after refactoring, master terminates the previous
worker by itself up to receiving "READY" status from the new one in
_send_status(). Master also sets at this moment HAPROXY_LOAD_SUCCESS env
variable and checks, if there are some other workers to terminate with
max_reloads exceeded.
So, now in master-worker mode we terminate old workers only, when the new one
has successfully done all initialization steps and has sent "READY" status to
master.
In the new master-worker architecture, when a worker process is forked and
successfully initialized it needs somehow to communicate its "READY" state to
the master, in order to terminate the previous worker and workers, that might
exceeded max_reloads counter.
So, let's implement for this a new master CLI _send_status command. A new
worker can send its status string "READY" to the master, when it's about
entering to the run poll loop, thus it can start to receive data.
In _send_status() in the master context we update the status of the new worker:
PROC_O_INIT flag is withdrawn.
When TERM signal is sent to a worker, worker terminates and this triggers the
mworker_catch_sigchld() handler in master. This handler deletes the exiting
process entry from the processes list.
In _send_status() we loop over the processes list twice. At the first time, in
order to stop workers that exceeded the max_reloads counter. At the second time,
in order to stop the worker forked before the last reload. In the corner case,
when max_reloads=1, we avoid to send SIGTERM twice to the same worker by
setting sigterm_sent flag during the first loop.
When master performs a reexec it should set for an already existed worker the
flag PROC_O_LEAVING. It means that existed worked is marked as the previous one
and will be terminated after the reload.
In the previous implementation master process was need to do the reexec
twice (the first time for parsing its configuration and the second time to free
unused ressources). So the logic of setting PROC_O_LEAVING was based on
comparing the number of reloads, performed by each process from the processes
list, except the master.
Now, as being mentioned before, reexec is performed only once. So, in this case
we need to set PROC_O_LEAVING flag, when we deserialize the list. It is done for
all processes, which have the number of reloads stricly positive.
