# Master Worker

2024-06-12

## History

### haproxy-systemd-wrapper

Back in 2013, distributions are discussing the adoption of systemd as the
default init, this was controversial but fedora and archlinux already uses it.
At this time HAProxy still had a multi-process model, and the way haproxy is
working was incompatible with the daemon mode.

Systemd is compatible with traditional forking services, but somehow HAProxy
is different. To work correctly, systemd needs a main PID, this is the PID of
the process that systemd will supervises.

With `nbproc 1` that could work, since systemd is able to guess the main PID,
and even to read a PID file. But there is something uncommon that HAProxy is
doing for a reload, which is not supported by systemd. Indeed the reload is in
fact a new haproxy process, which will ask the old one to leave. This means the
new main PID is supposed to change, but systemd is not supporting this, so it
will just see the previous leaving and consider that the service broke and kill
every other processes, meaning the new haproxy.

With `nbproc > 1` this is worse, systemd is confused with all the processes,
because they are independent, so there is not really a main process to
supervise.

The systemd-wrapper appeared in HAProxy 1.5, it's a separated binary, which
starts haproxy, so systemd can use the wrapper as the main PID, and the wrapper
never change PID. Upon a reload, which is done with a SIGUSR2 signal the wrapper
will launch a `haproxy -sf`. This was a non-intrusive work which a first step to
deploy in systemd environments. Later contributions would add the support for
upgrading the wrapper binary upon a reload.

However the wrapper suffered from several problems:

- It needed a intermediate haproxy process, it's basically a daemon mode, but
  instead of the first process leaving to daemonize, it is kept in foreground to
  waitpid() on all workers. Which means you need the wrapper + the -Ds + the
  haproxy workers, and each reload start a new -Ds.
- it was difficult to integrate new features since it wasn't in haproxy itself
- there were multiple issues with handling the failures during reload

### mworker V1

HAProxy 1.8 got rid of the wrapper which was replaced by the master worker
mode. This first version was basically a reintegration of the wrapper features
within HAProxy. HAProxy is launched with the -W flag, read the configuration and
then fork. In mworker mode, the master is usually launched as a root process,
and will do chroot operations then setuid in the workers.

Like the wrapper, the master handle the SIGURS2 signal to reload, it is also
able to forward the SIGUSR1 signal to the workers, to ask for a soft stop.
The reload uses the same logic than the standard `-sf` method, but instead of
starting a new process, it will exec() with -sf in the same PID. Which means
that haproxy could upgrade its binary during the reload.

Once the SIGUSR2 signal is received, the master would block signals and unregister
signals handler so no signals would halt haproxy reload, as it could kill the
master to receive a USR2 if the signal is not register yet after the exec.

When doing the exec() upon a reload, a new argv array is constructed by copying
the current argv and adding `-sf` and the list of PIDs in the children list, as
well as the oldpids list.

When the workers are started, the master will first deinit the poller and clean
the FDs that are not needed anymore (inherited fd need to be kept however), then
the master will do a wait() loop instead of the haproxy polling loop, which will
wait for its workers to leave, or for a signal.

When reloading haproxy, a non-working configuration could exits the master,
which could end in killing all previous workers. This is a complex situation to
handle, since all configuration parsing code was not written to let a process
alive upon a failure. To handle this problem, an atexit() callback was used, so
haproxy would reexec() upon a configuration loading failure, without any
configuration, and without trying to fork new workers. This is called the
master-worker "wait mode".

The master-worker mode also comes with a feature which automates the seamless
reload (-x), meaning it would select the stats socket from the configuration to
be added to the -x parameter for the next reload, so the FD of the bind could be
retrieved automatically.

The master is supervising the workers, when a current worker (not a previous one
from before the reload) is exiting without being asked for a reload, the master
will emit an "exit-on-failure" error and will kill every workers with a SIGTERM
and exits with the same error code than the failed worker, this behavior can be
changed by using the "no exit-on-failure" option in the global section.

While the master is supervising the workers using the wait() function, the
workers is also surpervising the master. To achieve this, there is a pipe
between the master and the workers. The FD of the worker side of the pipe is
inserted in the poller so it can watch for a close. When the pipe is closed this
means the master left, and this is not supposed to happen, so it could have
crash. When it happens all workers are leaving. To survive the reloads of the
master, the FD are saved in environment variables (HAPROXY_MWORKER_PIPE_{RD,WR})

The master-worker mode could be activated by using either "-W" or
"master-worker" in the global section of the configuration, but it is prefered
to use "-W".

The pidfile is usable in master-worker mode, instead of writing the PIDs of all
workers, this will only write the PID of the master.

A systemd mode (-Ws) could also be used, it behaves the same way as -W, but will
keep the master in foreground, and will send status messages to systemd using
the sd_notify API.

### mworker V2

HAProxy 1.9 go a little bit further with the master worker, instead of using the
mworker_wait() fuction from V1, it uses the haproxy polling loop, so the signals
will be handled directly by haproxy polling loop, removing the specific code.

Instead of using 1 pipe per haproxy instance, the V2 is using a socketpair per
worker and the polling loop allows real network communication using these
socketpairs. It needs to keep 1 FD per worker in the master, so they can be
reused after a reload. The master keeps a linked list of processes,
mworker_proc, containing socketpairs fd, PID, relative pid... This list is then
serialized in the HAPROXY_PROCESSES environment variable to be unserialized upon
a reload and the FD reinserted in the poller.

Since the FD are in the poller, there is a special flag in the listeners
LI_O_WORKER, which specify that some FD mustn't be used in the worker, these FD
are unbind once in the worker.

Meanwhile the thread support was implemented in haproxy, since mworker shares
more code than before when using the polling loop, the nbthread configuration
variable is not used for instancing the master, and the master will always
remain with only 1 thread.

The HAPROXY_PROCESSES structures allow to store a lot more thing, the number of
reload for each worker is kept, the PID etc...

The socketpairs are useful for bi-directional communication, so each socketpair
are connected to a stats applet on the worker side, so the master could access
to a stats socket for each worker.

The master implements a CLI proxy, which is an analyzer which is able to parse
CLI input, which will split individual CLI commands and redirect them to the
right worker. This is implemented like the HTTP pipelining with command being
sent and responsed one after another. This proxy could be accessed by using the
master CLI which is only bound using the -S option of the haproxy command.
Special prefixed using @ syntax are used to select the right worker.

The master CLI implements its own commands set like `show proc` which shows the
content of the HAPROXY_PROCESSES structure.

A 'reload' command was implemented so the reload could be asked from the master
CLI without using the SIGUSR2 signal.

### more features in mworker V2

HAProxy 2.0 implements a new configuration section called `program` this section
allows to handle the start and stop of executables with the master-worker. One
could launch the dataplane API from haproxy for example. The programs are
shown in the `show proc` command. The programs will be added to the
HAPROXY_PROCESSES structure. The option 'start-on-reload' allows to configure
the behavior of a program during an haproxy reload, it can either start a new
instance of the program or keep the previous one.

A `mworker-max-reloads` keyword was added in the global section, it allows to
limit the number of reload a worker can endure. That helps limiting the number
of remaining worker processes. This will send a SIGTERM to the worker once it
reach this value, instead of a SIGUSR1, so any stuck worker is killed.

Version and starting time were added to HAPROXY_PROCESSES so they could be
displayed in `show proc`.

HAProxy 2.1 added user/group to the program section so they could change their
uid after the fork.

HAProxy 2.5 added the reexec of haproxy in wait mode after a successful loading,
instead of doing it only after a configuration failure. It is useful to clear
the memory of the master because charging the configuration from the master can
take a lot of RAM, and there is no simple wait to free everything and decrease
the memory space of the process.

In HAProxy 2.6, the seamless reload with the master-worker changed, instead of
using a stats socket declared in the configuration, this uses the internal
socketpair of the previous worker. The change is actually simple, instead of
doing a `-x /path/to/previous/socket` it does a `-x sockpair@FD` using the FD
number that can be found in HAPROXY_PROCESSES. With this change the stats socket
in the configuration is less useful and everything can be done from the master
CLI.

With 2.7, the reload mechanism of the master CLI evolved, with previous versions,
this mechanism was asynchronous, so once the `reload` command was received, the
master would reload, the active master CLI connection was closed, and there was
no way to return a status as a response to the `reload` command. To achieve a
synchronous reload, a dedicated sockpair is used, one side uses a master CLI
applet and the other side wait to receive a socket. When the master CLI receives
the `reload` command, it takes the FD of the active master CLI session, sends it
in the socketpair and then does an exec. The FD is then stuck in the kernel
during the reload, because the poller is disabled. Once haproxy reloaded and the
poller active, the FD of the master CLI connection is received, so HAProxy can
reply a success or failure status for the reload. When built with
USE_SHM_OPEN=1, a shm is used to keep the warnings and errors when loading the
configuration in a shared buffer so this could survive the rexec in wait mode
and then be dumped as a response to the `reload` command after the status.

In 2.9 the master CLI command `hard-reload` was implemented, it works the same
way as the `reload` command, but instead of exec() with -sf for a soft-stop, it
starts with -st to achieve a hard stop on the previous worker.

Version 3.0 got rid of the libsystemd dependencies for sd_notify() after the
events of xz/openssh, the function is now implemented directly in haproxy in
src/systemd.c.

### mworker V3

This version was implemented with HAProxy 3.1, the goal was to stop parsing and
applying the configuration in the master process.

One of the caveats of the previous implementation was that the parser could take
a lot of time, and the master process would be stuck in the parser instead of
handling its polling loop, signals etc. Some parts of the configuration parsing
could also be less reliable with third-party code (EXTRA_OBJS), it could, for
example, allow opening FDs and not closing them before the reload which
would crash the master after a few reloads.

The startup of the master-worker was reorganized this way:

- the "discovery" mode, which is a lighter configuration parsing step, only
  applies the configuration which need to be effective for the master process.
  For example, "master-worker", "mworker-max-reloads" and less than 20 other
  keywords that are identified by KWF_DISCOVERY in the code. It is really fast
  as it don't need all the configuration to be applied in the master process.

- the master will then fork a worker, with a PROC_O_INIT flag. This worker has
  a temporary sockpair connected to the master CLI. Once the worker is forked,
  the master initializes its configuration and starts its polling loop.

- The newly forked worker will try to parse the configuration, which could
  result in a failure (exit 1), or any bad error code. In case of success, the
  worker will send a "READY" message to the master CLI then close this FD. At
  this step everything was initialized and the worker can enter its polling
  loop.

- The master then waits for the worker, it could:
   * receive the READY message over the mCLI, resulting in a successful loading
     of haproxy
   * receive a SIGCHLD, meaning the worker exited and couldn't load
