The socket-specific accept() code in listener_accept() has nothing to
do there. Let's move it to sock.c where it can be significantly cleaned
up. It will now directly return an accepted connection and provide a
status code instead of letting listener_accept() deal with various errno
values. Note that this doesn't support the sockpair specific code.
The function is now responsible for dealing with its own receiver's
polling state and calling fd_cant_recv() when facing EAGAIN.
One tiny change from the previous implementation is that the connection's
sockaddr is now allocated before trying accept(), which saves a memcpy()
of the resulting address for each accept at the expense of a cheap
pool_alloc/pool_free on the final accept returning EAGAIN. This still
apparently slightly improves accept performance in microbencharks.
This per-protocol function will be used to accept an incoming
connection and return it as a struct connection*. As such the protocol
stack's internal representation of a connection will not need to be
handled by the listener code.
This call was introduced by commit 5ced3e887 ("MINOR: sock: add
sock_accept_conn() to test a listening socket") but is actually quite
confusing because it makes one think the socket will accept a connection
(which is what we want to have in a new function) while it only tells
whether it's configured to accept connections. Let's call it
sock_accepting_conn() instead.
The same change was applied to sockpair which had the same issue.
accept_conn() will be used to accept an incoming connection and return it.
It will have to deal with various error codes. The currently identified
ones were created as CO_AC_*.
Now that this function is always called with an initialized connection
and that the control layer is always initialized, we don't need to play
games with fdtab[] to decide how to close, we can simply rely on the
regular close path using conn_ctrl_close(), which can be fused with
conn_xprt_close() into conn_full_close().
The code is cleaner because the FD is now used only for some
protocol-specific setup (that will eventually have to move) and to
try to send a hard-coded HTTP 500 error message on raw sockets.
Till now we would keep a per-thread queue of pending incoming connections
for which we would store:
- the listener
- the accepted FD
- the source address
- the source address' length
And these elements were first used in session_accept_fd() running on the
target thread to allocate a connection and duplicate them again. Doing
this induces various problems. The first one is that session_accept_fd()
may only run on file descriptors and cannot be reused for QUIC. The second
issue is that it induces lots of memory copies and that the listerner
queue thrashes a lot of cache, consuming 64 bytes per entry.
This patch changes this by allocating the connection before queueing it,
and by only placing the connection's pointer into the queue. Indeed, the
first two calls used to initialize the connection already store all the
information above, which can be retrieved from the connection pointer
alone. So we just have to pop one pointer from the target thread, and
pass it to session_accept_fd() which only needs the FD for the final
settings.
This starts to make the accept path a bit more transport-agnostic, and
saves memory and CPU cycles at the same time (1% connection rate increase
was noticed with 4 threads). Thanks to dividing the accept-queue entry
size from 64 to 8 bytes, its size could be increased from 256 to 1024
connections while still dividing the overall size by two. No single
queue full condition was met.
One minor drawback is that connection may be allocated from one thread's
pool to be used into another one. But this already happens a lot with
connection reuse so there is really nothing new here.
Roughly half of the calls to sockadr_alloc() are made to copy an already
known address. Let's optionally pass it in argument so that the function
can handle the copy at the same time, this slightly simplifies its usage.
fd_done_recv() used to be useful with the FD cache because it used to
allow to keep a file descriptor active in the poller without being
marked as ready in the cache, saving it from ringing immediately,
without incurring any system call. It was a way to make it yield
to wait for new events leaving a bit of time for others. The only
user left was the connection accepter (listen_accept()). We used
to suspect that with the FD cache removal it had become totally
useless since changing its readiness or not wouldn't change its
status regarding the poller itself, which would be the only one
deciding to report it again.
Careful tests showed that it indeed has exactly zero effect nowadays,
the syscall numbers are exactly the same with and without, including
when enabling edge-triggered polling.
Given that there's no more API available to manipulate it and that it
was directly called as an optimization from listener_accept(), it's
about time to remove it.
No protocol defines it anymore. The last user used to be the monitor-net
stuff that got partially broken already when the tcp_drain() function
moved to conn_sock_drain() with commit e215bba95 ("MINOR: connection:
make conn_sock_drain() work for all socket families") in 1.9-dev2.
A part of this will surely move back later when non-socket connections
arrive with QUIC but better keep the API clean and implement what's
needed in time instead.
As discussed here during 2.1-dev, "monitor-net" is totally obsolete:
https://www.mail-archive.com/haproxy@formilux.org/msg35204.html
It's fundamentally incompatible with usage of SSL, and imposes the
presence of file descriptors with hard-coded syscalls directly in the
generic accept path.
It's very unlikely that anyone has used it in the last 10 years for
anything beyond testing. In the worst case if anyone would depend
on it, replacing it with "http-request return status 200 if ..." and
"mode http" would certainly do the trick.
The keyword is still detected as special by the config parser to help
users update their configurations appropriately.
As discussed here during 2.1-dev, "mode health" is totally obsolete:
https://www.mail-archive.com/haproxy@formilux.org/msg35204.html
It's fundamentally incompatible with usage of SSL, doesn't support
source filtering, and imposes the presence of file descriptors with
hard-coded syscalls directly in the generic accept path.
It's very unlikely that anyone has used it in the last 10 years for
anything beyond testing. In the worst case if anyone would depend
on it, replacing it with "http-request return status 200" and "mode
http" would certainly do the trick.
The keyword is still detected as special by the config parser to help
users update their configurations appropriately.
FCGI mux is marked with HOL blocking. On safe reuse mode, the connection
using it are placed on the sessions instead of the available lists to
avoid sharing it with several clients. On detach, if they are no more
streams, remove the connection from the session before adding it to the
idle list. If there is still used streams, do not add it to available
list as it should be already on the session list.
H2 mux is marked with HOL blocking. On safe reuse mode, the connection
using it are placed on the sessions instead of the available lists to
avoid sharing it with several clients. On detach, if they are no more
streams, remove the connection from the session before adding it to the
idle list. If there is still used streams, do not add it to available
list as it should be already on the session list.
If a connection is using a mux protocol subject to HOL blocking, add it
to the session instead of the available list to avoid sharing it with
other clients on connection reuse.
When allocating a new session on connect_server, if the mux protocol is
marked as subject of HOL blocking, add it into session instead of
available list to avoid sharing it with other clients.
Do not check CO_FL_PRIVATE flag to check if the connection is in session
list on conn_free. This is necessary due to the future patches which add
server connections in the session list even if not private, if the mux
protocol is the subject of HOL blocking.
On server connection migration from one thread to another, the wrong
idle thread-specific counter is decremented. This bug was introduced
since commit 3d52f0f1f8 due to the
factorization with srv_use_idle_conn. However, this statement is only
executed from conn_backend_get. Extract the decrement from
srv_use_idle_conn in conn_backend_get and use the correct
thread-specific counter.
Rename the function to srv_use_conn to better reflect its purpose as it
is also used with a newly initialized connection not in the idle list.
As a side change, the connection insertion to available list has also
been extracted to conn_backend_get. This will be useful to be able to
specify an alternative list for protocol subject to HOL risk that should
not be shared between several clients.
This bug is only present in this release and thus do not need a backport.
The loop always missed one iteration due to the incrementation done on
the for check. Move the incrementation on the loop last statement to fix
this behaviour.
This bug has a very limited impact, not at all visible to the user, but
could be backported to 2.2.
When running a pure config check (haproxy -c) we go through the deinit
phase without having allocated fdtab, so we can't blindly dereference
it. The issue was added by recent commit ae7bc4a23 ("MEDIUM: deinit:
close all receivers/listeners before scanning proxies"), no backport is
needed.
In issue #894, Coverity suspects uninitialized values for a socket's
address whose family is AF_UNSPEC but it doesn't know that the address
is not used in this case. It's not on a critical path and working around
it is trivial, let's fully declare the address. We're doing it for both
TCP and UDP, because the same principle appears at two places.
This performs a connect(AF_UNSPEC) over an existing connection. This is
mainly for compatibility testing. At this step it only seems to work on
linux for TCP sockets (both listening and established), while SO_LINGER
successfully resets established connections on freebsd and aix.
It may happen that during a temporary listener pause resulting from a
SIGTTOU, one process gets one of its sockets disabled by another process
and will not be able to recover from this situation by itself. For the
protocols supporting this (TCPv4 and TCPv6 at the moment) this situation
is detectable, so when this happens, let's put the listener into the
PAUSED state so that it remains consistent with the real socket state.
One nice effect is that just sending the SIGTTIN signal to the process
is enough to recover the socket in this case.
There is no need to backport this, this behavior has been there forever
and the fix requires to reimplement the getsockopt() call there.
For socket pairs we don't rely on a real listening socket but we need to
have a properly connected UNIX stream socket. This is what the new
sockpair_accept_conn() tries to report. Some corner cases like half
shutdown will still not be detected but that should be sufficient for
most cases we really care about.
Now we introdce a new .rx_listening() function to report if a receiver is
actually a listening socket. The reason for this is to help detect shared
sockets that might have been broken by sibling processes.
At several places we need to check if a socket is still valid and still
willing to accept connections. Instead of open-coding this, each time,
let's add a new function for this.
Currently the suspend/resume mechanism for listeners only works on Linux
and we resort to a number of tricks involving shutdown+listen+shutdown
to try to detect failures on other operating systems that do not support
it. But on Linux connect(AF_UNSPEC) also works pretty well and is much
cleaner. It still doesn't work on other operating systems but the error
is easier to detect and appears safer. So let's switch to this.
When starting with a huge maxconn (say 1 billion), the only error seen
is "No polling mechanism available". This doesn't help at all to resolve
the problem. Let's add specific alerts for the failed mallocs. Now we can
get this instead:
[ALERT] 286/154439 (23408) : Not enough memory to allocate 2000000033 entries for fdtab!
This may be backported as far as 2.0 as it helps debugging bad configurations.
There are reports of a few "SC" in logs during reloads when H2 is used
on the backend side. Christopher analysed this as being caused by the
proxy disabled test in h2_process(). As the comment says, this was done
for frontends only, and must absolutely not send a GOAWAY to the backend,
as all it will result in is to make newly queued streams fail.
The fix consists in simply testing the connection side before deciding
to send the GOAWAY.
This may be backported as far as 2.0, though for whatever reason it seems
to manifest itself only since 2.2 (probably due to changes in the outgoing
connection setup sequence).
On some operating systems, RLIM_INFINITY is set to -1 so that when the
hard limit on the number of FDs is set to unlimited, taking the MAX
of both values keeps rlim_fd_cur and everything works. But on other
systems this values is defined as the highest positive integer. This
is what was observed on a 32-bit AIX 5.1. The effect is that maxsock
becomes 2^31-1 and that fdtab allocation fails.
Note that a simple workaround consists in manually setting maxconn in
the global section.
Let's ignore unlimited as soon as we retrieve rlim_fd_max so that all
systems behave consistently.
This may be backported as far as 2.0, though it doesn't seem like it
has annoyed anyone.
after 73b520b958 variables SSL_LIB, SSL_INC
are not set, but still used by BoringSSL builds. That leads to error
(I wish we could stop on such errors) and using stock openssl instead
of boringssl
Released version 2.3-dev6 with the following main changes :
- REGTESTS: use "command" instead of "which" for better POSIX compatibility
- BUILD: makefile: Update feature flags for OpenBSD
- DOC: agent-check: fix typo in "fail" word expected reply
- DOC: crt: advise to move away from cert bundle
- BUG/MINOR: ssl/crt-list: exit on warning out of crtlist_parse_line()
- REGTEST: fix host part in balance-uri-path-only.vtc
- REGTEST: make ssl_client_samples and ssl_server_samples requiret to 2.3
- REGTEST: the iif converter test requires 2.3
- REGTEST: make agent-check.vtc require 1.8
- REGTEST: make abns_socket.vtc require 1.8
- REGTEST: make map_regm_with_backref require 1.7
- BUILD: makefile: Update feature flags for FreeBSD
- OPTIM: backend/random: never queue on the server, always on the backend
- OPTIM: backend: skip LB when we know the backend is full
- BUILD: makefile: Fix building with closefrom() support enabled
- BUILD: makefile: add an EXTRAVERSION variable to ease local naming
- MINOR: tools: support for word expansion of environment in parse_line
- BUILD: tools: fix minor build issue on isspace()
- BUILD: makefile: Enable closefrom() support on Solaris
- CLEANUP: ssl: Use structured format for error line report during crt-list parsing
- MINOR: ssl: Add error if a crt-list might be truncated
- MINOR: ssl: remove uneeded check in crtlist_parse_file
- BUG/MINOR: Fix several leaks of 'log_tag' in init().
- DOC: tcp-rules: Refresh details about L7 matching for tcp-request content rules
- MEDIUM: tcp-rules: Warn if a track-sc* content rule doesn't depend on content
- BUG/MINOR: tcpcheck: Set socks4 and send-proxy flags before the connect call
- DOC: ssl: new "cert bundle" behavior
- BUG/MEDIUM: queue: make pendconn_cond_unlink() really thread-safe
- CLEANUP: ssl: "bundle" is not an OpenSSL wording
- MINOR: counters: fix a typo in comment
- BUG/MINOR: stats: fix validity of the json schema
- REORG: stats: export some functions
- MINOR: stats: add stats size as a parameter for csv/json dump
- MINOR: stats: hide px/sv/li fields in applet struct
- REORG: stats: extract proxy json dump
- REORG: stats: extract proxies dump loop in a function
- MINOR: hlua: Display debug messages on stderr only in debug mode
- MINOR: stats: define the concept of domain for statistics
- MINOR: stats: define additional flag px cap on domain
- MEDIUM: stats: add delimiter for static proxy stats on csv
- MEDIUM: stats: define an API to register stat modules
- MEDIUM: stats: add abstract type to store counters
- MEDIUM: stats: integrate static proxies stats in new stats
- MINOR: stats: support clear counters for dynamic stats
- MINOR: stats: display extra proxy stats on the html page
- MINOR: stats: add config "stats show modules"
- MINOR: dns/stats: integrate dns counters in stats
- MINOR: stats: remove for loop declaration
- DOC: ssl: fix typo about ocsp files
- BUG/MINOR: peers: Inconsistency when dumping peer status codes.
- DOC: update INSTALL with supported OpenBSD / FreeBSD versions
- BUG/MINOR: proto_tcp: Report warning messages when listeners are bound
- CLEANUP: cache: Fix leak of cconf->c.name during config check
- CLEANUP: ssl: Release cached SSL sessions on deinit
- BUG/MINOR: mux-h1: Be sure to only set CO_RFL_READ_ONCE for the first read
- BUG/MINOR: mux-h1: Always set the session on frontend h1 stream
- MINOR: mux-h1: Don't wakeup the H1C when output buffer become available
- CLEANUP: sock-unix: Remove an unreachable goto clause
- BUG/MINOR: proxy: inc req counter on new syslog messages.
- BUG/MEDIUM: log: old processes with log foward section don't die on soft stop.
- MINOR: stats: inc req counter on listeners.
- MINOR: channel: new getword and getchar functions on channel.
- MEDIUM: log: syslog TCP support on log forward section.
- BUG/MINOR: proxy/log: frontend/backend and log forward names must differ
- DOC: re-work log forward bind statement documentation.
- DOC: fix a confusing typo on a regsub example
- BUILD: Add a DragonFlyBSD target
- BUG/MINOR: makefile: fix a tiny typo in the target list
- BUILD: makefile: Update feature flags for NetBSD
- CI: travis-ci: help Coverity to detect BUG_ON() as a real stop
- DOC: Add missing stats fields in the management doc
- BUG/MEDIUM: mux-fcgi: Don't handle pending read0 too early on streams
- BUG/MEDIUM: mux-h2: Don't handle pending read0 too early on streams
- DOC: Fix typos in configuration.txt
- BUG/MINOR: http: Fix content-length of the default 500 error
- BUG/MINOR: http-htx: Expect no body for 204/304 internal HTTP responses
- REGTESTS: mark abns_socket as broken
- MEDIUM: fd: always wake up one thread when enabling a foreing FD
- MEDIUM: listeners: don't bounce listeners management between queues
- MEDIUM: init: stop disabled proxies after initializing fdtab
- MEDIUM: listeners: make unbind_listener() converge if needed
- MEDIUM: deinit: close all receivers/listeners before scanning proxies
- MEDIUM: listeners: remove the now unused ZOMBIE state
- MINOR: listeners: do not uselessly try to close zombie listeners in soft_stop()
- CLEANUP: proxy: remove the first_to_listen hack in zombify_proxy()
- MINOR: listeners: introduce listener_set_state()
- MINOR: proxy: maintain per-state counters of listeners
- MEDIUM: proxy: remove the unused PR_STFULL state
- MEDIUM: proxy: remove the PR_STERROR state
- MEDIUM: proxy: remove state PR_STPAUSED
- MINOR: startup: don't rely on PR_STNEW to check for listeners
- CLEANUP: peers: don't use the PR_ST* states to mark enabled/disabled
- MEDIUM: proxy: replace proxy->state with proxy->disabled
- MEDIUM: proxy: remove start_proxies()
- MEDIUM: proxy: merge zombify_proxy() with stop_proxy()
- MINOR: listeners: check the current listener state in pause_listener()
- MINOR: listeners: check the current listener earlier state in resume_listener()
- MEDIUM: listener/proxy: make the listeners notify about proxy pause/resume
- MINOR: protocol: introduce protocol_{pause,resume}_all()
- MAJOR: signals: use protocol_pause_all() and protocol_resume_all()
- CLEANUP: proxy: remove the now unused pause_proxies() and resume_proxies()
- MEDIUM: proto_tcp: make the pause() more robust in multi-process
- BUG/MEDIUM: listeners: correctly report pause() errors
- MINOR: listeners: move fd_stop_recv() to the receiver's socket code
- CLEANUP: protocol: remove the ->disable_all method
- CLEANUP: listeners: remove unused disable_listener and disable_all_listeners
- MINOR: listeners: export enable_listener()
- MINOR: protocol: directly call enable_listener() from protocol_enable_all()
- CLEANUP: protocol: remove the ->enable_all method
- CLEANUP: listeners: remove the now unused enable_all_listeners()
- MINOR: protocol: rename the ->listeners field to ->receivers
- MINOR: protocol: replace ->pause(listener) with ->rx_suspend(receiver)
- MINOR: protocol: implement an ->rx_resume() method
- MINOR: listener: use the protocol's ->rx_resume() method when available
- MINOR: sock: provide a set of generic enable/disable functions
- MINOR: protocol: add a new pair of rx_enable/rx_disable methods
- MINOR: protocol: add a new pair of enable/disable methods for listeners
- MEDIUM: listeners: now use the listener's ->enable/disable
- MINOR: listeners: split delete_listener() in two versions
- MINOR: listeners: count unstoppable jobs on creation, not deletion
- MINOR: listeners: add a new stop_listener() function
- MEDIUM: proxy: make stop_proxy() now use stop_listener()
- MEDIUM: proxy: add mode PR_MODE_PEERS to flag peers frontends
- MEDIUM: proxy: centralize proxy status update and reporting
- MINOR: protocol: add protocol_stop_now() to instant-stop listeners
- MEDIUM: proxy: make soft_stop() stop most listeners using protocol_stop_now()
- MEDIUM: udp: implement udp_suspend() and udp_resume()
- MINOR: listener: add a few BUG_ON() statements to detect inconsistencies
- MEDIUM: listeners: always close master vs worker listeners
- BROKEN/MEDIUM: listeners: rework the unbind logic to make it idempotent
- MEDIUM: listener: let do_unbind_listener() decide whether to close or not
- CLEANUP: listeners: remove the do_close argument to unbind_listener()
- MINOR: listeners: move the LI_O_MWORKER flag to the receiver
- MEDIUM: receivers: add an rx_unbind() method in the protocols
- MINOR: listeners: split do_unbind_listener() in two
- MEDIUM: listeners: implement protocol level ->suspend/resume() calls
- MEDIUM: config: mark "grace" as deprecated
- MEDIUM: config: remove the deprecated and dangerous global "debug" directive
- BUG/MINOR: proxy: respect the proper format string in sig_pause/sig_listen
- MINOR: peers: heartbeat, collisions and handshake information for "show peers" command.
- BUILD: makefile: Enable getaddrinfo() on OS/X
This patch adds "coll" new counter and the heartbeat timer values to "show peers"
command. It also adds the elapsed time since the last handshake to new "last_hdshk"
new peer dump field.
When factoring out the pause/resume error messages in commit 775e00158
("MAJOR: signals: use protocol_pause_all() and protocol_resume_all()")
I forgot that ha_warning() and send_log() take a format string and not
just a const string. No backport is needed, this is 2.3-dev.
This one was scheduled for removal in 2.3 since 2.2-dev3 by commit
1b85785bc ("MINOR: config: mark global.debug as deprecated"). Let's
remove it now. It remains totally possible to use -d on the command
line though.
This was introduced 15 years ago or so to delay the stopping of some
services so that a monitoring device could detect its port being down
before services were stopped. Since then, clean reloads were implemented
and this doesn't cope well with reload at all, preventing the new process
from seamlessly binding, and forcing processes to coexist with half-baked
configurations.
Now it has become a real problem because there's a significant code
portion in the proxies that is solely dedicated to this obsolete feature,
and dealing with its special cases eases the introduction of bugs in
other places so it's about time that it goes.
We could tentatively schedule its removal for 2.4 with a hard deadline
for 2.5 in any case.
Now we have ->suspend() and ->resume() for listeners at the protocol
level. This means that it now becomes possible for a protocol to redefine
its own way to suspend and resume. The default functions are provided for
TCP, UDP and unix, and they are pass-through to the receiver equivalent
as it used to be till now. Nothing was defined for sockpair since it does
not need to suspend/resume during reloads, hence it will succeed.
The inner part now goes into the protocol and is used to decide how to
unbind a given protocol's listener. The existing code which is able to
also unbind the receiver was provided as a default function that we
currently use everywhere. Some complex listeners like QUIC will use this
to decide how to unbind without impacting existing connections, possibly
by setting up other incoming paths for the traffic.
This is used as a generic way to unbind a receiver at the end of
do_unbind_listener(). This allows to considerably simplify that function
since we can now let the protocol perform the cleanup. The generic code
was moved to sock.c, along with the conditional rx_disable() call. Now
the code also supports that the ->disable() function of the protocol
which acts on the listener performs the close itself and adjusts the
RX_F_BUOND flag accordingly.
This listener flag indicates whether the receiver part of the listener
is specific to the master or to the workers. In practice it's only used
by the master's CLI right now. It's used to know whether or not the FD
must be closed before forking the workers. For this reason it's way more
of a receiver's property than a listener's property, so let's move it
there under the name RX_F_MWORKER. The rest of the code remains
unchanged.
And also remove it from its callers. This subtle distinction was added as
sort of a hack for the seamless reload feature but is not needed anymore
since the do_close turned unused since commit previous commit ("MEDIUM:
listener: let do_unbind_listener() decide whether to close or not").
This also removes the unbind_listener_no_close() function.
The listener contains all the information needed to decide to close on
unbind or not. The rule is the following (when we're not stopping):
- worker process unbinding from a worker's FD with socket transfer enabled => keep
- master process unbinding from a master's inherited FD => keep
- master process unbinding from a master's FD => close
- master process unbinding from a worker's FD => close
- worker process unbinding from a master's FD => close
- worker process unbinding from a worker's FD => close
Let's translate that into the function and stop using the do_close
argument that is a bit obscure for callers. It was not yet removed
to ease code testing.
BROKEN: the failure rate on reg-tests/seamless-reload/abns_socket.vtc has
significantly increased for no obvious reason. It fails 99% of the time vs
10% before.
do_unbind_listener() is not logical and is not even idempotent. It must
not touch the fd if already -1, which also means not touch the receiver.
In addition, when performing a partial stop on a socket (not closing),
we know the socket remains in the listening state yet it's marked as
LI_ASSIGNED, which is confusing as it doesn't translate its real state.
With this change, we make sure that FDs marked for close end up in
ASSIGNED state and that those which are really bound and on which a
listen() was made (i.e. not pause) remain in LISTEN state. This is what
is closest to reality.
Ideally this function should become a default proto->unbind() one but
it may still keep a bit too much state logic to become generalized to
other protocols (e.g. QUIC).
Right now in enable_listener(), we used to start all enabled
listeners then kill from the workers those that were for the master.
But this is incomplete. We must also close from the master the
listeners that are solely for workers, and do it before we even
start them. Otherwise we end up with a master responding to the
worker CLI connections if the listener remains in listen mode to
translate the socket's real state.
It doesn't seem like it could have caused bugs in the past because we
used to aggressively mark disabled listeners as LI_ASSIGNED despite
the fact that they were still bound and listening. If this patch were
ever seen as a candidate solution for any obscure bug, be careful in
that it subtly relies on the fact that fd_delete() doesn't close
inherited FDs anymore, otherwise that could break the master's ability
to pass inherited FDs on reloads.
