Commit Graph

4569 Commits

Author SHA1 Message Date
Willy Tarreau
18c20d28d7 MINOR: listeners: move the LI_O_MWORKER flag to the receiver
This listener flag indicates whether the receiver part of the listener
is specific to the master or to the workers. In practice it's only used
by the master's CLI right now. It's used to know whether or not the FD
must be closed before forking the workers. For this reason it's way more
of a receiver's property than a listener's property, so let's move it
there under the name RX_F_MWORKER. The rest of the code remains
unchanged.
2020-10-09 18:43:05 +02:00
Willy Tarreau
75c98d166e CLEANUP: listeners: remove the do_close argument to unbind_listener()
And also remove it from its callers. This subtle distinction was added as
sort of a hack for the seamless reload feature but is not needed anymore
since the do_close turned unused since commit previous commit ("MEDIUM:
listener: let do_unbind_listener() decide whether to close or not").
This also removes the unbind_listener_no_close() function.
2020-10-09 18:41:56 +02:00
Willy Tarreau
02e8557e88 MINOR: protocol: add protocol_stop_now() to instant-stop listeners
This will instantly stop all listeners except those which belong to
a proxy configured with a grace time. This means that UDP listeners,
and peers will also be stopped when called this way.
2020-10-09 18:29:04 +02:00
Willy Tarreau
acde152175 MEDIUM: proxy: centralize proxy status update and reporting
There are multiple ways a proxy may switch to the disabled state,
but now it's essentially once it loses its last listener. Instead
of keeping duplicate code around and reporting the state change
before actually seeing it, we now report it at the moment it's
performed (from the last listener leaving) which allows to remove
the message from all other places.
2020-10-09 18:29:04 +02:00
Willy Tarreau
a389c9e1e3 MEDIUM: proxy: add mode PR_MODE_PEERS to flag peers frontends
For now we cannot easily distinguish a peers frontend from another one,
which will be problematic to avoid reporting them when stopping their
listeners. Let's add PR_MODE_PEERS for this. It's not supposed to cause
any issue since all non-HTTP proxies are handled similarly now.
2020-10-09 18:28:21 +02:00
Willy Tarreau
caa7df1296 MINOR: listeners: add a new stop_listener() function
This function will be used to definitely stop a listener (e.g. during a
soft_stop). This is actually tricky because it may be called for a proxy
or for a protocol, both of which require locks and already hold some. The
function takes booleans indicating which ones are already held, hoping
this will be enough. It's not well defined wether proto->disable() and
proto->rx_disable() are supposed to be called with any lock held, and
they are used from do_unbind_listener() with all these locks. Some back
annotations ought to be added on this point.

The proxy's listeners count is updated, and the proxy is marked as
disabled and woken up after the last one is gone. Note that a
listener in listen state is already not attached anymore since it
was disabled.
2020-10-09 18:27:48 +02:00
Willy Tarreau
b4c083f5bf MINOR: listeners: split delete_listener() in two versions
We'll need an already locked variant of this function so let's make
__delete_listener() which will be called with the protocol lock held
and the listener's lock held.
2020-10-09 11:27:30 +02:00
Willy Tarreau
5ddf1ce9c4 MINOR: protocol: add a new pair of enable/disable methods for listeners
These methods will be used to enable/disable accepting new connections
so that listeners do not play with FD directly anymore. Since all the
currently supported protocols work on socket for now, these are identical
to the rx_enable/rx_disable functions. However they were not defined in
sock.c since it's likely that some will quickly start to differ. At the
moment they're not used.

We have to take care of fd_updt before calling fd_{want,stop}_recv()
because it's allocated fairly late in the boot process and some such
functions may be called very early (e.g. to stop a disabled frontend's
listeners).
2020-10-09 11:27:30 +02:00
Willy Tarreau
686fa3db50 MINOR: protocol: add a new pair of rx_enable/rx_disable methods
These methods will be used to enable/disable rx at the receiver level so
that callers don't play with FDs directly anymore. All our protocols use
the generic ones from sock.c at the moment. For now they're not used.
2020-10-09 11:27:30 +02:00
Willy Tarreau
e70c7977f2 MINOR: sock: provide a set of generic enable/disable functions
These will be used on receivers, to enable or disable receiving on a
listener, which most of the time just consists in enabling/disabling
the file descriptor.

We have to take care of the existence of fd_updt to know if we may
or not call fd_{want,stop}_recv() since it's not permitted in very
early boot.
2020-10-09 11:27:30 +02:00
Willy Tarreau
58e6b71bb0 MINOR: protocol: implement an ->rx_resume() method
This one undoes ->rx_suspend(), it tries to restore an operational socket.
It was only implemented for TCP since it's the only one we support right
now.
2020-10-09 11:27:30 +02:00
Willy Tarreau
cb66ea60cf MINOR: protocol: replace ->pause(listener) with ->rx_suspend(receiver)
The ->pause method is inappropriate since it doesn't exactly "pause" a
listener but rather temporarily disables it so that it's not visible at
all to let another process take its place. The term "suspend" is more
suitable, since the "pause" is actually what we'll need to apply to the
FULL and LIMITED states which really need to make a pause in the accept
process. And it goes well with the use of the "resume" function that
will also need to be made per-protocol.

Let's rename the function and make it act on the receiver since it's
already what it essentially does, hence the prefix "_rx" to make it
more explicit.

The protocol struct was a bit reordered because it was becoming a real
mess between the parts related to the listeners and those for the
receivers.
2020-10-09 11:27:30 +02:00
Willy Tarreau
d7f331c8b8 MINOR: protocol: rename the ->listeners field to ->receivers
Since the listeners were split into receiver+listener, this field ought
to have been renamed because it's confusing. It really links receivers
and not listeners, as most of the time it's used via rx.proto_list!
The nb_listeners field was updated accordingly.
2020-10-09 11:27:30 +02:00
Willy Tarreau
dae0692717 CLEANUP: listeners: remove the now unused enable_all_listeners()
It's not used anymore since previous commit. The good thing is that
no more listener function now directly acts on a protocol.
2020-10-09 11:27:30 +02:00
Willy Tarreau
078e1c7102 CLEANUP: protocol: remove the ->enable_all method
It's not used anymore, now the listeners are enabled from
protocol_enable_all().
2020-10-09 11:27:30 +02:00
Willy Tarreau
7834a3f70f MINOR: listeners: export enable_listener()
we'll soon call it from outside.
2020-10-09 11:27:30 +02:00
Willy Tarreau
d008009958 CLEANUP: listeners: remove unused disable_listener and disable_all_listeners
These ones have never been called, they were referenced by the protocol's
disable_all for some protocols but there are no traces of their use, so
in addition to not being sure the code works, it has never been tested.
Let's remove a bit of complexity starting from there.
2020-10-09 11:27:30 +02:00
Willy Tarreau
fb4ead8e8a CLEANUP: protocol: remove the ->disable_all method
This one has never been used, is only referenced by proto_uxst and
proto_sockpair, and it's not even certain it works at all. Let's
get rid of it.
2020-10-09 11:27:30 +02:00
Willy Tarreau
1accacbcc3 CLEANUP: proxy: remove the now unused pause_proxies() and resume_proxies()
They're not used anymore, delete them before someone thinks about using
them again!
2020-10-09 11:27:30 +02:00
Willy Tarreau
09819d1118 MINOR: protocol: introduce protocol_{pause,resume}_all()
These two functions are used to pause and resume all listeners of
all protocols. They use the standard listener functions for this
so they're supposed to handle the situation gracefully regardless
of the upper proxies' states, and they will report completion on
proxies once the switch is performed.

It might be nice to define a particular "failed" state for listeners
that cannot resume and to count them on proxies in order to mention
that they're definitely stuck. On the other hand, the current
situation is retryable which is quite appreciable as well.
2020-10-09 11:27:30 +02:00
Willy Tarreau
337c835d16 MEDIUM: proxy: merge zombify_proxy() with stop_proxy()
The two functions don't need to be distinguished anymore since they have
all the necessary info to act as needed on their listeners. Let's just
pass via stop_proxy() and make it check for each listener which one to
close or not.
2020-10-09 11:27:30 +02:00
Willy Tarreau
43ba3cf2b5 MEDIUM: proxy: remove start_proxies()
Its sole remaining purpose was to display "proxy foo started", which
has little benefit and pollutes output for those with plenty of proxies.
Let's remove it now.

The VTCs were updated to reflect this, because many of them had explicit
counts of dropped lines to match this message.

This is tagged as MEDIUM because some users may be surprized by the
loss of this quite old message.
2020-10-09 11:27:30 +02:00
Willy Tarreau
c3914d4fff MEDIUM: proxy: replace proxy->state with proxy->disabled
The remaining proxy states were only used to distinguish an enabled
proxy from a disabled one. Due to the initialization order, both
PR_STNEW and PR_STREADY were equivalent after startup, and they
would only differ from PR_STSTOPPED when the proxy is disabled or
shutdown (which is effectively another way to disable it).

Now we just have a "disabled" field which allows to distinguish them.
It's becoming obvious that start_proxies() is only used to print a
greeting message now, that we'd rather get rid of. Probably that
zombify_proxy() and stop_proxy() should be merged once their
differences move to the right place.
2020-10-09 11:27:30 +02:00
Willy Tarreau
1ad64acf6c CLEANUP: peers: don't use the PR_ST* states to mark enabled/disabled
The enabled/disabled config options were stored into a "state" field
that is an integer but contained only PR_STNEW or PR_STSTOPPED, which
is a bit confusing, and causes a dependency with proxies. This was
renamed to "disabled" and is used as a boolean. The field was also
moved to the end of the struct to stop creating a hole and fill another
one.
2020-10-09 11:27:30 +02:00
Willy Tarreau
f18d968830 MEDIUM: proxy: remove state PR_STPAUSED
This state was used to mention that a proxy was in PAUSED state, as opposed
to the READY state. This was causing some trouble because if a listener
failed to resume (e.g. because its port was temporarily in use during the
resume), it was not possible to retry the operation later. Now by checking
the number of READY or PAUSED listeners instead, we can accurately know if
something went bad and try to fix it again later. The case of the temporary
port conflict during resume now works well:

  $ socat readline /tmp/sock1
  prompt
  > disable frontend testme3

  > disable frontend testme3
  All sockets are already disabled.

  > enable frontend testme3
  Failed to resume frontend, check logs for precise cause (port conflict?).

  > enable frontend testme3

  > enable frontend testme3
  All sockets are already enabled.
2020-10-09 11:27:30 +02:00
Willy Tarreau
a17c91b37f MEDIUM: proxy: remove the PR_STERROR state
This state is only set when a pause() fails but isn't even set when a
resume() fails. And we cannot recover from this state. Instead, let's
just count remaining ready listeners to decide to emit an error or not.
It's more accurate and will better support new attempts if needed.
2020-10-09 11:27:30 +02:00
Willy Tarreau
6b3bf733dd MEDIUM: proxy: remove the unused PR_STFULL state
Since v1.4 or so, it's almost not possible anymore to set this state. The
only exception is by using the CLI to change a frontend's maxconn setting
below its current usage. This case makes no sense, and for other cases it
doesn't make sense either because "full" is a vague concept when only
certain listeners are full and not all. Let's just remove this unused
state and make it clear that it's not reported. The "ready" or "open"
states will continue to be reported without being misleading as they
will be opposed to "stop".
2020-10-09 11:27:30 +02:00
Willy Tarreau
efc0eec4c1 MINOR: proxy: maintain per-state counters of listeners
The proxy state tries to be synthetic but that doesn't work well with
many listeners, especially for transition phases or after a failed
pause/resume.

In order to address this, we'll instead rely on counters of listeners in
a given state for the 3 major states (ready, paused, listen) and a total
counter. We'll now be able to determine a proxy's state by comparing these
counters only.
2020-10-09 11:27:30 +02:00
Willy Tarreau
a37b244509 MINOR: listeners: introduce listener_set_state()
This function is used as a wrapper to set a listener's state everywhere.
We'll use it later to maintain some counters in a consistent state when
switching state so it's capital that all state changes go through it.
No functional change was made beyond calling the wrapper.
2020-10-09 11:27:30 +02:00
Willy Tarreau
c6dac6c7f5 MEDIUM: listeners: remove the now unused ZOMBIE state
The zombie state is not used anymore by the listeners, because in the
last two cases where it was tested it couldn't match as it was covered
by the test on the process mask. Instead now the FD is either in the
LISTEN state or the INIT state. This also avoids forcing the listener
to be single-dimensional because actually belonging to another process
isn't totally exclusive with the other states, which explains some of
the difficulties requiring to check the proc_mask and the fd sometimes.

So let's get rid of it now not to be tempted to reuse it.

The doc on the listeners state was updated.
2020-10-09 11:27:29 +02:00
Emeric Brun
b0c331f71f BUG/MINOR: proxy/log: frontend/backend and log forward names must differ
This patch disallow to use same name for a log forward section
and a frontend/backend section.
2020-10-08 08:53:26 +02:00
Emeric Brun
6d75616951 MINOR: channel: new getword and getchar functions on channel.
This patch adds two new functions to get a char
or a word from a channel.
2020-10-07 17:17:27 +02:00
Emeric Brun
2897644ae5 MINOR: stats: inc req counter on listeners.
This patch enables count of requests for listeners
if listener's counters are enabled.
2020-10-07 17:17:27 +02:00
Amaury Denoyelle
fbd0bc98fe MINOR: dns/stats: integrate dns counters in stats
Use the new stats module API to integrate the dns counters in the
standard stats. This is done in order to avoid code duplication, keep
the code related to cli out of dns and use the full possibility of the
stats function, allowing to print dns stats in csv or json format.
2020-10-05 12:02:14 +02:00
Amaury Denoyelle
0b70a8a314 MINOR: stats: add config "stats show modules"
By default, hide the extra statistics on the html page. Define a new
flag STAT_SHMODULES which is activated if the config "stats show
modules" is set.
2020-10-05 12:02:14 +02:00
Amaury Denoyelle
d3700a7fda MINOR: stats: support clear counters for dynamic stats
Add a boolean 'clearable' on stats module structure. If set, it forces
all the counters to be reset on 'clear counters' cli command. If not,
the counters are reset only when 'clear counters all' is used.
2020-10-05 12:02:14 +02:00
Amaury Denoyelle
ee63d4bd67 MEDIUM: stats: integrate static proxies stats in new stats
This is executed on startup with the registered statistics module. The
existing statistics have been merged in a list containing all
statistics for each domain. This is useful to print all available
statistics in a generic way.

Allocate extra counters for all proxies/servers/listeners instances.
These counters are allocated with the counters from the stats modules
registered on startup.
2020-10-05 12:02:14 +02:00
Amaury Denoyelle
730c727ea3 MEDIUM: stats: add abstract type to store counters
Implement a small API to easily add extra counters inside a structure
instance. This will be used to implement dynamic statistics linked on
every type of object as needed.

The counters are stored in a dynamic array inside the relevant objects.
2020-10-05 12:02:14 +02:00
Amaury Denoyelle
58d395e0d6 MEDIUM: stats: define an API to register stat modules
A stat module can be registered to quickly add new statistics on
haproxy. It must be attached to one of the available stats domain. The
register must be done using INITCALL on STG_REGISTER.

The stat module has a name which should be unique for each new module in
a domain. It also contains a statistics list with their name/desc and a
pointer to a function used to fill the stats from the module counters.

The module also provides the initial counters values used on
automatically allocated counters. The offset for these counters
are stored in the module structure.
2020-10-05 12:02:14 +02:00
Amaury Denoyelle
72b16e5173 MINOR: stats: define additional flag px cap on domain
This flag can be used to determine on what type of proxy object the
statistics should be relevant. It will be useful when adding dynamic
statistics. Currently, this flag is not used.
2020-10-05 12:02:14 +02:00
Amaury Denoyelle
072f97eddf MINOR: stats: define the concept of domain for statistics
The domain option will be used to have statistics attached to other
objects than proxies/listeners/servers. At the moment, only the PROXY
domain is available.

Add an argument 'domain' on the 'show stats' cli command to specify the
domain. Only 'domain proxy' is available now. If not specified, proxy
will be considered the default domain.

For HTML output, only proxy statistics will be displayed.
2020-10-05 12:02:14 +02:00
Amaury Denoyelle
da5b6d1cd9 MINOR: stats: hide px/sv/li fields in applet struct
Use an opaque pointer to store proxy instance. Regroup server/listener
as a single opaque pointer. This has the benefit to render the structure
more evolutive to support statistics on other types of objects in the
future.

This patch is needed to extend stat support for components other than
proxies objects.

The prometheus module has been adapted for these changes.
2020-10-05 10:48:58 +02:00
Amaury Denoyelle
97323c9ed4 MINOR: stats: add stats size as a parameter for csv/json dump
Render the stats size parametric in csv/json dump functions. This is
needed for the future patch which provides dynamic stats. For now the
static value ST_F_TOTAL_FIELDS is provided.

Remove unused parameter px on stats_dump_one_line.

This patch is needed to extend stat support to components other than
proxies objects.
2020-10-05 09:06:10 +02:00
Amaury Denoyelle
3ca927e68f REORG: stats: export some functions
Un-mark stats_dump_one_line and stats_putchk as static and export them
in the header file. These functions will be reusable by other components to
print their statistics.

This patch is needed to extend stat support to components other than
proxies objects.
2020-10-05 09:06:10 +02:00
Amaury Denoyelle
cd3de50779 MINOR: counters: fix a typo in comment
Wrong copy/paste comment, replace listeners/frontends by
servers/backends

This may be backported up to 1.7.
2020-10-05 09:05:57 +02:00
Willy Tarreau
fac0f645df BUG/MEDIUM: queue: make pendconn_cond_unlink() really thread-safe
A crash reported in github issue #880 looks impossible unless
pendconn_cond_unlink() occasionally sees a null leaf_p when attempting
to remove an entry, which seems to be confirmed by the reporter. What
seems to be happening is that depending on compiler optimizations,
this pointer can appear as null while pointers are moved if one of
the node's parents is removed from or inserted into the tree. There's
no explicit null of the pointer during these operations but those
pointers are rewritten in multiple steps and nothing prevents this
situation from happening, and there are no particular barrier nor
atomic ops around this.

This test was used to avoid unnecessary locking, for already deleted
entries, but looking at the code it appears that pendconn_free() already
resets s->pend_pos that's used as <p> there, and that the other call
reasons are after an error where the connection will be dropped as
well. So we don't save anything by doing this test, and make it
unsafe. The older code used to check for list emptiness there and
not inside pendconn_unlink(), which explains why the code has stayed
there. Let's just remove this now.

Thanks to @jaroslawr for reporting this issue in great details and for
testing the proposed fix.

This should be backpored to 1.8, where the test on LIST_ISEMPTY should
be moved to pendconn_unlink() instead (inside the lock, just like 2.0+).
2020-10-02 18:10:26 +02:00
Amaury Denoyelle
fa41cb6792 MINOR: tools: support for word expansion of environment in parse_line
Allow the syntax "${...[*]}" to expand an environment variable
containing several values separated by spaces as individual arguments. A
new flag PARSE_OPT_WORD_EXPAND has been added to toggle this feature on
parse_line invocation. In case of an invalid syntax, a new error
PARSE_ERR_WRONG_EXPAND will be triggered.

This feature has been asked on the github issue #165.
2020-10-01 17:24:14 +02:00
Willy Tarreau
3ca2365904 BUG/MEDIUM: h2: report frame bits only for handled types
As part of his GREASE experiments on Chromium, Bence Bky reported in
https://lists.w3.org/Archives/Public/ietf-http-wg/2020JulSep/0202.html
and https://bugs.chromium.org/p/chromium/issues/detail?id=1127060 that
a certain combination of frame type and frame flags was causing an error
on app.slack.com. It turns out that it's haproxy that is causing this
issue because the frame type is wrongly assumed to support padding, the
frame flags indicate padding is present, and the frame is too short for
this, resulting in an error.

The reason why only some frame types are affected is due to the frame
type being used in a bit shift to match against a mask, and where the
5 lower bits of the frame type only are used to compute the frame bit.
If the resulting frame bit matches a DATA, HEADERS or PUSH_PROMISE frame
bit, then padding support is assumed and the test is enforced, resulting
in a PROTOCOL_ERROR or FRAME_SIZE_ERROR depending on the payload size.

We must never match any such bit for unsupported frame types so let's
add a check for this. This must be backported as far as 1.8.

Thanks to Cooper Bethea for providing enough context to help narrow the
issue down and to Bence Bky for creating a simple reproducer.
2020-09-18 08:05:03 +02:00
Willy Tarreau
2b5e0d8b6a MEDIUM: proto_udp: replace last AF_CUST_UDP* with AF_INET*
We don't need to cheat with the sock_domain anymore, we now always have
the SOCK_DGRAM sock_type as a complementary selector. This patch restores
the sock_domain to AF_INET* in the udp* protocols and removes all traces
of the now unused AF_CUST_*.
2020-09-16 22:08:08 +02:00
Willy Tarreau
910c64da96 MEDIUM: protocol: store the socket and control type in the protocol array
The protocol array used to be only indexed by socket family, which is very
problematic with UDP (requiring an extra family) and with the forthcoming
QUIC (also requiring an extra family), especially since that binds them to
certain families, prevents them from supporting dgram UNIX sockets etc.

In order to address this, we now start to register the protocols with more
info, namely the socket type and the control type (either stream or dgram).
This is sufficient for the protocols we have to deal with, but could also
be extended further if multiple protocol variants were needed. But as is,
it still fits nicely in an array, which is convenient for lookups that are
instant.
2020-09-16 22:08:08 +02:00
Willy Tarreau
a54553f74f MINOR: protocol: add the control layer type in the protocol struct
This one will be needed to more accurately select a protocol. It may
differ from the socket type for QUIC, which uses dgram at the socket
layer and provides stream at the control layer. The upper level requests
a control layer only so we need this field.
2020-09-16 22:08:08 +02:00
Willy Tarreau
65ec4e3ff7 MEDIUM: tools: make str2sa_range() check that the protocol has ->connect()
Most callers of str2sa_range() need the protocol only to check that it
provides a ->connect() method. It used to be used to verify that it's a
stream protocol, but it might be a bit early to get rid of it. Let's keep
the test for now but move it to str2sa_range() when the new flag PA_O_CONNECT
is present. This way almost all call places could be cleaned from this.

There's a strange test in the server address parsing code that rechecks
the family from the socket which seems to be a duplicate of the previously
removed tests. It will have to be rechecked.
2020-09-16 22:08:08 +02:00
Willy Tarreau
5fc9328aa2 MINOR: tools: make str2sa_range() directly return the protocol
We'll need this so that it can return pointers to stacked protocol in
the future (for QUIC). In addition this removes a lot of tests for
protocol validity in the callers.

Some of them were checked further apart, or after a call to
str2listener() and they were simplified as well.

There's still a trick, we can fail to return a protocol in case the caller
accepts an fqdn for use later. This is what servers do and in this case it
is valid to return no protocol. A typical example is:

   server foo localhost:1111
2020-09-16 22:08:08 +02:00
Willy Tarreau
9b3178df23 MINOR: listener: pass the chosen protocol to create_listeners()
The function will need to use more than just a family, let's pass it
the selected protocol. The caller will then be able to do all the fancy
stuff required to pick the best protocol.
2020-09-16 22:08:08 +02:00
Willy Tarreau
aa333123f2 MINOR: cfgparse: add str2receiver() to parse dgram receivers
This is at least temporary, as the migration at once is way too difficuly.
For now it still creates listeners but only allows DGRAM sockets. This
aims at easing the split between listeners and receivers.
2020-09-16 22:08:08 +02:00
Willy Tarreau
a93e5c7fae MINOR: tools: make str2sa_range() optionally return the fd
If a file descriptor was passed, we can optionally return it. This will
be useful for listening sockets which are both a pre-bound FD and a ready
socket.
2020-09-16 22:08:08 +02:00
Willy Tarreau
909c23b086 MINOR: listener: remove the inherited arg to create_listener()
This argument can now safely be determined from fd != -1, let's just
drop it.
2020-09-16 22:08:08 +02:00
Willy Tarreau
328199348b MINOR: tools: add several PA_O_* flags in str2sa_range() callers
These flags indicate whether the call is made to fill a bind or a server
line, or even just send/recv calls (like logs or dns). Some special cases
are made for outgoing FDs (e.g. pipes for logs) or socket FDs (e.g external
listeners), and there's a distinction between stream or dgram usage that's
expected to significantly help str2sa_range() proceed appropriately with
the input information. For now they are not used yet.
2020-09-16 22:08:08 +02:00
Willy Tarreau
809587635e MINOR: tools: add several PA_O_PORT_* flags in str2sa_range() callers
These flags indicate what is expected regarding port specifications. Some
callers accept none, some need fixed ports, some have it mandatory, some
support ranges, and some take an offset. Each possibilty is reflected by
an option. For now they are not exploited, but the goal is to instrument
str2sa_range() to properly parse that.
2020-09-16 22:08:07 +02:00
Willy Tarreau
cd3a5591f6 MINOR: tools: make str2sa_range() take more options than just resolve
We currently have an argument to require that the address is resolved
but we'll soon add more, so let's turn it into a bit field. The old
"resolve" boolean is now PA_O_RESOLVE.
2020-09-16 22:08:07 +02:00
Willy Tarreau
a5b325f92c MINOR: protocol: add a real family for existing FDs
At some places (log fd@XXX, bind fd@XXX) we support using an explicit
file descriptor number, that is placed into the sockaddr for later use.
The problem is that till now it was done with an AF_UNSPEC family, which
is also used for other situations like missing info or rings (for logs).

Let's create an "official" family AF_CUST_EXISTING_FD for this case so
that we are certain the FD can be found in the address when it is set.
2020-09-16 22:08:07 +02:00
Willy Tarreau
1e984b73f0 CLEANUP: protocol: remove family-specific fields from struct protocol
This removes the following fields from struct protocol that are now
retrieved from the protocol family instead: .sock_family, .sock_addrlen,
.l3_addrlen, .addrcmp, .bind, .get_src, .get_dst.

This also removes the UDP-specific udp{,6}_get_{src,dst}() functions
which were referenced but not used yet. Their goal was only to remap
the original AF_INET* addresses to AF_CUST_UDP*.

Note that .sock_domain is still there as it's used as a selector for
the protocol struct to be used.
2020-09-16 22:08:07 +02:00
Willy Tarreau
f1f660978c MINOR: protocol: retrieve the family-specific fields from the family
We now take care of retrieving sock_family, l3_addrlen, bind(),
addrcmp(), get_src() and get_dst() from the protocol family and
not just the protocol itself. There are very few places, this was
only seldom used. Interestingly in sock_inet.c used to rely on
->sock_family instead of ->sock_domain, and sock_unix.c used to
hard-code PF_UNIX instead of using ->sock_domain.

Also it appears obvious we have something wrong it the protocol
selection algorithm because sock_domain is the one set to the custom
protocols while it ought to be sock_family instead, which would avoid
having to hard-code some conversions for UDP namely.
2020-09-16 22:08:07 +02:00
Willy Tarreau
b0254cb361 MINOR: protocol: add a new proto_fam structure for protocol families
We need to specially handle protocol families which regroup common
functions used for a given address family. These functions include
bind(), addrcmp(), get_src() and get_dst() for now. Some fields are
also added about the address family, socket domain (protocol family
passed to the socket() syscall), and address length.

These protocol families are referenced from the protocols but not yet
used.
2020-09-16 22:08:07 +02:00
Willy Tarreau
62292b28a3 MEDIUM: sockpair: implement sockpair_bind_receiver()
Note that for now we don't have a sockpair.c file to host that unusual
family, so the new function was placed directly into proto_sockpair.c.
It's no big deal given that this family is currently not shared with
multiple protocols.

The function does almost nothing but setting up the receiver. This is
normal as the socket the FDs are passed onto are supposed to have been
already created somewhere else, and the only usable identifier for such
a socket pair is the receiving FD itself.

The function was assigned to sockpair's ->bind() and is not used yet.
2020-09-16 22:08:07 +02:00
Willy Tarreau
1e0a860099 MEDIUM: sock_unix: implement sock_unix_bind_receiver()
This function performs all the bind-related stuff for UNIX sockets that
was previously done in uxst_bind_listener(). There is a very tiny
difference however, which is that previously, in the unlikely event
where listen() would fail, it was still possible to roll back the binding
and rename the backup to the original socket. Now we have to rename it
before calling returning, hence it will be done before calling listen().
However, this doesn't cover any particular use case since listen() has no
reason to fail there (and the rollback is not done for inherited sockets),
that was just done that way as a generic error processing path.

The code is not used yet and is referenced in the uxst proto's ->bind().
2020-09-16 22:08:07 +02:00
Willy Tarreau
d69ce1ffbc MEDIUM: sock_inet: implement sock_inet_bind_receiver()
This function collects all the receiver-specific code from both
tcp_bind_listener() and udp_bind_listener() in order to provide a more
generic AF_INET/AF_INET6 socket binding function. For now the API is
not very elegant because some info are still missing from the receiver
while there's no ideal place to fill them except when calling ->listen()
at the protocol level. It looks like some polishing code is needed in
check_config_validity() or somewhere around this in order to finalize
the receivers' setup. The main issue is that listeners and receivers
are created *before* bind_conf options are parsed and that there's no
finishing step to resolve some of them.

The function currently sets up a receiver and subscribes it to the
poller. In an ideal world we wouldn't subscribe it but let the caller
do it after having finished to configure the L4 stuff. The problem is
that the caller would then need to perform an fd_insert() call and to
possibly set the exported flag on the FD while it's not its job. Maybe
an improvement could be to have a separate sock_start_receiver() call
in sock.c.

For now the function is not used but it will soon be. It's already
referenced as tcp and udp's ->bind().
2020-09-16 22:08:07 +02:00
Willy Tarreau
3e5c7ab7ce MINOR: protocol: add a new ->bind() entry to bind the receiver
This will be the function that must be used to bind the receiver. It
solely depends on the address family but for now it's simpler to have
it per protocol.
2020-09-16 22:08:07 +02:00
Willy Tarreau
b3580b19c8 MINOR: protocol: rename the ->bind field to ->listen
The function currently is doing both the bind() and the listen(), so
let's call it ->listen so that the bind() operation can move to another
place.
2020-09-16 22:08:07 +02:00
Willy Tarreau
c049c0d5ad MINOR: sock: make sock_find_compatible_fd() only take a receiver
We don't need to have a listener anymore to find an fd, a receiver with
its settings properly set is enough now.
2020-09-16 22:08:07 +02:00
Willy Tarreau
3fd3bdc836 MINOR: receiver: move the FOREIGN and V6ONLY options from listener to settings
The new RX_O_FOREIGN, RX_O_V6ONLY and RX_O_V4V6 options are now set into
the rx_settings part during the parsing, so that we don't need to adjust
them in each and every listener anymore. We have to keep both v4v6 and
v6only due to the precedence from v6only over v4v6.
2020-09-16 22:08:07 +02:00
Willy Tarreau
43046fa4f4 MINOR: listener: move the INHERITED flag down to the receiver
It's the receiver's FD that's inherited from the parent process, not
the listener's so the flag must move to the receiver so that appropriate
actions can be taken.
2020-09-16 22:08:07 +02:00
Willy Tarreau
0b9150155e MINOR: receiver: add a receiver-specific flag to indicate the socket is bound
In order to split the receiver from the listener, we'll need to know that
a socket is already bound and ready to receive. We used to do that via
tha LI_O_ASSIGNED state but that's not sufficient anymore since the
receiver might not belong to a listener anymore. The new RX_F_BOUND flag
is used for this.
2020-09-16 22:08:07 +02:00
Willy Tarreau
eef454224d MINOR: receiver: link the receiver to its owner
A receiver will have to pass a context to be installed into the fdtab
for use by the handler. We need to set this into the receiver struct
as the bind will happen longer after the configuration.
2020-09-16 22:08:07 +02:00
Willy Tarreau
0fce6bce34 MINOR: receiver: link the receiver to its settings
Just like listeners keep a pointer to their bind_conf, receivers now also
have a pointer to their rx_settings. All those belonging to a listener are
automatically initialized with a pointer to the bind_conf's settings.
2020-09-16 22:08:07 +02:00
Willy Tarreau
d45693d85c REORG: listener: move the receiver part to a new file
We'll soon add flags for the receivers, better add them to the final
file, so it's time to move the definition to receiver-t.h. The struct
receiver and rx_settings were placed there.
2020-09-16 22:08:07 +02:00
Willy Tarreau
b743661f04 REORG: listener: move the listener's proto to the receiver
The receiver is the one which depends on the protocol while the listener
relies on the receiver. Let's move the protocol there. Since there's also
a list element to get back to the listener from the proto list, this list
element (proto_list) was moved as well. For now when scanning protos, we
still see listeners which are linked by their rx.proto_list part.
2020-09-16 22:08:05 +02:00
Willy Tarreau
38ba647f9f REORG: listener: move the receiving FD to struct receiver
The listening socket is represented by its file descriptor, which is
generic to all receivers and not just listeners, so it must move to
the rx struct.

It's worth noting that in order to extend receivers and listeners to
other protocols such as QUIC, we'll need other handles than file
descriptors here, and that either a union or a cast to uintptr_t
will have to be used. This was not done yet and the field was
preserved under the name "fd" to avoid adding confusion.
2020-09-16 22:08:03 +02:00
Willy Tarreau
371590661e REORG: listener: move the listening address to a struct receiver
The address will be specific to the receiver so let's move it there.
2020-09-16 22:08:01 +02:00
Willy Tarreau
37d9d6721a REORG: listener: create a new struct receiver
In order to start to split the listeners into the listener part and the
event receiver part, we introduce a new field "rx" into struct listener
that will eventually become a separate struct receiver. This patch only
adds the struct with an options field that the receivers will need.
2020-09-16 22:07:58 +02:00
Willy Tarreau
be56c1038f MINOR: listener: move the network namespace to the struct settings
The netns is common to all listeners/receivers and is used to bind the
listening socket so it must be in the receiver settings and not in the
listener. This removes some yet another set of unnecessary loops.
2020-09-16 20:13:13 +02:00
Willy Tarreau
7e307215e8 MINOR: listener: move the interface to the struct settings
The interface is common to all listeners/receivers and is used to bind
the listening socket so it must be in the receiver settings and not in
the listener. This removes some unnecessary loops.
2020-09-16 20:13:13 +02:00
Willy Tarreau
e26993c098 MINOR: listener: move bind_proc and bind_thread to struct settings
As mentioned previously, these two fields come under the settings
struct since they'll be used to bind receivers as well.
2020-09-16 20:13:13 +02:00
Willy Tarreau
6e459d7f92 MINOR: listener: create a new struct "settings" in bind_conf
There currently is a large inconsistency in how binding parameters are
split between bind_conf and listeners. It happens that for historical
reasons some parameters are available at the listener level but cannot
be configured per-listener but only for a bind_conf, and thus, need to
be replicated. In addition, some of the bind_conf parameters are in fact
for the listening socket itself while others are for the instanciated
sockets.

A previous attempt at splitting listeners into receivers failed because
the boundary between all these settings is not well defined.

This patch introduces a level of listening socket settings in the
bind_conf, that will be detachable later. Such settings that are solely
for the listening socket are:
  - unix socket permissions (used only during binding)
  - interface (used for binding)
  - network namespace (used for binding)
  - process mask and thread mask (used during startup)

The rest seems to be used only to initialize the resulting sockets, or
to control the accept rate. For now, only the unix params (bind_conf->ux)
were moved there.
2020-09-16 20:13:13 +02:00
William Lallemand
70bf06e5f0 BUILD: fix build with openssl < 1.0.2 since bundle removal
Bundle removal broke the build with openssl version < 1.0.2.

Remove the #ifdef around SSL_SOCK_KEYTYPE_NAMES.
2020-09-16 18:10:00 +02:00
William Lallemand
e7eb1fec2f CLEANUP: ssl: remove utility functions for bundle
Remove the last utility functions for handling the multi-cert bundles
and remove the multi-variable from the ckch structure.

With this patch, the bundles are completely removed.
2020-09-16 16:28:26 +02:00
William Lallemand
bd8e6eda59 CLEANUP: ssl: remove test on "multi" variable in ckch functions
Since the removal of the multi-certificates bundle support, this
variable is not useful anymore, we can remove all tests for this
variable and suppose that every ckch contains a single certificate.
2020-09-16 16:28:26 +02:00
Willy Tarreau
441b6c31e9 BUILD: connection: fix build on clang after the VAR_ARRAY cleanup
Commit 4987a4744 ("CLEANUP: tree-wide: use VAR_ARRAY instead of [0] in
various definitions") broke the build on clang due to the tlv field used
to receive/send the proxy protocol. The problem is that struct tlv is
included at the beginning of struct tlv_ssl, which doesn't make much
sense. In fact the value[] array isn't really a var array but just an
end of struct marker, and must really be an array of size zero.
2020-09-14 08:43:51 +02:00
Willy Tarreau
4987a47446 CLEANUP: tree-wide: use VAR_ARRAY instead of [0] in various definitions
Surprisingly there were still a number of [0] definitions for variable
sized arrays in certain structures all over the code. We need to use
VAR_ARRAY instead of zero to accommodate various compilers' preferences,
as zero was used only on old ones and tends to report errors on new ones.
2020-09-12 20:56:41 +02:00
Ilya Shipitsin
4a034f2212 BUILD: introduce possibility to define ABORT_NOW() conditionally
code analysis tools recognize abort() better, so let us introduce
such possibility
2020-09-12 13:11:27 +02:00
Willy Tarreau
00c363ba9d REORG: tools: move PARSE_OPT_* from tools.h to tools-t.h
These would better be placed into the low-level type files with other
similar macros.
2020-09-11 11:27:22 +02:00
Willy Tarreau
76296dce68 BUILD: trace: always have an argument before variadic args in macros
tcc supports variadic macros provided that there is always at least one
argument, like older gcc versions. Thus we need to always keep one and
define args as the remaining ones. It's not an issue at all and doesn't
change the way to use them, just the internal definitions.
2020-09-10 09:35:54 +02:00
Willy Tarreau
d966f1497c BUILD: intops: on x86_64, the bswap instruction is called bswapq
Building with tcc fails on "bswap" which in fact ought to be called
"bswapq". Let's rename it as gas doesn't care.
2020-09-10 09:31:50 +02:00
Willy Tarreau
f6afda6539 BUILD: compiler: workaround a glibc madness around __attribute__()
For whatever reason, glibc decided that the __attribute__ keyword is
the exclusive property of gcc, and redefines it to an empty macro on
other compilers. Some non-gcc compilers also support it (possibly
partially), tinycc is one of them. By doing this, glibc silently
broke all constructors, resulting in code that arrives in main() with
uninitialized variables.

The solution we use here consists in undefining the macro on non-gcc
compilers, and redefining it to itself in order to cause a conflict in
the event the redefinition would happen afterwards. This visibly solved
the problem.
2020-09-10 09:26:50 +02:00
Willy Tarreau
d9537f6082 BUILD: compiler: reserve the gcc version checks to the gcc compiler
Some checks on __GNUC__ imply that if it's undefined it will match a
low value but that's not always what we want, like for example in the
VAR_ARRAY definition which is not needed on tcc. Let's always be explicit
on these tests.
2020-09-10 08:35:28 +02:00
Christopher Faulet
5a89175ac8 BUG/MEDIUM: dns: Don't store additional records in a linked-list
A SRV record keeps a reference on the corresponding additional record, if
any. But this additional record is also inserted in a separate linked-list into
the dns response. The problems arise when obsolete additional records are
released. The additional records list is purged but the SRV records always
reference these objects, leading to an undefined behavior. Worst, this happens
very quickly because additional records are never renewed. Thus, once received,
an additional record will always expire.

Now, the addtional record are only associated to a SRV record or simply
ignored. And the last version is always used.

This patch helps to fix the issue #841. It must be backported to 2.2.
2020-09-08 10:44:39 +02:00
Willy Tarreau
e91bff2134 MAJOR: init: start all listeners via protocols and not via proxies anymore
Ever since the protocols were added in 1.3.13, listeners used to be
started twice:
  - once by start_proxies(), which iteratees over all proxies then all
    listeners ;
  - once by protocol_bind_all() which iterates over all protocols then
    all listeners ;

It's a real mess because error reporting is not even consistent, and
more importantly now that some protocols do not appear in regular
proxies (peers, logs), there is no way to retry their binding should
it fail on the last step.

What this patch does is to make sure that listeners are exclusively
started by protocols. The failure to start a listener now causes the
emission of an error indicating the proxy's name (as it used to be
the case per proxy), and retryable failures are silently ignored
during all but last attempts.

The start_proxies() function was kept solely for setting the proxy's
state to READY and emitting the "Proxy started" message and log that
some have likely got used to seeking in their logs.
2020-09-02 11:11:43 +02:00
Willy Tarreau
576a633868 CLEANUP: protocol: remove all ->bind_all() and ->unbind_all() functions
These ones were not used anymore since the two previous patches, let's
drop them.
2020-09-02 10:40:33 +02:00
Christopher Faulet
bde2c4c621 MINOR: http-htx: Handle an optional reason when replacing the response status
When calling the http_replace_res_status() function, an optional reason may now
be set. It is ignored if it points to NULL and the original reason is
preserved. Only the response status is replaced. Otherwise both the status and
the reason are replaced.

It simplifies the API and most of time, avoids an extra call to
http_replace_res_reason().
2020-09-01 10:55:36 +02:00
Christopher Faulet
b8ce505c6f MINOR: http-htx: Add an option to eval query-string when the path is replaced
The http_replace_req_path() function now takes a third argument to evaluate the
query-string as part of the path or to preserve it. If <with_qs> is set, the
query-string is replaced with the path. Otherwise, only the path is replaced.

This patch is mandatory to fix issue #829. The next commit depends on it. So be
carefull during backports.
2020-09-01 10:55:14 +02:00