Commit Graph

682 Commits

Author SHA1 Message Date
Christopher Faulet
2f7c82bfdf BUG/MINOR: haproxy: Fix option to disable the fast-forward
The option was renamed to only permit to disable the fast-forward. First
there is no reason to enable it because it is the default behavior. Then it
introduced a bug because there is no way to be sure the command line has
precedence over the configuration this way. So, the option is now named
"tune.disable-fast-forward" and does not support any argument. And of
course, the commande line option "-dF" has now precedence over the
configuration.

No backport needed.
2023-02-21 11:44:55 +01:00
Christopher Faulet
a62201df5a DEBUG: stream: Add a BUG_ON to never exit process_stream with an expired task
We must never exit for the stream processing function with an expired
task. Otherwise, we are pretty sure this will ends with a spinning loop. It
is really better to abort as far as possible and with the original buggy
state. This will ease the debug sessions.
2023-02-21 11:35:09 +01:00
Christopher Faulet
d4eaa8af6b MINOR: global: Add an option to disable the data fast-forward
The new global option "tune.fast-forward" can be set to "off" to disable the
data fast-forward. It is an debug option, thus it is internally marked as
experimental. The directive "expose-experimental-directives" must be set
first to use this one. By default, the data fast-forward is enable.

It could be usefull to force to wake the stream up when data are
received. To be sure, evreything works fine in this case. The data
fast-forward is an optim. It must work without it. But some code may rely on
the fact the stream will not be woken up. With this option, it is possible
to spot some hidden bugs.
2023-02-17 10:17:02 +01:00
Willy Tarreau
7dbd4187dc MINOR: listener: move the nice field to the bind_conf
This is another bind line setting which can move to the bind_conf.
Note that it leaves a 2-byte hole in the listener struct.
2023-02-03 18:00:20 +01:00
Willy Tarreau
d5983cef80 MINOR: listener: remove the useless ->default_target field
This field is used by stream_new() to optionally set the applet the
stream will connect to for simple proxies like the CLI for example.
But it has never been configurable to anything and is always strictly
equal to the frontend's ->default_target. Let's just drop it and make
stream_new() only use the frontend's. It makes more sense anyway as
we don't want the proxy to work differently based on the "bind" line.
This idea was brought in 1.6 hoping that the h2 implementation would
use applets for decoding (which was dropped after the very first
attempt in 1.8).
2023-02-03 18:00:20 +01:00
Willy Tarreau
7866e8e50d MEDIUM: listener: move the analysers mask to the bind_conf
When bind_conf were created, some elements such as the analysers mask
ought to have moved there but that wasn't the case. Now that it's
getting clearer that bind_conf provides all binding parameters and
the listener is essentially a listener on an address, it's starting
to get really confusing to keep such parameters in the listener, so
let's move the mask to the bind_conf. We also take this opportunity
for pre-setting the mask to the frontend's upon initalization. Now
several loops have one less argument to take care of.
2023-02-03 18:00:20 +01:00
Christopher Faulet
da89e9b95b MINOR: channel/applets: Stop to test CF_WRITE_ERROR flag if CF_SHUTW is enough
In applets, we stop processing when a write error (CF_WRITE_ERROR) or a shutdown
for writes (CF_SHUTW) is detected. However, any write error leads to an
immediate shutdown for writes. Thus, it is enough to only test if CF_SHUTW is
set.
2023-01-09 18:41:08 +01:00
Christopher Faulet
4b490b7517 MINOR: channel: Stop to test CF_READ_ERROR flag if CF_SHUTR is enough
When a read error (CF_READ_ERROR) is reported, a shutdown for reads is
always performed (CF_SHUTR). Thus, there is no reason to check if
CF_READ_ERROR is set if CF_SHUTR is also checked.
2023-01-09 18:41:08 +01:00
Christopher Faulet
2357718217 MEDIUM: channel: Remove CF_READ_ATTACHED and report CF_READ_EVENT instead
CF_READ_ATTACHED flag is only used in input events for stream analyzers,
CF_MASK_ANALYSER. A read event can be reported instead and this flag can be
removed. We must only take care to report a read event when the client
connection is upgraded from TCP to HTTP.
2023-01-09 18:41:08 +01:00
Christopher Faulet
049fbcd36a MINOR: channel: Remove CF_ANA_TIMEOUT and report CF_READ_EVENT instead
It appears CF_ANA_TIMEOUT is flag only used in CF_MASK_ANALYSER. All
analyzer timeout relies on the analysis expiration date (chn->analyse_exp).
Worst, once set, this flag is never removed. Thus this flag can be removed
and replaced by a read event (CF_READ_EVENT).
2023-01-09 18:41:08 +01:00
Christopher Faulet
a63f8f379f MINOR: channel: Remove CF_WRITE_ACTIVITY
Thanks to previous changes, CF_WRITE_ACTIVITY flags can be removed.
Everywhere it was used, its value is now directly used
(CF_WRITE_EVENT|CF_WRITE_ERROR).
2023-01-09 18:41:08 +01:00
Christopher Faulet
33e03cec5f MINOR: channel: Remove CF_READ_ACTIVITY
Thanks to previous changes, CF_READ_ACTIVITY flags can be removed.
Everywhere it was used, its value is now directly used
(CF_READ_EVENT|CF_READ_ERROR).
2023-01-09 18:41:08 +01:00
Christopher Faulet
d898841530 MEDIUM: channel: Use CF_WRITE_EVENT instead of CF_WRITE_PARTIAL
Just like CF_READ_PARTIAL, CF_WRITE_PARTIAL is now merged with
CF_WRITE_EVENT. There a subtlety in sc_notify(). The "connect" event
(formely CF_WRITE_NULL) is now detected with
(CF_WRITE_EVENT + sc->state < SC_ST_EST).
2023-01-09 18:41:08 +01:00
Christopher Faulet
285f7616ee MEDIUM: channel: Use CF_READ_EVENT instead of CF_READ_PARTIAL
CF_READ_PARTIAL flag is now merged with CF_READ_EVENT. It means
CF_READ_EVENT is set when a read0 is received (formely CF_READ_NULL) or when
data are received (formely CF_READ_ACTIVITY).

There is nothing special here, except conditions to wake the stream up in
sc_notify(). Indeed, the test was a bit changed to reflect recent
change. read0 event is now formalized by (CF_READ_EVENT + CF_SHUTR).
2023-01-09 18:41:08 +01:00
Christopher Faulet
b96f2aa380 REORG: channel: Rename CF_WRITE_NULL to CF_WRITE_EVENT
As for CF_READ_NULL, it appears CF_WRITE_NULL and other write events on a
channel are mainly used to wake up the stream and may be replace by on write
event.

In this patch, we introduce CF_WRITE_EVENT flag as a replacement to
CF_WRITE_EVENT_NULL. There is no breaking change for now, it is just a
rename. Gradually, other write events will be merged with this one.
2023-01-09 18:41:08 +01:00
Christopher Faulet
6e1bbc446b REORG: channel: Rename CF_READ_NULL to CF_READ_EVENT
CF_READ_NULL flag is not really useful and used. It is a transient event
used to wakeup the stream. As we will see, all read events on a channel may
be resumed to only one and are all used to wake up the stream.

In this patch, we introduce CF_READ_EVENT flag as a replacement to
CF_READ_NULL. There is no breaking change for now, it is just a
rename. Gradually, other read events will be merged with this one.
2023-01-09 18:41:08 +01:00
Willy Tarreau
6c0117168e MEDIUM: stick-table: set the track-sc limit at boottime via tune.stick-counters
The number of stick-counter entries usable by track-sc rules is currently
set at build time. There is no good value for this since the vast majority
of users don't need any, most need only a few and rare users need more.
Adding more counters for everyone increases memory and CPU usages for no
reason.

This patch moves the per-session and per-stream arrays to a pool of a size
defined at boot time. This way it becomes possible to set the number of
entries at boot time via a new global setting "tune.stick-counters" that
sets the limit for the whole process. When not set, the MAX_SESS_STR_CTR
value still applies, or 3 if not set, as before.

It is also possible to lower the value to 0 to save a bit of memory if
not used at all.

Note that a few low-level sample-fetch functions had to be protected due
to the ability to use sample-fetches in the global section to set some
variables.
2023-01-06 18:08:49 +01:00
Christopher Faulet
a92480462c MINOR: http-rules: Add missing actions in http-after-response ruleset
This patch adds the support of following actions in the http-after-response
ruleset:

  * set-map, del-map and del-acl
  * set-log-level
  * sc-inc-gpc, sc-inc-gpc0 and set-inc-gpc1
  * sc-inc-gpt and sc-set-gpt0

This patch should solve the issue #1980.
2023-01-05 11:23:59 +01:00
Aurelien DARRAGON
5ad2b64262 BUG/MINOR: http_ana/txn: don't re-initialize txn and req var lists
In http_create_txn(): vars_init_head() was performed on both s->vars_txn
and s->var_reqres lists.

But this is wrong, these two lists are already initialized upon stream
creation in stream_new().
Moreover, between stream_new() and http_create_txn(), some variable may
be defined (e.g.: by the frontend), resulting in lists not being empty.

Because of this "extra" list initialization, already defined variables
can be lost.
This causes txn dependant code not being able to access previously defined
variables as well as memory leak because http_destroy_txn() relies on these
lists to perform the purge.

This proved to be the case when a frontend sets variables and lua sample
fetch is used in backend section as described in GH #1935.
Many thanks to Darragh O'Toole for his detailed report.

Removing extra var_init_head (x2) in http_create_txn() to fix the issue.
Adding somme comments in the code in an attempt to prevent future misuses
of s->var_reqres, and s->var_txn lists.

It should be backported in every stable version.
(This is an old bug that seems to exist since 1.6-dev6)

[cf: On 2.0 and 1.8, for the legacy HTTP code, vars_init() are used during
     the TXN cleanup, when the stream is reused. So, these calls must be
     moved from http_init_txn() to http_reset_txn() and not removed.]
2022-11-18 10:20:44 +01:00
Willy Tarreau
469847945c BUILD: stream: use __fallthrough in stats_dump_full_strm_to_buffer()
This avoids 1 build warning when preprocessing happens before compiling
with gcc >= 7.
2022-11-14 11:14:02 +01:00
Christopher Faulet
b976640fe1 BUG/MAJOR: stick-table: don't process store-response rules for applets
The commit bc7c207f74 ("BUG/MAJOR: stick-tables: do not try to index a
server name for applets") tried to catch applets case when we tried to index
the server name. However, there is still an issue. The applets are
unconditionally casted to servers and this bug exists since a while. it's
just luck if it doesn't crash.

Now, when store rules are processed, we skip the rule if the stream's target
is not a server or, of course, if it is a server but the "non-stick" option
is set. However, we still take care to release the sticky session.

This patch must be backported to all stable versions.
2022-10-25 18:04:54 +02:00
Willy Tarreau
bc7c207f74 BUG/MAJOR: stick-tables: do not try to index a server name for applets
Since commit 03cdf55e6 ("MINOR: stream: Stickiness server lookup by name.")
in 2.0-dev6, server names may be used instead of their IDs, in order to
perform stickiness. However the commit above may end up trying to insert
an empty server name in the dictionary when the server is an applet
instead, resulting in an immediate segfault. This is typically what
happens when a "stick-store" rule is present in a backend featuring a
"stats" directive. As there doesn't seem to be an easy way around it,
it seems to imply that "stick-store" is not much used anymore.

The solution here is to only try to insert non-null keys into the
dictionary. The patch moves the check of the key type before the
first lock so that the test on the key can be performed under the lock
instead of locking twice (the patch is more readable with diff -b).

Note that before 2.4, there's no <key> variable there as it was
introduced by commit 92149f9a8 ("MEDIUM: stick-tables: Add srvkey
option to stick-table"), but the __objt_server(s->target)->id still
needs to be tested.

This needs to be backported as far as 2.0.
2022-10-12 14:19:05 +02:00
Erwan Le Goas
2a2e46ff20 MINOR: cli: Add anonymization on a missed element for 'show sess all'
Add an anonymization for an element missed in the first merge
for 'show sess all'.

No backport needed, except if anonymization mechanism is backported.
2022-09-29 10:53:14 +02:00
Christopher Faulet
4cfc038cb1 BUG/MINOR: stream: Perform errors handling in right order in stream_new()
The frontend SC is attached before the backend one is allocated. Thus an
allocation error on backend SC must be handled before an error on the
frontend SC.

This patch must be backported to 2.6.
2022-09-27 10:56:51 +02:00
Erwan Le Goas
57e35f4b87 MINOR: cli: anonymize commands 'show sess' and 'show sess all'
Modify stream.c in order to hash the following confidential data if the
anonymized mode is enabled:
  - configuration elements such as frontend/backend/server names
  - IP addresses
2022-09-17 11:27:09 +02:00
Willy Tarreau
6a28a30efa MINOR: tasks: do not keep cpu and latency times in struct task
It was a mistake to put these two fields in the struct task. This
was added in 1.9 via commit 9efd7456e ("MEDIUM: tasks: collect per-task
CPU time and latency"). These fields are used solely by streams in
order to report the measurements via the lat_ns* and cpu_ns* sample
fetch functions when task profiling is enabled. For the rest of the
tasks, this is pure CPU waste when profiling is enabled, and memory
waste 100% of the time, as the point where these latencies and usages
are measured is in the profiling array.

Let's move the fields to the stream instead, and have process_stream()
retrieve the relevant info from the thread's context.

The struct task is now back to 120 bytes, i.e. almost two cache lines,
with 32 bit still available.
2022-09-08 14:19:15 +02:00
Willy Tarreau
beee600491 BUG/MINOR: stream/sched: take into account CPU profiling for the last call
When task profiling is enabled, the reported CPU time for short requests
and responses (e.g. redirect) is always zero in the logs, because
process_stream() is only called once and the CPU time is measured after
it returns. This is particuarly annoying when dealing with denies and in
general anything that deals with parasitic traffic because it can be
difficult to figure where the CPU is spent.

The solution taken in this patch consists in having process_stream()
update the cpu time itself before logging and quitting. It's very simple.
It will not take into account the time taken to produce the log nor
freeing the stream, but that's marginal compared to always logging zero.
The task's wake_date is also reset so that the scheduler doesn't have to
perform these operations again. This is dependent on the following patch:

   MINOR: sched: store the current profile entry in the thread context

It should be backported to 2.6 as it does help for troubleshooting.
2022-09-08 14:19:15 +02:00
Willy Tarreau
ce57777660 MINOR: muxes: add a "show_sd" helper to complete "show sess" dumps
This helper will be called for muxes that provide it and will be used
to let the mux provide extra information about the stream attached to
a stream descriptor. A line prefix is passed in argument so that the
mux is free to break long lines without breaking indent. No prefix
means no line breaks should be produced (e.g. for short dumps).
2022-09-02 15:48:50 +02:00
Willy Tarreau
714900a3c9 MINOR: debug: report applet pointer and handler in crashes when known
When an appctx is found looping over itself, we report a number of info
but not the pointers to the definition nor the handler, which can be quite
handy in some cases. Let's add them and try to decode the symbol.
2022-09-02 15:48:10 +02:00
Willy Tarreau
6d3c501c08 MEDIUM: fd/poller: turn update_mask to group-local IDs
From now on, the FD's update_mask only refers to local thread IDs. However,
there remains a limitation, in updt_fd_polling(), we temporarily have to
check and set shared FDs against .thread_mask, which still contains global
ones. As such, nbtgroups > 1 may break (but this is not yet supported without
special build options).
2022-07-15 20:16:30 +02:00
Willy Tarreau
91f7a1af34 CLEANUP: applet: remove the obsolete command context from the appctx
The "ctx" and "st2" parts in the appctx were marked for removal in 2.7
and were emulated using memcpy/memset etc for possible external code.
Let's remove this now.
2022-07-15 19:41:26 +02:00
Willy Tarreau
dd75b64cdf MINOR: cli/streams: show a stream's tgid next to its thread ID
We now display both the global thread ID and the tgid/ltid pair so that
it's easier to match it with the FD.
2022-07-15 19:41:26 +02:00
Willy Tarreau
0ad00befc1 CLEANUP: task: remove thread_mask from the struct task
It was not used anymore since everything moved to ->tid, so let's
remove it.
2022-07-01 19:15:14 +02:00
Willy Tarreau
6ef52f4479 MEDIUM: task: add and preset a thread ID in the task struct
The tasks currently rely on a mask but do not have an assigned thread ID,
contrary to tasklets. However, in practice they're either running on a
single thread or on any thread, so that it will be worth simplifying all
this in order to ease the transition to the thread groups.

This patch introduces a "tid" field in the task struct, that's either
the number of the thread the task is attached to, or a negative value
if the task is not bound to a thread, (i.e. its mask is all_threads_mask).

The new ID is only set and updated but not used yet.
2022-07-01 19:15:14 +02:00
Willy Tarreau
47af317389 BUG/MINOR: stream: only free the req/res captures when set
There's a subtle bug in stream_free() when releasing captures. The
pools may be NULL when no capture is defined, and the calls to
pool_free() are inconditional. The only reason why this doesn't
cause trouble is because the pointer to be freed is always NULL in
this case and we don't go further down the chain. That's particularly
ugly and it complicates debugging, so let's only call these ones when
the pointers are set.

There's no impact on running code, it only fools those trying to debug
pools manually. There's no need to backport it though it unless it helps
for debugging sessions.
2022-06-23 11:49:09 +02:00
Christopher Faulet
9b8d7a11c0 MINOR: stream: Rely on stconn flags to abort stream destructive upgrade
On destructive connection upgrade, instead of using the new mux name to
abort the old stream, we can relay on the stream connector flags. If it is
detached after the upgrade, it means the stream will not be resused by the
new mux and it must be aborted.

This patch may be backported to 2.6.
2022-06-17 13:25:02 +02:00
Christopher Faulet
b68f77d626 BUG/MEDIUM: stream: Properly handle destructive client connection upgrades
When the protocol is changed for a client connection at the stream level
(from TCP to H1/H2), there are two cases. The stream may be reused or
not. The first case, when the stream is reused is working. The second one is
buggy since the conn-stream refactoring and leads to a crash.

In this case, the new mux don't reuse the stream. It must be silently
aborted. However, its front stream connector is still referencing the
connection. So it must be detached. But it must be performed in two stages,
to be sure to not loose the context for the upgrade and to be able to
rollback on error. So now, before the upgrade, we prepare to detach the
stconn and it is finally detached if the upgrade succeeds. There is a trick
here. Because we pretend the stconn is detached but its state is preserved.

This patch must be backported to 2.6.
2022-06-17 13:25:02 +02:00
Willy Tarreau
d46b5b94f0 BUILD: htx: use the unchecked version of htx_get_head_blk() where needed
stream.c and mux_fcgi.c may cause a warning for a possible NULL deref
at -Os, while that is not possible thanks to the previous test. Let's
just switch to __htx_get_head_blk() instead.
2022-05-30 16:27:48 +02:00
Willy Tarreau
369d5aa208 CLEANUP: stream: remove unneeded test on appctx during initialization
Now that the data consumption from the endpoint is the default setting,
we can generalize the pre-clearing of the wont_consume flag, which is
no more specific to applets. In practice it's not needed anymore to do
it, but since streams might be initiatied from asynchronous applets,
these might have blocked their consumption side before creating the
stream thus it's safer to preserve the clearing of the flag.
2022-05-27 19:33:35 +02:00
Willy Tarreau
e68bc6178a CLEANUP: stconn: replace a few remaining occurrences of CS in comments or traces
A few "CS" desginating stconns were still present in code comments and
stream traces. This addresses them.
2022-05-27 19:33:35 +02:00
Willy Tarreau
df1a2fc234 CLEANUP: stream: rename stream_upgrade_from_cs() to stream_upgrade_from_sc()
It upgrades the protocol on a stream connector, let's update the name.
2022-05-27 19:33:35 +02:00
Willy Tarreau
c12b321661 CLEANUP: applet: rename appctx_cs() to appctx_sc()
It returns a stream connector, not a conn_stream anymore, so let's
fix its name.
2022-05-27 19:33:35 +02:00
Willy Tarreau
b49672d21f CLEANUP: stream: rename all occurrences of stconn "cs" to "sc"
Function arguments and local variables called "cs" were renamed to "sc"
to avoid future confusion. The HTTP analyser and the backend functions
were all updated after being reviewed. Function stream_update_both_cs()
was renamed to stream_update_both_sc()
2022-05-27 19:33:35 +02:00
Willy Tarreau
cb086c6de1 REORG: stconn: rename conn_stream.{c,h} to stconn.{c,h}
There's no more reason for keepin the code and definitions in conn_stream,
let's move all that to stconn. The alphabetical ordering of include files
was adjusted.
2022-05-27 19:33:35 +02:00
Willy Tarreau
5edca2f0e1 REORG: rename cs_utils.h to sc_strm.h
This file contains all the stream-connector functions that are specific
to application layers of type stream. So let's name it accordingly so
that it's easier to figure what's located there.

The alphabetical ordering of include files was preserved.
2022-05-27 19:33:35 +02:00
Willy Tarreau
74568cf023 CLEANUP: stconn: rename final state manipulation functions from cs_* to sc_*
This applies the following renaming. It's a bit large but pretty
mechanical:

cs_state -> sc_state  (enum)

cs_alloc_ibuf() -> sc_alloc_ibuf()
cs_is_conn_error() -> sc_is_conn_error()
cs_opposite() -> sc_opposite()
cs_report_error() -> sc_report_error()
cs_set_state() -> sc_set_state()
cs_state_bit() -> sc_state_bit()
cs_state_in() -> sc_state_in()
cs_state_str() -> sc_state_str()
2022-05-27 19:33:35 +02:00
Willy Tarreau
f61dd19284 CLEANUP: stconn: rename cs_{shut,chk}* to sc_*
This applies the following renaming:

cs_shutr() -> sc_shutr()
cs_shutw() -> sc_shutw()
cs_chk_rcv() -> sc_chk_rcv()
cs_chk_snd() -> sc_chk_snd()
cs_must_kill_conn() -> sc_must_kill_conn()
2022-05-27 19:33:35 +02:00
Willy Tarreau
19c65a9ded CLEANUP: stconn: rename remaining management functions from cs_* to sc_*
This is the end of the renaming for the generic SC management functions
and macros:

cs_applet_process() -> sc_applet_process()
cs_attach_applet()  -> sc_attach_applet()
cs_attach_mux()     -> sc_attach_mux()
cs_attach_strm()    -> sc_attach_strm()
cs_detach_app()     -> sc_detach_app()
cs_detach_endp()    -> sc_detach_endp()
cs_notify()         -> sc_notify()
cs_reset_endp()     -> sc_reset_endp()
cs_state_in()       -> sc_state_in()
cs_update()         -> sc_update()
cs_update_rx()      -> sc_update_rx()
cs_update_tx()      -> sc_update_tx()
IS_HTX_CS()         -> IS_HTX_SC()
2022-05-27 19:33:35 +02:00
Willy Tarreau
a0b58b537d CLEANUP: stconn: rename cs_{new,create,free,destroy}_* to sc_*
This renames the following functions:

cs_new_from_endp()  -> sc_new_from_endp()
cs_new_from_strm()  -> sc_new_from_strm()
cs_new_from_check() -> sc_new_from_check()
cs_applet_create()  -> sc_applet_create()
cs_destroy()        -> sc_destroy()
cs_free()           -> sc_free()
2022-05-27 19:33:35 +02:00
Willy Tarreau
90e8b455b7 CLEANUP: stconn: rename cs_cant_get() to se_need_more_data()
An equivalent applet_need_more_data() was added as well since that function
is mostly used from applet code. It makes it much clearer that the applet
is waiting for data from the stream layer.
2022-05-27 19:33:35 +02:00
Willy Tarreau
75a8f8e290 CLEANUP: stconn: rename cs_{want,stop}_get() to se_{will,wont}_consume()
These ones are essentially for the stream endpoint, let's give them a
name that matches the intent. Equivalent versions were provided in the
applet namespace to ease code legibility.
2022-05-27 19:33:35 +02:00
Willy Tarreau
15252cd9c0 MEDIUM: stconn: move the RXBLK flags to the stream connector
The following flags are not at all related to the endpoint but to the
connector itself:
  - SE_FL_RXBLK_ROOM
  - SE_FL_RXBLK_BUFF
  - SE_FL_RXBLK_CHAN

As such they have no business staying in the endpoint descriptor and
they must move to the stream connector. They've also been renamed
accordingly to better match what they correspond to (the same name
as the function that sets them).

The rare occurrences of cs_rx_blocked() were replaced by an explicit
test on the list of flags. The reason is that cs_rx_blocked() used to
preserve some tests that are not needed at certain places since already
known. For the same reason SE_FL_RXBLK_ANY wasn't converted. As such it
will later be possible to carefully review these few locations and
eliminate the unneeded flags from the tests. No particular function
was made to test them since they're explicit enough.

It now looks like ci_putchk() and friends could very well place the flag
themselves on the connector when they detect a buffer full condition, as
this would significantly simplify the high-level API. But all usages must
first be reviewed before this simplification can be done. For now it
remains done by applet_put*() instead.
2022-05-27 19:33:35 +02:00
Willy Tarreau
13d63afacd MINOR: stconn: add sc_is_recv_allowed() to check for ability to receive
At plenty of places we combine multiple flags checks to determine if we
can receive (endp_ready, rx_blocked, cf_shutr etc). Let's group them
under a single function that is meant to replace existing tests.

Some tests were only checking the rxblk flags at the connection level,
so for now they were not converted, this requires a bit of auditing
first, and probably a test to determine whether or not to check for
cf_shutr (e.g. there is none if no stream is present).
2022-05-27 19:33:35 +02:00
Willy Tarreau
4164eb94f3 MINOR: stconn: start to rename cs_rx_endp_{more,done}() to se_have_{no_,}more_data()
The analysis of cs_rx_endp_more() showed that the purpose is for a stream
endpoint to inform the connector that it's ready to deliver more data to
that one, and conversely cs_rx_endp_done() that it's done delivering data
so it should not be bothered again for this.

This was modified two ways:
  - the operation is no longer performed on the connector but on the
    endpoint so that there is no more doubt when reading applet code
    about what this rx refers to; it's the endpoint that has more or
    no more data.

  - an applet implementation is also provided and mostly used from
    applet code since it saves the caller from having to access the
    endpoint descriptor.

It's visible that the flag ought to be inverted because some places
have to set it by default for no reason.
2022-05-27 19:33:35 +02:00
Willy Tarreau
0ed73c376c CLEANUP: stconn: rename cs_rx_buff_{blk,rdy} to sc_{need,have}_buff()
These functions are used by the application layer to disable or enable
reading at the stream connector's level when the input buffer failed to
be allocated (or was finally allocated). The new names makes things
clearer.
2022-05-27 19:33:35 +02:00
Willy Tarreau
b73262fc85 MEDIUM: stconn: take SE_FL_APPLET_NEED_CONN out of the RXBLK_ANY flags
This makes SE_FL_APPLET_NEED_CONN autonomous, in that we check for it
everywhere we have a relevant cs_rx_blocked(), so that the flag doesn't
need anymore to be covered by cs_rx_blocked(). Indeed, this flag doesn't
really translate a receive blocking condition but rather a refusal to
wake up an applet that is waiting for a connection to finish to setup.

This also ensures we will not risk to set it back on a new endpoint
after cs_reset_endp() via SE_FL_APP_MASK, because the flag being
specific to the endpoint only and not to the connector, we don't
want to preserve it when replacing the endpoint.

It's possible that cs_chk_rcv() could later be further simplified if
we can demonstrate that the two tests in it can be merged.
2022-05-27 19:33:35 +02:00
Willy Tarreau
e7866b1ff7 MEDIUM: stconn: always rely on CF_SHUTR in addition to cs_rx_blocked()
One flag (RXBLK_SHUT) is always set with CF_SHUTR, so in order to remove
it, we first need to make sure we always check for CF_SHUTR where
cs_rx_blocked() is being used.
2022-05-27 19:33:35 +02:00
Willy Tarreau
902ba7e2bc CLEANUP: stconn: use a single function to know if SC may send to SE
sc_is_send_allowed() is now used everywhere instead of the combination
of cs_tx_endp_ready() && !cs_tx_blocked(). There's no place where we
need them individually thus it's simpler. The test was placed in cs_util
as we'll complete it later.
2022-05-27 19:33:34 +02:00
Willy Tarreau
462b989d4c CLEANUP: stconn: rename cs_conn_*() to sc_conn_*()
The following functions which act on a connection-based stream connector
were renamed to sc_conn_* (~60 places):

  cs_conn_drain_and_shut
  cs_conn_process
  cs_conn_read0
  cs_conn_ready
  cs_conn_recv
  cs_conn_send
  cs_conn_shut
  cs_conn_shutr
  cs_conn_shutw
2022-05-27 19:33:34 +02:00
Willy Tarreau
f8d0ab54ec CLEANUP: stconn: rename cs_get_data_name() to sc_get_data_name()
Only used twice to dump stream debug info.
2022-05-27 19:33:34 +02:00
Willy Tarreau
8e7c6e6907 CLEANUP: stconn: rename cs_appctx() to sc_appctx()
Nothing special, just s/cs/sc/, roughly 50-60 entries.
2022-05-27 19:33:34 +02:00
Willy Tarreau
417a31bb55 CLEANUP: stconn: rename cs_conn_mux() to sc_mux_ops()
This effectively returns the mux_ops from the connection when it exists
on an stconn.
2022-05-27 19:33:34 +02:00
Willy Tarreau
fd9417ba3f CLEANUP: stconn: rename cs_conn() to sc_conn()
It's mostly used from upper layers. Both the checked and unchecked
functions were updated, or ~150 entries.
2022-05-27 19:33:34 +02:00
Willy Tarreau
ea27f48c5a CLEANUP: stconn: rename cs_{check,strm,strm_task} to sc_strm_*
These functions return the app-layer associated with an stconn, which
is a check, a stream or a stream's task. They're used a lot to access
channels, flags and for waking up tasks. Let's just name them
appropriately for the stream connector.
2022-05-27 19:33:34 +02:00
Willy Tarreau
40a9c32e3a CLEANUP: stconn: rename cs_{i,o}{b,c} to sc_{i,o}{b,c}
We're starting to propagate the stream connector's new name through the
API. Most call places of these functions that retrieve the channel or its
buffer are in applets. The local variable names are not changed in order
to keep the changes small and reviewable. There were ~92 uses of cs_ic(),
~96 of cs_oc() (due to co_get*() being less factorizable than ci_put*),
and ~5 accesses to the buffer itself.
2022-05-27 19:33:34 +02:00
Willy Tarreau
d0a06d52f4 CLEANUP: applet: use applet_put*() everywhere possible
This applies the change so that the applet code stops using ci_putchk()
and friends everywhere possible, for the much saferapplet_put*() instead.
The change is mechanical but large. Two or three functions used to have no
appctx and a cs derived from the appctx instead, which was a reminiscence
of old times' stream_interface. These were simply changed to directly take
the appctx. No sensitive change was performed, and the old (more complex)
API is still usable when needed (e.g. the channel is already known).

The change touched roughly a hundred of locations, with no less than 124
lines removed.

It's worth noting that the stats applet, the oldest of the series, could
get a serious lifting, as it's still very channel-centric instead of
propagating the appctx along the chain. Given that this code doesn't
change often, there's no emergency to clean it up but it would look
better.
2022-05-27 19:33:34 +02:00
Willy Tarreau
026e8fb290 CLEANUP: stconn: tree-wide rename stconn states CS_ST/SB_* to SC_ST/SB_*
This also follows the natural naming. There are roughly 238 changes, all
totally trivial. conn_stream-t.h has become completely void of any
"conn_stream" related stuff now (except its name).
2022-05-27 19:33:34 +02:00
Willy Tarreau
cb04166525 CLEANUP: stconn: tree-wide rename stream connector flags CS_FL_* to SC_FL_*
This follows the natural naming. There are roughly 100 changes, all
totally trivial.
2022-05-27 19:33:34 +02:00
Willy Tarreau
7cb9e6c6ba CLEANUP: stream: rename "csf" and "csb" to "scf" and "scb"
These are the stream connectors, let's give them consistent names. The
patch is large (405 locations) but totally trivial.
2022-05-27 19:33:34 +02:00
Willy Tarreau
4596fe20d9 CLEANUP: conn_stream: tree-wide rename to stconn (stream connector)
This renames the "struct conn_stream" to "struct stconn" and updates
the descriptions in all comments (and the rare help descriptions) to
"stream connector" or "connector". This touches a lot of files but
the change is minimal. The local variables were not even renamed, so
there's still a lot of "cs" everywhere.
2022-05-27 19:33:34 +02:00
Willy Tarreau
798465b02c CLEANUP: conn_stream: rename the conn_stream's endp to sedesc
Just like for the appctx, this is a pointer to a stream endpoint descriptor,
so let's make this explicit and not confuse it with the full endpoint. There
are very few changes thanks to the preliminary refactoring of the flags
manipulation.
2022-05-27 19:33:34 +02:00
Willy Tarreau
65d0597b2b CLEANUP: conn_stream: rename the cs_endpoint's target to "se"
That's the "stream endpoint" pointer. Let's change it now while it's
not much spread. The function __cs_endp_target() wasn't yet renamed
because that will change more globally soon.
2022-05-27 19:33:34 +02:00
Willy Tarreau
b605c4213f CLEANUP: conn_stream: rename the stream endpoint flags CS_EP_* to SE_FL_*
Let's now use the new flag names for the stream endpoint.
2022-05-27 19:33:34 +02:00
Willy Tarreau
0cfcc40812 CLEANUP: conn_stream: apply cs_endp_flags.cocci tree-wide
This changes all main uses of cs->endp->flags to the sc_ep_*() equivalent
by applying coccinelle script cs_endp_flags.cocci.

Note: 143 locations were touched, manually reviewed and found to be OK,
except a single one that was adjusted in cs_reset_endp() where the flags
are read and filtered to be used as-is and not as a boolean, hence was
replaced with sc_ep_get() & $FLAGS.

The script was applied with all includes:

  spatch --in-place --recursive-includes -I include --sp-file $script $files
2022-05-27 19:33:34 +02:00
Tim Duesterhus
7ad27d41b4 CLEANUP: http_ana: Make use of the return value of stream_generate_unique_id()
Even if `unique_id` and `s->unique_id` are identical it is a bit odd to
`isttest()` `unique_id` and then use `s->unique_id` in the call to `http_add_header()`.

This "issue" was introduced in a17e66289c,
because before that commit the function returned the length of the ID, as it
was not an ist.
2022-05-18 07:19:01 +02:00
Christopher Faulet
16c0d9cda0 MINOR: applet: Add appctx_init() helper fnuction
It is just a helper function that call the .init applet callback function,
if it exists. This will simplify a bit the init stage when a new applet is
started. For now, this callback function is only used when a new service is
started.
2022-05-17 16:13:21 +02:00
Christopher Faulet
ab5d1dceed MINOR: stream: Export stream_free()
The stream_free() function is now public. It is mandatory to properly handle
errors when a new applet is started.
2022-05-17 16:13:21 +02:00
Christopher Faulet
92202da2da MINOR: applet: Let the frontend appctx release the session
The session created for frontend applets is now totally owns by the
corresponding appctx. It means the appctx is now responsible to release
it. This removes the hack in stream_free() about frontend applets to be sure
to release the session.
2022-05-17 16:13:21 +02:00
Willy Tarreau
0698c80a58 CLEANUP: applet: remove the unneeded appctx->owner
This one is the pointer to the conn_stream which is always in the
endpoint that is always present in the appctx, thus it's not needed.
This patch removes it and replaces it with appctx_cs() instead. A
few occurences that were using __cs_strm(appctx->owner) were moved
directly to appctx_strm() which does the equivalent.
2022-05-13 14:28:48 +02:00
Willy Tarreau
c7afedc140 BUILD: applet: mark the appctx's st2 variable as deprecated
This one has been misused for a while as well, it's time to deprecate it
since we don't use it anymore. It will be removed in 2.7 and for now is
only marked as deprecated. Since we need to guarantee that it's zeroed
before starting any applet or CLI command, it was moved into an anonymous
union where its sibling is not marked as deprecated so that we can
continue to initialize it without triggering a warning.

If you found this commit after a bisect session you initiated to figure
why you got some build warnings and don't know what to do, have a look
at the code that deals with the "show fd", "show sess" or "show servers"
commands, as it's supposed to be self-explanatory about the tiny changes
to apply to your code to port it. If you find APPLET_MAX_SVCCTX to be
too small for your use case, either kindly ask for a tiny extension
(and try to get your code merged), or just use a pool.
2022-05-06 18:13:36 +02:00
Willy Tarreau
6177cfc3b5 CLEANUP: stream/cli: remove the now unneeded dump state from "show sess"
The state was a constant, let's remove what remains of the switch/case.
The code from the "case" statement was only reindented as can be checked
with "git show -b".
2022-05-06 18:13:35 +02:00
Willy Tarreau
bb4e289fa9 CLEANUP: stream/cli: remove the unneeded STATE_FIN state from "show sess"
This state is only an alias for "thr >= global.nbthread" which is the
sole condition that indicates the end and reaches it. Let's just drop
this state. There's only the STATE_LIST that's left.
2022-05-06 18:13:35 +02:00
Willy Tarreau
f3629f88ac CLEANUP: stream/cli: remove the unneeded init state from "show sess"
This state was only used to preset the list element. Now that we can
guarantee that the context can be properly preset during the parsing
we don't need this state anymore. The first pointer has to be set to
point to the first stream during the initial call which is detected
by the pointer not yet being set (null). Thanks to this we can also
remove one state check on the abort path.
2022-05-06 18:13:35 +02:00
Willy Tarreau
7fb591a4e4 CLEANUP: stream/cli: stop using appctx->st2 for the dump state
Let's instead define a 3-state enum solely for this use case, and place
it into the command's context.
2022-05-06 18:13:35 +02:00
Willy Tarreau
39f097d965 CLEANUP: stream/cli: take the "show sess" context definition out of the appctx
This makes use of the generic command context allocation so that the
appctx doesn't have to declare a specific one anymore. The context is
created during parsing.
2022-05-06 18:13:35 +02:00
Willy Tarreau
7e2e4f8401 CLEANUP: tree-wide: remove 25 occurrences of unneeded fcntl.h
There were plenty of leftovers from old code that were never removed
and that are not needed at all since these files do not use any
definition depending on fcntl.h, let's drop them.
2022-04-26 10:59:48 +02:00
Christopher Faulet
eb50c01fef MINOR: conn-stream: Make cs_detach_* private and use cs_destroy() from outside
A conn-stream is never detached from an endpoint or an application alone,
except on a reset. Thus, to avoid any error, these functions are now
private. And cs_destroy() function is added to destroy a conn-stream. This
function is called when a stream is released, on the front and back
conn-streams, and when a health-check is finished.
2022-04-22 14:32:30 +02:00
Christopher Faulet
c6b60f1e00 MINOR: stream: Don't needlessly detach server endpoint on early client abort
When a client abort is detected with the server conn-stream in CS_ST_INI
state, there is no reason to detach the endpoing because we know there is no
endpoint attached to this conn-stream. This patch depends on the commit
"BUG/MEDIUM: conn-stream: Set back CS to RDY state when the appctx is
created".
2022-04-22 14:32:30 +02:00
Christopher Faulet
0264212ba3 DEV: stream: Fix conn-streams dump in full stream message
Since the recent changes about the conn-streams, the stream dump in "show
sess all" command is a bit mangled. front and back conn-stream are now
properly displayed (csf and csb). In addition, when there is no backend
endpoint, "APPCTX" was always reported. Now, "NONE" is reported in this
case.

It is 2.6-specific. No backport needed.
2022-04-22 14:14:27 +02:00
Willy Tarreau
a544c66716 BUG/MEDIUM: stream: do not abort connection setup too early
Github issue #472 reports a problem with short client connections making
stick-table entries disappear. The problem is in fact totally different
and stems at the connection establishment step.

What happens is that the stick-table there has a single entry. The
"stick-on" directive is forced to purge an existing entry before being
able to create a new one. The new entry will be committed during the
call to process_store_rules() on the response path.

But if the client sends the FIN immediately after the connection is set
up (e.g. using nc -z) then the SHUTR is received and will cancel the
connection setup just after it starts. This cancellation will induce a
call to cs_shutw() which will in turn leave the server-side state in
ST_DIS. This transition from ST_CON to ST_DIS doesn't belong to the
list of handled transition during the connection setup so it will be
handled right after on the regular path, causing the connection to be
closed. Because of this, we never pass through back_establish() and
the backend's analysers are never set on the response channel, which
is why process_store_rules() is not called and the stick-tables entry
never committed.

The comment above the code that causes this transition clearly says
that the function is to be used after the connection is established
with the server, but there's no such protection, and we always have
the AUTO_CLOSE flag there (but there's hardly any available condition
to eliminate it).

This patch adds a test for the connection not being in ST_CON or for
option abortonclose being set. It's sufficient to do the job and it
should not cause issues.

One concern was that the transition could happen during cs_recv()
after the connection switches from CON to RDY then the read0 would
be taken into account and would cause DIS to appear, which is not
handled either. But that cannot happen because cs_recv() doesn't do
anything until it's in ST_EST state, hence the read0() cannot be
called from CON/RDY. Thus the transition from CON to DIS is only
possible in back_handle_st_con() and back_handle_st_rdy() both of
which are called when dealing with the transition already, or when
abortonclose is set and the client aborts before connect() succeeds.

It's possible that some further improvements could be made to detect
this specific transition but it doesn't seem like anything would have
to be added.

This issue was first reported on 2.1. The abortonclose area is very
sensitive so it would be wise to backport slowly, and probably no
further than 2.4.
2022-04-14 17:39:48 +02:00
Christopher Faulet
1336ccffab CLEANUP: conn-stream: rename cs_register_applet() to cs_applet_create()
cs_register_applet() was not a good name because it suggests it happens
during startup, just like any other registration mechanisms..
2022-04-13 15:10:16 +02:00
Christopher Faulet
6b0a0fb2f9 CLEANUP: tree-wide: Remove any ref to stream-interfaces
Stream-interfaces are gone. Corresponding files can be safely be removed. In
addition, comments are updated accordingly.
2022-04-13 15:10:16 +02:00
Christopher Faulet
c77ceb6ad1 MEDIUM: stream: Don't use the stream-int anymore in process_stream()
process_stream() and all associated functions now manipulate conn-streams.
stream-interfaces are no longer used. In addition, function to dump info
about a stream no longer print info about stream-interfaces.
2022-04-13 15:10:16 +02:00
Christopher Faulet
0eb32c0dd1 MINOR: stream: Use conn-stream to report server error
the stream's srv_error callback function now manipulates a conn-stream
instead of a stream-interface.
2022-04-13 15:10:16 +02:00
Christopher Faulet
a0bdec350f MEDIUM: stream-int/conn-stream: Move blocking flags from SI to CS
Remaining flags and associated functions are move in the conn-stream
scope. These flags are added on the endpoint and not the conn-stream
itself. This way it will be possible to get them from the mux or the
applet. The functions to get or set these flags are renamed accordingly with
the "cs_" prefix and updated to manipualte a conn-stream instead of a
stream-interface.
2022-04-13 15:10:15 +02:00
Christopher Faulet
431ce2e3c1 MINOR: stream-int/conn-stream: Move si_sync_recv/send() in conn-stream scope
si_sync_recv() and si_sync_send() are renamesd cs_conn_recv() and
cs_conn_send() and updated to manipulate conn-streams instead of
stream-interfaces.
2022-04-13 15:10:15 +02:00
Christopher Faulet
ef285c18f2 MINOR: stream-int/stream: Move si_update_both in stream scope
si_update_both() is renamed stream_update_both_cs() and moved in stream.c.
The function is slightly changed to manipulate the stream instead the front
and back conn-streams.
2022-04-13 15:10:15 +02:00
Christopher Faulet
13045f0eae MINOR: stream-int-conn-stream: Move si_update_* in conn-stream scope
si_update_rx(), si_update_tx() and si_update() are renamed cs_update_rx(),
cs_upate_tx() and cs_update() and updated to manipulate a conn-stream
instead of a stream-interface.
2022-04-13 15:10:15 +02:00
Christopher Faulet
3704663e5f MINOR: applet: Use the CS to register and release applets instead of SI
si_register_applet() and si_applet_release() are renamed
cs_register_applet() and cs_applet_release() and now manipulate a
conn-stream instead of a stream-inteface.
2022-04-13 15:10:15 +02:00
Christopher Faulet
da098e6c17 MINOR: stream-int/conn-stream: Move si_shut* and si_chk* in conn-stream scope
si_shutr(), si_shutw(), si_chk_rcv() and si_chk_snd() are moved in the
conn-stream scope and renamed, respectively, cs_shutr(), cs_shutw(),
cs_chk_rcv(), cs_chk_snd() and manipulate a conn-stream instead of a
stream-interface.
2022-04-13 15:10:15 +02:00