Commit Graph

708 Commits

Author SHA1 Message Date
Christopher Faulet
3db538ac2f MINOR: channel: Forwad close to other side on abort
Most of calls to channel_abort() are associated to a call to
channel_auto_close(). Others are in areas where the auto close is the
default. So, it is now systematically enabled when an abort is performed on
a channel, as part of channel_abort() function.
2023-04-14 13:56:28 +02:00
Christopher Faulet
dbad8ec787 MINOR: stream: Uninline and export sess_set_term_flags() function
This function will be used to set termination flags on TCP streams from
outside of process_stream(). Thus, it must be uninlined and exported.
2023-04-14 12:13:09 +02:00
Christopher Faulet
915ba08b57 BUG/MEDIUM: stream: Report write timeouts before testing the flags
A regression was introduced when stream's timeouts were refactored. Write
timeouts are not testing is the right order. When timeous of the front SC
are handled, we must then test the read timeout on the request channel and
the write timeout on the response channel. But write timeout is tested on
the request channel instead. On the back SC, the same mix-up is performed.

We must be careful to handle timeouts before checking channel flags. To
avoid any confusions, all timeuts are handled first, on front and back SCs.
Then flags of the both channels are tested.

It is a 2.8-specific issue. No backport needed.
2023-04-14 12:13:09 +02:00
Christopher Faulet
925279ccf2 BUG/MINOR: stream: Fix test on SE_FL_ERROR on the wrong entity
There is a bug at begining of process_stream(). The SE_FL_ERROR flag is
tested against backend stream-connector's flags instead of its SE
descriptor's flags. It is an old typo, introduced when the stream-interfaces
were replaced by the conn-streams.

This patch must be backported as far as 2.6.
2023-04-14 12:13:09 +02:00
Christopher Faulet
7faac7cf34 MINOR: tree-wide: Simplifiy some tests on SHUT flags by accessing SCs directly
At many places, we simplify the tests on SHUT flags to remove calls to
chn_prod() or chn_cons() function because the corresponding SC is available.
2023-04-05 08:57:06 +02:00
Christopher Faulet
87633c3a11 MEDIUM: tree-wide: Move flags about shut from the channel to the SC
The purpose of this patch is only a one-to-one replacement, as far as
possible.

CF_SHUTR(_NOW) and CF_SHUTW(_NOW) flags are now carried by the
stream-connecter. CF_ prefix is replaced by SC_FL_ one. Of course, it is not
so simple because at many places, we were testing if a channel was shut for
reads and writes in same time. To do the same, shut for reads must be tested
on one side on the SC and shut for writes on the other side on the opposite
SC. A special care was taken with process_stream(). flags of SCs must be
saved to be able to detect changes, just like for the channels.
2023-04-05 08:57:06 +02:00
Christopher Faulet
68ef218a72 MINOR: stconn/channel: Move CF_NEVER_WAIT into the SC and rename it
The channel flag CF_NEVER_WAIT is renamed to SC_FL_SND_NEVERWAIT and moved
into the stream-connector.
2023-04-05 08:57:05 +02:00
Christopher Faulet
9a790f63ed MINOR: stconn/channel: Move CF_READ_DONTWAIT into the SC and rename it
The channel flag CF_READ_DONTWAIT is renamed to SC_FL_RCV_ONCE and moved
into the stream-connector.
2023-04-05 08:57:05 +02:00
Christopher Faulet
947b2e5922 BUG/MINOR: stream: Fix test on channels flags to set clientfin/serverfin touts
There is a bug in a way the channels flags are checked to set clientfin or
serverfin timeout. Indeed, to set the clientfin timeout, the request channel
must be shut for reads (CF_SHUTR) or the response channel must be shut for
writes (CF_SHUTW). As the opposite, the serverfin timeout must be set when
the request channel is shut for writes (CF_SHUTW) or the response channel is
shut for reads (CF_SHUTR).

It is a 2.8-dev specific issue. No backport needed.
2023-04-05 08:46:06 +02:00
Willy Tarreau
a45e7e81ec BUG/MEDIUM: stream: do not try to free a failed stream-conn
In stream_free() if we fail to allocate s->scb() we go to the path where
we try to free it, and it doesn't like being called with a null at all.
It's easily reproducible with -dMfail,no-cache and "tune.fail-alloc 10"
in the global section.

This must be backported to 2.6.
2023-03-20 19:58:38 +01:00
Christopher Faulet
e758b5c703 MEDIUM: stream: Eventually handle stream timeouts when exiting process_stream()
When we exit from process_stream(), if the task is expired, we try to handle
the stream timeouts and we resync the stream-connectors. This avoids a
useless immediate wakeup. It is not really an issue, but it is a small
improvement in edge cases.
2023-02-27 17:45:45 +01:00
Christopher Faulet
85e568f594 MINOR: stream: Handle stream's timeouts in a dedicated function
This will be mandatory to be able to handle stream's timeouts before exiting
process_stream(). So, to not duplicate code, all this stuff is moved in a
dedicated function.
2023-02-27 17:45:45 +01:00
Christopher Faulet
3bbd2baab3 BUG/MINOR: stream: Remove BUG_ON about the task expiration in process_stream()
At the end of process_stream(), A BUG_ON was recently added to abort if we
leave the function with an expired task. However, it may happen if an event
prevents the timeout to be handled but nothing evolved. In this case, the
task expiration is not updated and we expect to catch the timeout on the
immediate task wakeup.

No backport needed.
2023-02-27 17:45:45 +01:00
Christopher Faulet
be5cc766b0 MINOR: stconn: Remove half-closed timeout
The half-closed timeout is now directly retrieved from the proxy
settings. There is no longer usage for the .hcto field in the stconn
structure. So let's remove it.
2023-02-22 15:59:16 +01:00
Christopher Faulet
15315d6c0a CLEANUP: stconn: Remove old read and write expiration dates
Old read and write expiration dates are no longer used. Thus we can safely
remove them.
2023-02-22 15:59:16 +01:00
Christopher Faulet
80e4532105 MINOR: stream: Use relative expiration date in trace messages
Expiration dates in trace messages are now relative to now_ms. It is fairly
easier to read traces this way. And an expired date is now negative. So, it
is also easy to detect when a timeout was reached.
2023-02-22 15:59:16 +01:00
Christopher Faulet
03d5e62e13 MINOR: stream: Report rex/wex value using the sedesc date in trace messages
Becasue read and write timeout are now detected using .lra and .fsb fields
of the stream-endpoint descriptor, it is better to also use these fields to
report read and write expiration date in trace messages. Especially because
old rex and wex fields will be removed.
2023-02-22 15:59:16 +01:00
Christopher Faulet
6e59e871bf MINOR: stream: Dump the task expiration date in trace messages
The expiration date of the stream's task is now diplayed in the trace
messages. This will help to track changes in the stream traces.
2023-02-22 15:59:16 +01:00
Christopher Faulet
b374ba563a MAJOR: stream: Use SE descriptor date to detect read/write timeouts
We stop to use the channel's expiration dates to detect read and write
timeouts on the channels. We now rely on the stream-endpoint descriptor to
do so. All the stuff is handled in process_stream().

The stream relies on 2 helper functions to know if the receives or sends may
expire: sc_rcv_may_expire() and sc_snd_may_expire().
2023-02-22 15:57:16 +01:00
Christopher Faulet
4c13568b49 MEDIUM: stconn: Add two date to track successful reads and blocked sends
The stream endpoint descriptor now owns two date, lra (last read activity) and
fsb (first send blocked).

The first one is updated every time a read activity is reported, including data
received from the endpoint, successful connect, end of input and shutdown for
reads. A read activity is also reported when receives are unblocked. It will be
used to detect read timeouts.

The other one is updated when no data can be sent to the endpoint and reset
when some data are sent. It is the date of the first send blocked by the
endpoint. It will be used to detect write timeouts.

Helper functions are added to report read/send activity and to retrieve lra/fsb
date.
2023-02-22 14:52:15 +01:00
Christopher Faulet
5aaacfbccd MEDIUM: stconn: Replace read and write timeouts by a unique I/O timeout
Read and write timeouts (.rto and .wto) are now replaced by an unique
timeout, call .ioto. Since the recent refactoring on channel's timeouts,
both use the same value, the client timeout on client side and the server
timeout on the server side. Thus, this part may be simplified. Now it
represents the I/O timeout.
2023-02-22 14:52:15 +01:00
Christopher Faulet
f8413cba2a MEDIUM: channel/stconn: Move rex/wex timer from the channel to the sedesc
These timers are related to the I/O. Thus it is logical to move them into
the SE descriptor. The patch is a bit huge but it is just a
replacement. However it is error-prone.

From the stconn or the stream, helper functions are used to get, set or
reset these timers. This simplify the timers manipulations.
2023-02-22 14:52:15 +01:00
Christopher Faulet
ed7e66fe1a MINOR: channel/stconn: Move rto/wto from the channel to the stconn
Read and write timeouts concerns the I/O. Thus, it is logical to move it into
the stconn. At the end, the stream is responsible to detect the timeouts. So
it is logcial to have these values in the stconn and not in the SE
descriptor. But it may change depending on the recfactoring.

So, now:
  * scf->rto is used instead of req->rto
  * scf->wto is used instead of res->wto
  * scb->rto is used instead of res->rto
  * scb->wto is used instead of req->wto
2023-02-22 14:52:15 +01:00
Christopher Faulet
6362934793 DEBUG: stream/trace: Add sedesc flags in trace messages
In trace messages, when info about stream-connector are dumped, the
stream-endpoint descriptor flags are now also dumped.
2023-02-22 14:52:15 +01:00
Christopher Faulet
2e56a73459 MAJOR: channel: Remove flags to report READ or WRITE errors
This patch removes CF_READ_ERROR and CF_WRITE_ERROR flags. We now rely on
SE_FL_ERR_PENDING and SE_FL_ERROR flags. SE_FL_ERR_PENDING is used for write
errors and SE_FL_ERROR for read or unrecoverable errors.

When a connection error is reported, SE_FL_ERROR and SE_FL_EOS are now set and a
read event and a write event are reported to be sure the stream will properly
process the error. At the stream-connector level, it is similar. When an error
is reported during a send, a write event is triggered. On the read side, nothing
more is performed because an error at this stage is enough to wake the stream
up.

A major change is brought with this patch. We stop to check flags of the
ooposite channel to report abort or timeout. It also means when an read or
write error is reported on a side, we no longer update the other side. Thus
a read error on the server side does no long lead to a write error on the
client side. This should ease errors report.
2023-02-22 14:52:15 +01:00
Christopher Faulet
81fdeb8ce2 MEDIUM: channel: Remove CF_READ_NOEXP flag
This flag was introduced in 1.3 to fix a design issue. It was untouch since
then but there is no reason to still have this trick. Note it could be good
to review what happens in HTTP with the server is waiting for the end of the
request. It could be good to be sure a client timeout is always reported.
2023-02-22 14:52:14 +01:00
Christopher Faulet
2f7c82bfdf BUG/MINOR: haproxy: Fix option to disable the fast-forward
The option was renamed to only permit to disable the fast-forward. First
there is no reason to enable it because it is the default behavior. Then it
introduced a bug because there is no way to be sure the command line has
precedence over the configuration this way. So, the option is now named
"tune.disable-fast-forward" and does not support any argument. And of
course, the commande line option "-dF" has now precedence over the
configuration.

No backport needed.
2023-02-21 11:44:55 +01:00
Christopher Faulet
a62201df5a DEBUG: stream: Add a BUG_ON to never exit process_stream with an expired task
We must never exit for the stream processing function with an expired
task. Otherwise, we are pretty sure this will ends with a spinning loop. It
is really better to abort as far as possible and with the original buggy
state. This will ease the debug sessions.
2023-02-21 11:35:09 +01:00
Christopher Faulet
d4eaa8af6b MINOR: global: Add an option to disable the data fast-forward
The new global option "tune.fast-forward" can be set to "off" to disable the
data fast-forward. It is an debug option, thus it is internally marked as
experimental. The directive "expose-experimental-directives" must be set
first to use this one. By default, the data fast-forward is enable.

It could be usefull to force to wake the stream up when data are
received. To be sure, evreything works fine in this case. The data
fast-forward is an optim. It must work without it. But some code may rely on
the fact the stream will not be woken up. With this option, it is possible
to spot some hidden bugs.
2023-02-17 10:17:02 +01:00
Willy Tarreau
7dbd4187dc MINOR: listener: move the nice field to the bind_conf
This is another bind line setting which can move to the bind_conf.
Note that it leaves a 2-byte hole in the listener struct.
2023-02-03 18:00:20 +01:00
Willy Tarreau
d5983cef80 MINOR: listener: remove the useless ->default_target field
This field is used by stream_new() to optionally set the applet the
stream will connect to for simple proxies like the CLI for example.
But it has never been configurable to anything and is always strictly
equal to the frontend's ->default_target. Let's just drop it and make
stream_new() only use the frontend's. It makes more sense anyway as
we don't want the proxy to work differently based on the "bind" line.
This idea was brought in 1.6 hoping that the h2 implementation would
use applets for decoding (which was dropped after the very first
attempt in 1.8).
2023-02-03 18:00:20 +01:00
Willy Tarreau
7866e8e50d MEDIUM: listener: move the analysers mask to the bind_conf
When bind_conf were created, some elements such as the analysers mask
ought to have moved there but that wasn't the case. Now that it's
getting clearer that bind_conf provides all binding parameters and
the listener is essentially a listener on an address, it's starting
to get really confusing to keep such parameters in the listener, so
let's move the mask to the bind_conf. We also take this opportunity
for pre-setting the mask to the frontend's upon initalization. Now
several loops have one less argument to take care of.
2023-02-03 18:00:20 +01:00
Christopher Faulet
da89e9b95b MINOR: channel/applets: Stop to test CF_WRITE_ERROR flag if CF_SHUTW is enough
In applets, we stop processing when a write error (CF_WRITE_ERROR) or a shutdown
for writes (CF_SHUTW) is detected. However, any write error leads to an
immediate shutdown for writes. Thus, it is enough to only test if CF_SHUTW is
set.
2023-01-09 18:41:08 +01:00
Christopher Faulet
4b490b7517 MINOR: channel: Stop to test CF_READ_ERROR flag if CF_SHUTR is enough
When a read error (CF_READ_ERROR) is reported, a shutdown for reads is
always performed (CF_SHUTR). Thus, there is no reason to check if
CF_READ_ERROR is set if CF_SHUTR is also checked.
2023-01-09 18:41:08 +01:00
Christopher Faulet
2357718217 MEDIUM: channel: Remove CF_READ_ATTACHED and report CF_READ_EVENT instead
CF_READ_ATTACHED flag is only used in input events for stream analyzers,
CF_MASK_ANALYSER. A read event can be reported instead and this flag can be
removed. We must only take care to report a read event when the client
connection is upgraded from TCP to HTTP.
2023-01-09 18:41:08 +01:00
Christopher Faulet
049fbcd36a MINOR: channel: Remove CF_ANA_TIMEOUT and report CF_READ_EVENT instead
It appears CF_ANA_TIMEOUT is flag only used in CF_MASK_ANALYSER. All
analyzer timeout relies on the analysis expiration date (chn->analyse_exp).
Worst, once set, this flag is never removed. Thus this flag can be removed
and replaced by a read event (CF_READ_EVENT).
2023-01-09 18:41:08 +01:00
Christopher Faulet
a63f8f379f MINOR: channel: Remove CF_WRITE_ACTIVITY
Thanks to previous changes, CF_WRITE_ACTIVITY flags can be removed.
Everywhere it was used, its value is now directly used
(CF_WRITE_EVENT|CF_WRITE_ERROR).
2023-01-09 18:41:08 +01:00
Christopher Faulet
33e03cec5f MINOR: channel: Remove CF_READ_ACTIVITY
Thanks to previous changes, CF_READ_ACTIVITY flags can be removed.
Everywhere it was used, its value is now directly used
(CF_READ_EVENT|CF_READ_ERROR).
2023-01-09 18:41:08 +01:00
Christopher Faulet
d898841530 MEDIUM: channel: Use CF_WRITE_EVENT instead of CF_WRITE_PARTIAL
Just like CF_READ_PARTIAL, CF_WRITE_PARTIAL is now merged with
CF_WRITE_EVENT. There a subtlety in sc_notify(). The "connect" event
(formely CF_WRITE_NULL) is now detected with
(CF_WRITE_EVENT + sc->state < SC_ST_EST).
2023-01-09 18:41:08 +01:00
Christopher Faulet
285f7616ee MEDIUM: channel: Use CF_READ_EVENT instead of CF_READ_PARTIAL
CF_READ_PARTIAL flag is now merged with CF_READ_EVENT. It means
CF_READ_EVENT is set when a read0 is received (formely CF_READ_NULL) or when
data are received (formely CF_READ_ACTIVITY).

There is nothing special here, except conditions to wake the stream up in
sc_notify(). Indeed, the test was a bit changed to reflect recent
change. read0 event is now formalized by (CF_READ_EVENT + CF_SHUTR).
2023-01-09 18:41:08 +01:00
Christopher Faulet
b96f2aa380 REORG: channel: Rename CF_WRITE_NULL to CF_WRITE_EVENT
As for CF_READ_NULL, it appears CF_WRITE_NULL and other write events on a
channel are mainly used to wake up the stream and may be replace by on write
event.

In this patch, we introduce CF_WRITE_EVENT flag as a replacement to
CF_WRITE_EVENT_NULL. There is no breaking change for now, it is just a
rename. Gradually, other write events will be merged with this one.
2023-01-09 18:41:08 +01:00
Christopher Faulet
6e1bbc446b REORG: channel: Rename CF_READ_NULL to CF_READ_EVENT
CF_READ_NULL flag is not really useful and used. It is a transient event
used to wakeup the stream. As we will see, all read events on a channel may
be resumed to only one and are all used to wake up the stream.

In this patch, we introduce CF_READ_EVENT flag as a replacement to
CF_READ_NULL. There is no breaking change for now, it is just a
rename. Gradually, other read events will be merged with this one.
2023-01-09 18:41:08 +01:00
Willy Tarreau
6c0117168e MEDIUM: stick-table: set the track-sc limit at boottime via tune.stick-counters
The number of stick-counter entries usable by track-sc rules is currently
set at build time. There is no good value for this since the vast majority
of users don't need any, most need only a few and rare users need more.
Adding more counters for everyone increases memory and CPU usages for no
reason.

This patch moves the per-session and per-stream arrays to a pool of a size
defined at boot time. This way it becomes possible to set the number of
entries at boot time via a new global setting "tune.stick-counters" that
sets the limit for the whole process. When not set, the MAX_SESS_STR_CTR
value still applies, or 3 if not set, as before.

It is also possible to lower the value to 0 to save a bit of memory if
not used at all.

Note that a few low-level sample-fetch functions had to be protected due
to the ability to use sample-fetches in the global section to set some
variables.
2023-01-06 18:08:49 +01:00
Christopher Faulet
a92480462c MINOR: http-rules: Add missing actions in http-after-response ruleset
This patch adds the support of following actions in the http-after-response
ruleset:

  * set-map, del-map and del-acl
  * set-log-level
  * sc-inc-gpc, sc-inc-gpc0 and set-inc-gpc1
  * sc-inc-gpt and sc-set-gpt0

This patch should solve the issue #1980.
2023-01-05 11:23:59 +01:00
Aurelien DARRAGON
5ad2b64262 BUG/MINOR: http_ana/txn: don't re-initialize txn and req var lists
In http_create_txn(): vars_init_head() was performed on both s->vars_txn
and s->var_reqres lists.

But this is wrong, these two lists are already initialized upon stream
creation in stream_new().
Moreover, between stream_new() and http_create_txn(), some variable may
be defined (e.g.: by the frontend), resulting in lists not being empty.

Because of this "extra" list initialization, already defined variables
can be lost.
This causes txn dependant code not being able to access previously defined
variables as well as memory leak because http_destroy_txn() relies on these
lists to perform the purge.

This proved to be the case when a frontend sets variables and lua sample
fetch is used in backend section as described in GH #1935.
Many thanks to Darragh O'Toole for his detailed report.

Removing extra var_init_head (x2) in http_create_txn() to fix the issue.
Adding somme comments in the code in an attempt to prevent future misuses
of s->var_reqres, and s->var_txn lists.

It should be backported in every stable version.
(This is an old bug that seems to exist since 1.6-dev6)

[cf: On 2.0 and 1.8, for the legacy HTTP code, vars_init() are used during
     the TXN cleanup, when the stream is reused. So, these calls must be
     moved from http_init_txn() to http_reset_txn() and not removed.]
2022-11-18 10:20:44 +01:00
Willy Tarreau
469847945c BUILD: stream: use __fallthrough in stats_dump_full_strm_to_buffer()
This avoids 1 build warning when preprocessing happens before compiling
with gcc >= 7.
2022-11-14 11:14:02 +01:00
Christopher Faulet
b976640fe1 BUG/MAJOR: stick-table: don't process store-response rules for applets
The commit bc7c207f74 ("BUG/MAJOR: stick-tables: do not try to index a
server name for applets") tried to catch applets case when we tried to index
the server name. However, there is still an issue. The applets are
unconditionally casted to servers and this bug exists since a while. it's
just luck if it doesn't crash.

Now, when store rules are processed, we skip the rule if the stream's target
is not a server or, of course, if it is a server but the "non-stick" option
is set. However, we still take care to release the sticky session.

This patch must be backported to all stable versions.
2022-10-25 18:04:54 +02:00
Willy Tarreau
bc7c207f74 BUG/MAJOR: stick-tables: do not try to index a server name for applets
Since commit 03cdf55e6 ("MINOR: stream: Stickiness server lookup by name.")
in 2.0-dev6, server names may be used instead of their IDs, in order to
perform stickiness. However the commit above may end up trying to insert
an empty server name in the dictionary when the server is an applet
instead, resulting in an immediate segfault. This is typically what
happens when a "stick-store" rule is present in a backend featuring a
"stats" directive. As there doesn't seem to be an easy way around it,
it seems to imply that "stick-store" is not much used anymore.

The solution here is to only try to insert non-null keys into the
dictionary. The patch moves the check of the key type before the
first lock so that the test on the key can be performed under the lock
instead of locking twice (the patch is more readable with diff -b).

Note that before 2.4, there's no <key> variable there as it was
introduced by commit 92149f9a8 ("MEDIUM: stick-tables: Add srvkey
option to stick-table"), but the __objt_server(s->target)->id still
needs to be tested.

This needs to be backported as far as 2.0.
2022-10-12 14:19:05 +02:00
Erwan Le Goas
2a2e46ff20 MINOR: cli: Add anonymization on a missed element for 'show sess all'
Add an anonymization for an element missed in the first merge
for 'show sess all'.

No backport needed, except if anonymization mechanism is backported.
2022-09-29 10:53:14 +02:00
Christopher Faulet
4cfc038cb1 BUG/MINOR: stream: Perform errors handling in right order in stream_new()
The frontend SC is attached before the backend one is allocated. Thus an
allocation error on backend SC must be handled before an error on the
frontend SC.

This patch must be backported to 2.6.
2022-09-27 10:56:51 +02:00
Erwan Le Goas
57e35f4b87 MINOR: cli: anonymize commands 'show sess' and 'show sess all'
Modify stream.c in order to hash the following confidential data if the
anonymized mode is enabled:
  - configuration elements such as frontend/backend/server names
  - IP addresses
2022-09-17 11:27:09 +02:00
Willy Tarreau
6a28a30efa MINOR: tasks: do not keep cpu and latency times in struct task
It was a mistake to put these two fields in the struct task. This
was added in 1.9 via commit 9efd7456e ("MEDIUM: tasks: collect per-task
CPU time and latency"). These fields are used solely by streams in
order to report the measurements via the lat_ns* and cpu_ns* sample
fetch functions when task profiling is enabled. For the rest of the
tasks, this is pure CPU waste when profiling is enabled, and memory
waste 100% of the time, as the point where these latencies and usages
are measured is in the profiling array.

Let's move the fields to the stream instead, and have process_stream()
retrieve the relevant info from the thread's context.

The struct task is now back to 120 bytes, i.e. almost two cache lines,
with 32 bit still available.
2022-09-08 14:19:15 +02:00
Willy Tarreau
beee600491 BUG/MINOR: stream/sched: take into account CPU profiling for the last call
When task profiling is enabled, the reported CPU time for short requests
and responses (e.g. redirect) is always zero in the logs, because
process_stream() is only called once and the CPU time is measured after
it returns. This is particuarly annoying when dealing with denies and in
general anything that deals with parasitic traffic because it can be
difficult to figure where the CPU is spent.

The solution taken in this patch consists in having process_stream()
update the cpu time itself before logging and quitting. It's very simple.
It will not take into account the time taken to produce the log nor
freeing the stream, but that's marginal compared to always logging zero.
The task's wake_date is also reset so that the scheduler doesn't have to
perform these operations again. This is dependent on the following patch:

   MINOR: sched: store the current profile entry in the thread context

It should be backported to 2.6 as it does help for troubleshooting.
2022-09-08 14:19:15 +02:00
Willy Tarreau
ce57777660 MINOR: muxes: add a "show_sd" helper to complete "show sess" dumps
This helper will be called for muxes that provide it and will be used
to let the mux provide extra information about the stream attached to
a stream descriptor. A line prefix is passed in argument so that the
mux is free to break long lines without breaking indent. No prefix
means no line breaks should be produced (e.g. for short dumps).
2022-09-02 15:48:50 +02:00
Willy Tarreau
714900a3c9 MINOR: debug: report applet pointer and handler in crashes when known
When an appctx is found looping over itself, we report a number of info
but not the pointers to the definition nor the handler, which can be quite
handy in some cases. Let's add them and try to decode the symbol.
2022-09-02 15:48:10 +02:00
Willy Tarreau
6d3c501c08 MEDIUM: fd/poller: turn update_mask to group-local IDs
From now on, the FD's update_mask only refers to local thread IDs. However,
there remains a limitation, in updt_fd_polling(), we temporarily have to
check and set shared FDs against .thread_mask, which still contains global
ones. As such, nbtgroups > 1 may break (but this is not yet supported without
special build options).
2022-07-15 20:16:30 +02:00
Willy Tarreau
91f7a1af34 CLEANUP: applet: remove the obsolete command context from the appctx
The "ctx" and "st2" parts in the appctx were marked for removal in 2.7
and were emulated using memcpy/memset etc for possible external code.
Let's remove this now.
2022-07-15 19:41:26 +02:00
Willy Tarreau
dd75b64cdf MINOR: cli/streams: show a stream's tgid next to its thread ID
We now display both the global thread ID and the tgid/ltid pair so that
it's easier to match it with the FD.
2022-07-15 19:41:26 +02:00
Willy Tarreau
0ad00befc1 CLEANUP: task: remove thread_mask from the struct task
It was not used anymore since everything moved to ->tid, so let's
remove it.
2022-07-01 19:15:14 +02:00
Willy Tarreau
6ef52f4479 MEDIUM: task: add and preset a thread ID in the task struct
The tasks currently rely on a mask but do not have an assigned thread ID,
contrary to tasklets. However, in practice they're either running on a
single thread or on any thread, so that it will be worth simplifying all
this in order to ease the transition to the thread groups.

This patch introduces a "tid" field in the task struct, that's either
the number of the thread the task is attached to, or a negative value
if the task is not bound to a thread, (i.e. its mask is all_threads_mask).

The new ID is only set and updated but not used yet.
2022-07-01 19:15:14 +02:00
Willy Tarreau
47af317389 BUG/MINOR: stream: only free the req/res captures when set
There's a subtle bug in stream_free() when releasing captures. The
pools may be NULL when no capture is defined, and the calls to
pool_free() are inconditional. The only reason why this doesn't
cause trouble is because the pointer to be freed is always NULL in
this case and we don't go further down the chain. That's particularly
ugly and it complicates debugging, so let's only call these ones when
the pointers are set.

There's no impact on running code, it only fools those trying to debug
pools manually. There's no need to backport it though it unless it helps
for debugging sessions.
2022-06-23 11:49:09 +02:00
Christopher Faulet
9b8d7a11c0 MINOR: stream: Rely on stconn flags to abort stream destructive upgrade
On destructive connection upgrade, instead of using the new mux name to
abort the old stream, we can relay on the stream connector flags. If it is
detached after the upgrade, it means the stream will not be resused by the
new mux and it must be aborted.

This patch may be backported to 2.6.
2022-06-17 13:25:02 +02:00
Christopher Faulet
b68f77d626 BUG/MEDIUM: stream: Properly handle destructive client connection upgrades
When the protocol is changed for a client connection at the stream level
(from TCP to H1/H2), there are two cases. The stream may be reused or
not. The first case, when the stream is reused is working. The second one is
buggy since the conn-stream refactoring and leads to a crash.

In this case, the new mux don't reuse the stream. It must be silently
aborted. However, its front stream connector is still referencing the
connection. So it must be detached. But it must be performed in two stages,
to be sure to not loose the context for the upgrade and to be able to
rollback on error. So now, before the upgrade, we prepare to detach the
stconn and it is finally detached if the upgrade succeeds. There is a trick
here. Because we pretend the stconn is detached but its state is preserved.

This patch must be backported to 2.6.
2022-06-17 13:25:02 +02:00
Willy Tarreau
d46b5b94f0 BUILD: htx: use the unchecked version of htx_get_head_blk() where needed
stream.c and mux_fcgi.c may cause a warning for a possible NULL deref
at -Os, while that is not possible thanks to the previous test. Let's
just switch to __htx_get_head_blk() instead.
2022-05-30 16:27:48 +02:00
Willy Tarreau
369d5aa208 CLEANUP: stream: remove unneeded test on appctx during initialization
Now that the data consumption from the endpoint is the default setting,
we can generalize the pre-clearing of the wont_consume flag, which is
no more specific to applets. In practice it's not needed anymore to do
it, but since streams might be initiatied from asynchronous applets,
these might have blocked their consumption side before creating the
stream thus it's safer to preserve the clearing of the flag.
2022-05-27 19:33:35 +02:00
Willy Tarreau
e68bc6178a CLEANUP: stconn: replace a few remaining occurrences of CS in comments or traces
A few "CS" desginating stconns were still present in code comments and
stream traces. This addresses them.
2022-05-27 19:33:35 +02:00
Willy Tarreau
df1a2fc234 CLEANUP: stream: rename stream_upgrade_from_cs() to stream_upgrade_from_sc()
It upgrades the protocol on a stream connector, let's update the name.
2022-05-27 19:33:35 +02:00
Willy Tarreau
c12b321661 CLEANUP: applet: rename appctx_cs() to appctx_sc()
It returns a stream connector, not a conn_stream anymore, so let's
fix its name.
2022-05-27 19:33:35 +02:00
Willy Tarreau
b49672d21f CLEANUP: stream: rename all occurrences of stconn "cs" to "sc"
Function arguments and local variables called "cs" were renamed to "sc"
to avoid future confusion. The HTTP analyser and the backend functions
were all updated after being reviewed. Function stream_update_both_cs()
was renamed to stream_update_both_sc()
2022-05-27 19:33:35 +02:00
Willy Tarreau
cb086c6de1 REORG: stconn: rename conn_stream.{c,h} to stconn.{c,h}
There's no more reason for keepin the code and definitions in conn_stream,
let's move all that to stconn. The alphabetical ordering of include files
was adjusted.
2022-05-27 19:33:35 +02:00
Willy Tarreau
5edca2f0e1 REORG: rename cs_utils.h to sc_strm.h
This file contains all the stream-connector functions that are specific
to application layers of type stream. So let's name it accordingly so
that it's easier to figure what's located there.

The alphabetical ordering of include files was preserved.
2022-05-27 19:33:35 +02:00
Willy Tarreau
74568cf023 CLEANUP: stconn: rename final state manipulation functions from cs_* to sc_*
This applies the following renaming. It's a bit large but pretty
mechanical:

cs_state -> sc_state  (enum)

cs_alloc_ibuf() -> sc_alloc_ibuf()
cs_is_conn_error() -> sc_is_conn_error()
cs_opposite() -> sc_opposite()
cs_report_error() -> sc_report_error()
cs_set_state() -> sc_set_state()
cs_state_bit() -> sc_state_bit()
cs_state_in() -> sc_state_in()
cs_state_str() -> sc_state_str()
2022-05-27 19:33:35 +02:00
Willy Tarreau
f61dd19284 CLEANUP: stconn: rename cs_{shut,chk}* to sc_*
This applies the following renaming:

cs_shutr() -> sc_shutr()
cs_shutw() -> sc_shutw()
cs_chk_rcv() -> sc_chk_rcv()
cs_chk_snd() -> sc_chk_snd()
cs_must_kill_conn() -> sc_must_kill_conn()
2022-05-27 19:33:35 +02:00
Willy Tarreau
19c65a9ded CLEANUP: stconn: rename remaining management functions from cs_* to sc_*
This is the end of the renaming for the generic SC management functions
and macros:

cs_applet_process() -> sc_applet_process()
cs_attach_applet()  -> sc_attach_applet()
cs_attach_mux()     -> sc_attach_mux()
cs_attach_strm()    -> sc_attach_strm()
cs_detach_app()     -> sc_detach_app()
cs_detach_endp()    -> sc_detach_endp()
cs_notify()         -> sc_notify()
cs_reset_endp()     -> sc_reset_endp()
cs_state_in()       -> sc_state_in()
cs_update()         -> sc_update()
cs_update_rx()      -> sc_update_rx()
cs_update_tx()      -> sc_update_tx()
IS_HTX_CS()         -> IS_HTX_SC()
2022-05-27 19:33:35 +02:00
Willy Tarreau
a0b58b537d CLEANUP: stconn: rename cs_{new,create,free,destroy}_* to sc_*
This renames the following functions:

cs_new_from_endp()  -> sc_new_from_endp()
cs_new_from_strm()  -> sc_new_from_strm()
cs_new_from_check() -> sc_new_from_check()
cs_applet_create()  -> sc_applet_create()
cs_destroy()        -> sc_destroy()
cs_free()           -> sc_free()
2022-05-27 19:33:35 +02:00
Willy Tarreau
90e8b455b7 CLEANUP: stconn: rename cs_cant_get() to se_need_more_data()
An equivalent applet_need_more_data() was added as well since that function
is mostly used from applet code. It makes it much clearer that the applet
is waiting for data from the stream layer.
2022-05-27 19:33:35 +02:00
Willy Tarreau
75a8f8e290 CLEANUP: stconn: rename cs_{want,stop}_get() to se_{will,wont}_consume()
These ones are essentially for the stream endpoint, let's give them a
name that matches the intent. Equivalent versions were provided in the
applet namespace to ease code legibility.
2022-05-27 19:33:35 +02:00
Willy Tarreau
15252cd9c0 MEDIUM: stconn: move the RXBLK flags to the stream connector
The following flags are not at all related to the endpoint but to the
connector itself:
  - SE_FL_RXBLK_ROOM
  - SE_FL_RXBLK_BUFF
  - SE_FL_RXBLK_CHAN

As such they have no business staying in the endpoint descriptor and
they must move to the stream connector. They've also been renamed
accordingly to better match what they correspond to (the same name
as the function that sets them).

The rare occurrences of cs_rx_blocked() were replaced by an explicit
test on the list of flags. The reason is that cs_rx_blocked() used to
preserve some tests that are not needed at certain places since already
known. For the same reason SE_FL_RXBLK_ANY wasn't converted. As such it
will later be possible to carefully review these few locations and
eliminate the unneeded flags from the tests. No particular function
was made to test them since they're explicit enough.

It now looks like ci_putchk() and friends could very well place the flag
themselves on the connector when they detect a buffer full condition, as
this would significantly simplify the high-level API. But all usages must
first be reviewed before this simplification can be done. For now it
remains done by applet_put*() instead.
2022-05-27 19:33:35 +02:00
Willy Tarreau
13d63afacd MINOR: stconn: add sc_is_recv_allowed() to check for ability to receive
At plenty of places we combine multiple flags checks to determine if we
can receive (endp_ready, rx_blocked, cf_shutr etc). Let's group them
under a single function that is meant to replace existing tests.

Some tests were only checking the rxblk flags at the connection level,
so for now they were not converted, this requires a bit of auditing
first, and probably a test to determine whether or not to check for
cf_shutr (e.g. there is none if no stream is present).
2022-05-27 19:33:35 +02:00
Willy Tarreau
4164eb94f3 MINOR: stconn: start to rename cs_rx_endp_{more,done}() to se_have_{no_,}more_data()
The analysis of cs_rx_endp_more() showed that the purpose is for a stream
endpoint to inform the connector that it's ready to deliver more data to
that one, and conversely cs_rx_endp_done() that it's done delivering data
so it should not be bothered again for this.

This was modified two ways:
  - the operation is no longer performed on the connector but on the
    endpoint so that there is no more doubt when reading applet code
    about what this rx refers to; it's the endpoint that has more or
    no more data.

  - an applet implementation is also provided and mostly used from
    applet code since it saves the caller from having to access the
    endpoint descriptor.

It's visible that the flag ought to be inverted because some places
have to set it by default for no reason.
2022-05-27 19:33:35 +02:00
Willy Tarreau
0ed73c376c CLEANUP: stconn: rename cs_rx_buff_{blk,rdy} to sc_{need,have}_buff()
These functions are used by the application layer to disable or enable
reading at the stream connector's level when the input buffer failed to
be allocated (or was finally allocated). The new names makes things
clearer.
2022-05-27 19:33:35 +02:00
Willy Tarreau
b73262fc85 MEDIUM: stconn: take SE_FL_APPLET_NEED_CONN out of the RXBLK_ANY flags
This makes SE_FL_APPLET_NEED_CONN autonomous, in that we check for it
everywhere we have a relevant cs_rx_blocked(), so that the flag doesn't
need anymore to be covered by cs_rx_blocked(). Indeed, this flag doesn't
really translate a receive blocking condition but rather a refusal to
wake up an applet that is waiting for a connection to finish to setup.

This also ensures we will not risk to set it back on a new endpoint
after cs_reset_endp() via SE_FL_APP_MASK, because the flag being
specific to the endpoint only and not to the connector, we don't
want to preserve it when replacing the endpoint.

It's possible that cs_chk_rcv() could later be further simplified if
we can demonstrate that the two tests in it can be merged.
2022-05-27 19:33:35 +02:00
Willy Tarreau
e7866b1ff7 MEDIUM: stconn: always rely on CF_SHUTR in addition to cs_rx_blocked()
One flag (RXBLK_SHUT) is always set with CF_SHUTR, so in order to remove
it, we first need to make sure we always check for CF_SHUTR where
cs_rx_blocked() is being used.
2022-05-27 19:33:35 +02:00
Willy Tarreau
902ba7e2bc CLEANUP: stconn: use a single function to know if SC may send to SE
sc_is_send_allowed() is now used everywhere instead of the combination
of cs_tx_endp_ready() && !cs_tx_blocked(). There's no place where we
need them individually thus it's simpler. The test was placed in cs_util
as we'll complete it later.
2022-05-27 19:33:34 +02:00
Willy Tarreau
462b989d4c CLEANUP: stconn: rename cs_conn_*() to sc_conn_*()
The following functions which act on a connection-based stream connector
were renamed to sc_conn_* (~60 places):

  cs_conn_drain_and_shut
  cs_conn_process
  cs_conn_read0
  cs_conn_ready
  cs_conn_recv
  cs_conn_send
  cs_conn_shut
  cs_conn_shutr
  cs_conn_shutw
2022-05-27 19:33:34 +02:00
Willy Tarreau
f8d0ab54ec CLEANUP: stconn: rename cs_get_data_name() to sc_get_data_name()
Only used twice to dump stream debug info.
2022-05-27 19:33:34 +02:00
Willy Tarreau
8e7c6e6907 CLEANUP: stconn: rename cs_appctx() to sc_appctx()
Nothing special, just s/cs/sc/, roughly 50-60 entries.
2022-05-27 19:33:34 +02:00
Willy Tarreau
417a31bb55 CLEANUP: stconn: rename cs_conn_mux() to sc_mux_ops()
This effectively returns the mux_ops from the connection when it exists
on an stconn.
2022-05-27 19:33:34 +02:00
Willy Tarreau
fd9417ba3f CLEANUP: stconn: rename cs_conn() to sc_conn()
It's mostly used from upper layers. Both the checked and unchecked
functions were updated, or ~150 entries.
2022-05-27 19:33:34 +02:00
Willy Tarreau
ea27f48c5a CLEANUP: stconn: rename cs_{check,strm,strm_task} to sc_strm_*
These functions return the app-layer associated with an stconn, which
is a check, a stream or a stream's task. They're used a lot to access
channels, flags and for waking up tasks. Let's just name them
appropriately for the stream connector.
2022-05-27 19:33:34 +02:00
Willy Tarreau
40a9c32e3a CLEANUP: stconn: rename cs_{i,o}{b,c} to sc_{i,o}{b,c}
We're starting to propagate the stream connector's new name through the
API. Most call places of these functions that retrieve the channel or its
buffer are in applets. The local variable names are not changed in order
to keep the changes small and reviewable. There were ~92 uses of cs_ic(),
~96 of cs_oc() (due to co_get*() being less factorizable than ci_put*),
and ~5 accesses to the buffer itself.
2022-05-27 19:33:34 +02:00
Willy Tarreau
d0a06d52f4 CLEANUP: applet: use applet_put*() everywhere possible
This applies the change so that the applet code stops using ci_putchk()
and friends everywhere possible, for the much saferapplet_put*() instead.
The change is mechanical but large. Two or three functions used to have no
appctx and a cs derived from the appctx instead, which was a reminiscence
of old times' stream_interface. These were simply changed to directly take
the appctx. No sensitive change was performed, and the old (more complex)
API is still usable when needed (e.g. the channel is already known).

The change touched roughly a hundred of locations, with no less than 124
lines removed.

It's worth noting that the stats applet, the oldest of the series, could
get a serious lifting, as it's still very channel-centric instead of
propagating the appctx along the chain. Given that this code doesn't
change often, there's no emergency to clean it up but it would look
better.
2022-05-27 19:33:34 +02:00
Willy Tarreau
026e8fb290 CLEANUP: stconn: tree-wide rename stconn states CS_ST/SB_* to SC_ST/SB_*
This also follows the natural naming. There are roughly 238 changes, all
totally trivial. conn_stream-t.h has become completely void of any
"conn_stream" related stuff now (except its name).
2022-05-27 19:33:34 +02:00
Willy Tarreau
cb04166525 CLEANUP: stconn: tree-wide rename stream connector flags CS_FL_* to SC_FL_*
This follows the natural naming. There are roughly 100 changes, all
totally trivial.
2022-05-27 19:33:34 +02:00
Willy Tarreau
7cb9e6c6ba CLEANUP: stream: rename "csf" and "csb" to "scf" and "scb"
These are the stream connectors, let's give them consistent names. The
patch is large (405 locations) but totally trivial.
2022-05-27 19:33:34 +02:00
Willy Tarreau
4596fe20d9 CLEANUP: conn_stream: tree-wide rename to stconn (stream connector)
This renames the "struct conn_stream" to "struct stconn" and updates
the descriptions in all comments (and the rare help descriptions) to
"stream connector" or "connector". This touches a lot of files but
the change is minimal. The local variables were not even renamed, so
there's still a lot of "cs" everywhere.
2022-05-27 19:33:34 +02:00
Willy Tarreau
798465b02c CLEANUP: conn_stream: rename the conn_stream's endp to sedesc
Just like for the appctx, this is a pointer to a stream endpoint descriptor,
so let's make this explicit and not confuse it with the full endpoint. There
are very few changes thanks to the preliminary refactoring of the flags
manipulation.
2022-05-27 19:33:34 +02:00
Willy Tarreau
65d0597b2b CLEANUP: conn_stream: rename the cs_endpoint's target to "se"
That's the "stream endpoint" pointer. Let's change it now while it's
not much spread. The function __cs_endp_target() wasn't yet renamed
because that will change more globally soon.
2022-05-27 19:33:34 +02:00
Willy Tarreau
b605c4213f CLEANUP: conn_stream: rename the stream endpoint flags CS_EP_* to SE_FL_*
Let's now use the new flag names for the stream endpoint.
2022-05-27 19:33:34 +02:00
Willy Tarreau
0cfcc40812 CLEANUP: conn_stream: apply cs_endp_flags.cocci tree-wide
This changes all main uses of cs->endp->flags to the sc_ep_*() equivalent
by applying coccinelle script cs_endp_flags.cocci.

Note: 143 locations were touched, manually reviewed and found to be OK,
except a single one that was adjusted in cs_reset_endp() where the flags
are read and filtered to be used as-is and not as a boolean, hence was
replaced with sc_ep_get() & $FLAGS.

The script was applied with all includes:

  spatch --in-place --recursive-includes -I include --sp-file $script $files
2022-05-27 19:33:34 +02:00