Commit Graph

713 Commits

Author SHA1 Message Date
Tristan
18582ede05 MEDIUM: socket: add zero-terminated ABNS alternative
When an abstract unix socket is bound by HAProxy (using "abns@" prefix),
NUL bytes are appended at the end of its path until sun_path is filled
(for a total of 108 characters).

Here we add an alternative to pass only the non-NUL length of that path
to connect/bind calls, such that the effective path of the socket's name
is as humanly written. This may be useful to interconnect with existing
softwares that implement abstract sockets with this logic instead of the
default haproxy one.

This is achieved by implementing the "abnsz" socket prefix (instead of
"abns"), which stands for "zero-terminated ABNS". "abnsz" prefix may be
used anywhere "abns" is. Internally, haproxy uses the custom socket
family (AF_CUST_ABNS vs AF_CUST_ABNSZ) to differentiate default abns
sockets from zero-terminated ones.

Documentation was updated and regtest was added.

Fixes GH issues #977 and #2479

Co-authored-by: Aurelien DARRAGON <adarragon@haproxy.com>
2024-10-29 12:15:24 +01:00
Christopher Faulet
e7be13da87 REGTESTS: Never reuse server connection in http-messaging/truncated.vtc
A "Connection: close" header is added to responses to avoid any connection
reuse. This should avoid errors on the client side.
2024-10-17 14:44:01 +02:00
Valentine Krasnobaeva
5bbcdc003a REGTESTS: cli: add delay 0.1 before connect to cli
When vtest starts haproxy process, it loops until the moment, when haproxy
pidfile is created. When pidfile is created, vtest considers that haproxy
process is ready and it starts to perform test commands, in particular, it
connects to CLI. It's not very reliable approach to base the check of the
process readiness on the PID file. After master-worker architecture
refactoring pidfile is created in the early init stage, but master and worker
are not yet finished its initialization routines. So, all mcli tests and some
tests where we sent commands to CLI start to fail regularly.

In vtest at the moment there is no any other approach to check that the
process is really ready. So let's add a delay 0.1s before connecting to CLI in
all mcli tests and in acl_cli_spaces test.
2024-10-16 22:00:58 +02:00
Willy Tarreau
93c9f19af7 REGTESTS: fix a reload race in abns_socket.vtc
This test issues a reload over the master CLI, but it is totally
possible that the master has not yet finished starting up the master
CLI when the command is issued, resulting in a failure. This was much
more visible on the new master-worker model, but definitely affects the
old one and could be the reason for this test to occasionally fail on
the CI.
2024-10-14 19:15:21 +02:00
Aurelien DARRAGON
3f4a788329 REGTESTS: add some tests for 'do-log' action
Now that 'do-log' action may be used for all existing action contexts,
let's add some tests in reg-tests/log/log_profile.vtc to ensure it works
as expected. quic-ini is not tested as it may not be builtin depending on
build options..
2024-10-04 21:38:19 +02:00
Christopher Faulet
27ee292731 MINOR: tcpcheck: Add support for an option host header value for httpchk option
Support for headers and body hidden in the version for the "option httpchk"
directive was removed. However a Host header is mandatory for HTTP/1.1
requests and some servers may return an error if it is not set. For now, to
add it, an "http-check send" rule must be added. But it is not really handy
to use an extra config line for this purpose.

So now, it is possible to set the host header value, a log-format string, as
extra argument to "option httpchk" directive. It must be the fourth argument:

  option httpchk GET / HTTP/1.1 www.srv.com

While this patch is not a bug fix, it is simple enough to be backported if
necessary. On 2.9 and older, lf_init_expr() does not exist and LIST_INIT() must
be used instead.
2024-10-02 10:22:51 +02:00
Aurelien DARRAGON
0c94b2efec REGTESTS: add a test for proxy "log-steps"
Now that proxy "log-steps" keyword was implemented and is usable since
("MEDIUM: log: consider log-steps proxy setting for existing log origins")
let's add some tests for it in reg-tests/log/log_profile.vtc.
2024-09-26 16:53:07 +02:00
Aurelien DARRAGON
cdaa749ba0 REGTESTS: log: fix log-profile.vtc
Add missing wait for Slg4 introduced in f8299bc ("MINOR: log: "drop"
support for log-profile steps"), and missing barrier increase due to
the use of barrier sync, which could have resulted in the regtest
being timing-sentive and thus less-reliable.

Also, the "error" check in Slg4 wasn't even considered because it is
emitted by frontend 4, not frontend 2..

No backport needed unless f8299bc is.
2024-09-23 20:15:47 +02:00
Christopher Faulet
4de6632693 MINOR: proxy: Rename accept-invalid-http-* options
With these options, it is possible to accept some invalid messages that may
considered as unsafe and may result as vulnerabilities. The naming is not
explicit enough on this point. These option must really be considered as
dangerous and only used as a temporary workaround. Unfortunately, when used,
it is probably because there are some legacy and unsupported applications in
place. Nevermind. The documentation warns about the use of these
options. Now the name of the options itself is a warning.

So now, "accept-invalid-http-request" and "accept-invalid-http-response"
options are deprecated and replaced by
"accept-unsafe-violations-in-http-request" and
"accept-unsafe-violations-in-http-response" options.
2024-09-16 22:55:25 +02:00
Willy Tarreau
33deb4babe REGTESTS: shorten a bit the delay for the h1/h2 upgrade test
Commit d6c4ed9a96 ("REGTESTS: h1/h2: Update script testing H1/H2
protocol upgrades") introduced a 0.5 second delay which is higher
than those of most other tests (usually 0.05 or 0.2) and triggers
timeouts on my side. Let's just shorten it to 0.2 since its goal
is only to send data separately.

Note: maybe a barrier approach would be possible, though not
      studied.
2024-09-10 10:36:59 +02:00
Willy Tarreau
036ab62231 REGTESTS: fix random failures with wrong_ip_port_logging.vtc under load
This test has an expect rule for syslog that looks for [cC]D, to
indicate a client abort or timeout during the data phase. The purpose
was to say that when it fails it must be this, but the very low timeout
(1ms) still makes it prone to succeeding if the machine is highly loaded.

This has become more visible since commit e8b1ad4c2b ("BUG/MEDIUM: clock:
also update the date offset on time jumps") because the clock drift
adjustments are more systematic. Since this commit, running 50 such tests
at twice more than the number of CPUs in parallel is sufficient to yield
errors due to some lines appearing as succeeding:

   make reg-tests -- --j $((($(nproc)+1)*2)) --vtestparams -n50 reg-tests/log/wrong_ip_port_logging.vtc

It was observed that pauses up to 300ms were observed in epoll_wait() in
such circumstances, which were properly fixed by the time drift detection..
Another approach would consist in increasing the permitted margin during
which we don't fix the clock drift but that would not be logical since the
base time had really been awaited for.

This should be backported to all stable releases since the commit above
will trigger the issue more often.
2024-09-09 19:38:28 +02:00
Christopher Faulet
d6c4ed9a96 REGTESTS: h1/h2: Update script testing H1/H2 protocol upgrades
"http-messaging/protocol_upgrade.vtc" script was updated to test upgrades
for requests with a payload. It should fail when the request is sent to a H2
server. When sent to a H1 server, it should succeed, except if the server
replies before the end of the request.
2024-09-06 14:18:02 +02:00
Aurelien DARRAGON
f8299bc5ea MINOR: log: "drop" support for log-profile steps
It is now possible to use "drop" keyword for "on" lines under a
log-profile section to specify that no log at all should be emitted for
the specified step (setting an empty format was not sufficient to do so
because only the log payload would be empty, not the log header, thus the
log would still be emitted).

It may be useful to selectively disable logging at specific steps for a
given log target (since the log profile may be set on log directives):

log-profile myprof
  on request format "blabla" sd "custom sd"
  on response drop

New testcase was added to reg-tests/log/log_profiles.vtc
2024-08-19 18:53:01 +02:00
William Lallemand
fe5ddcc490 REGTESTS: mcli: test the pipelined commands on master CLI
A recent fix broke the pipelined command on the master CLI, this
reg-tests implement a simple test that allow to check its right
behavior.

This could be backported as far as 2.6.
2024-08-08 17:29:37 +02:00
Amaury Denoyelle
7a5a30d28a BUG/MINOR: h2: reject extended connect for h2c protocol
This commit prevents forwarding of an HTTP/2 Extended CONNECT when "h2c"
or "h2" token is set as targetted protocol. Contrary to the previous
commit which deals with HTTP/1 mux, this time the request is rejected
and a RESET_STREAM is reported to the client.

This must be backported up to 2.4 after a period of observation.
2024-08-01 18:23:44 +02:00
Amaury Denoyelle
7b89aa5b19 BUG/MINOR: h1: do not forward h2c upgrade header token
haproxy supports tunnel establishment through HTTP Upgrade mechanism.
Since the following commit, extended CONNECT is also supported for
HTTP/2 both on frontend and backend side.

  commit 9bf957335e
  MEDIUM: mux_h2: generate Extended CONNECT from htx upgrade

As specified by HTTP/2 rfc, "h2c" can be used by an HTTP/1.1 client to
request an upgrade to HTTP/2. In haproxy, this is not supported so it
silently ignores this. However, Connection and Upgrade headers are
forwarded as-is on the backend side.

If using HTTP/1 on the backend side and the server supports this upgrade
mechanism, haproxy won't be able to parse the HTTP response. If using
HTTP/2, mux backend tries to incorrectly convert the request to an
Extended CONNECT with h2c protocol, which may also prevent the response
to be transmitted.

To fix this, flag HTTP/1 request with "h2c" or "h2" token in an upgrade
header. On converting the header list to HTX, the upgrade header is
skipped if any of this token is present and the H1_MF_CONN_UPG flag is
removed.

This issue can easily be reproduced using curl --http2 argument to
connect to an HTTP/1 frontend.

This must be backported up to 2.4 after a period of observation.
2024-08-01 18:23:32 +02:00
Christopher Faulet
06547dcf52 REGTESTS: check/spoe: Re-enable the script performing SPOP health-checks
Thanks to previous patches, it is now possible to re-enable the test on SPOP
health-checks support.
2024-07-24 14:19:10 +02:00
William Lallemand
4bac38d088 REGTESTS: ssl: test the issuers-chain-path keyword
Add a reg-test which test the completion of the issuers-chain-path
keyword

Note that it could be interesting to have the loading of a .ocsp
combined with this, but our pki for OCSP tests lacks
the SubjectKeyIdentifier extensions.
2024-07-17 16:52:06 +02:00
Christopher Faulet
1bea73612a MEDIUM: check/spoe: Use SPOP multiplexer to perform SPOP health-checks
The SPOP health-checks are now performed using the SPOP multiplexer. This
will be fixed later, but for now, it is considered as a L4 health-check and
no specific status code is reported. It means the corresponding vtest script
is marked as broken for now.

Functionnaly speaking, the same is performed. A connection is opened, a
HELLO frame is sent to the agent and we wait for the HELLO frame from the
agent in reply. But only L4OK, L4KO or L4TOUT will be reported.

The related issue is #2502.
2024-07-12 15:27:04 +02:00
William Lallemand
c268313f60 REGTESTS: ssl: activate new SSL reg-tests with AWS-LC
Prerequisites are now available in AWS-LC, so we can enable these
reg-tests.

With this patch, aws-lc only has 5 reg-tests that are not working:
- reg-tests/ssl/ssl_reuse.vtc: stateful session resumption is only supported with TLSv1.2
- reg-tests/ssl/ssl_curve_name.vtc: function to extract curve name is not available
- reg-tests/ssl/ssl_errors.vtc: errors are not the same than OpenSSL
- reg-tests/ssl/ssl_dh.vtc: AWS-LC does not support DH
- reg-tests/ssl/ssl_curves.vtc: not working correctly

Which means most of the features are working correctly.
2024-06-17 17:43:22 +02:00
William Lallemand
6da0879083 REGTESTS: ssl: fix some regtests 'feature cmd' start condition
Since patch fde517b ("REGTESTS: wolfssl: temporarly disable some failing
reg-tests") some 'feature cmd' lines have an extra quotation mark, so
they were disable in every cases.

Must be backported to 2.9.
2024-06-17 16:12:57 +02:00
Aurelien DARRAGON
bcad26c814 REGTESTS: log: add a test for log-profile
Try to cover some common use-cases for "log-profile" feature. The tests
mainly focus on log-profile section declaration, and testing the behavior
of logformat / log-tag overriding capabilities.

For now, the use of log-profiles is somewhat limited because we lack
the ability to explicitly trigger the log building process at specific
steps during the stream handling. Indeed, for now we rely on
"option logasap" and proxy log-format string content "hacks" to force
the log emission at some specific steps, thus more tests should be added
over the time, when new mechanisms allowing the emission of logs at
expected processing steps will be added, or if new keywords are added to
the log-profile section.

This test requires versions >= 3.0-dev1
2024-06-13 15:43:10 +02:00
Tim Duesterhus
e349159a34 REGTESTS: Remove REQUIRE_VERSION=2.2 from all tests
HAProxy 2.2 is the lowest supported version, thus this always matches.

see 7aff1bf6b9
2024-05-29 22:36:15 +02:00
Tim Duesterhus
10418b6b5a REGTESTS: Remove REQUIRE_VERSION=2.1 from all tests
HAProxy 2.2 is the lowest supported version, thus this always matches.

see 7aff1bf6b9
2024-05-29 22:36:15 +02:00
Amaury Denoyelle
b0e1f77fea REGTESTS: complete http-reuse test with pool-conn-name
Add new test cases in http_reuse_conn_hash vtest. Ensure new server
parameter "pool-conn-name" is used as expected for idle connection name,
both alone and mixed with a SNI.
2024-05-28 15:00:54 +02:00
Willy Tarreau
45a187304e REGTESTS: acl_cli_spaces: avoid a warning caused by undefined logs
There's a warning being reported in this reg test in the detailed startup
logs because of "log global" and "option httplog" while there's no global
section hence no logger. Let's just drop both options since they're not
relevant to this test.
2024-05-24 17:50:19 +02:00
Christopher Faulet
8d2514e087 BUG/MINOR: http-htx: Support default path during scheme based normalization
As stated in RFC3986, for an absolute-form URI, an empty path should be
normalized to a path of "/". This is part of scheme based normalization
rules. This kind of normalization is already performed for default ports. So
we might as well deal with the case of empty path.

The associated reg-tests was updated accordingly.

This patch should fix the issue #2573. It may be backported as far as 2.4 if
necessary.
2024-05-24 16:17:24 +02:00
Christopher Faulet
eb89a7da33 MAJOR: spoe: Let the SPOE back into the game
This reverts commits 885e40494c and
dff9807188.

We decided to spend some time to refactor and rationnalize the SPOE for the
3.1. Thus there is no reason to still consider it as deprecated for the
3.0. Compatibility between the both versions will be maintained.

See #2502 for more info.
2024-05-22 09:04:38 +02:00
William Lallemand
e6657fd108 MEDIUM: ssl: don't load file by discovering them in crt-store
In commit 55e9e9591 ("MEDIUM: ssl: temporarily load files by detecting
their presence in crt-store"), ssl_sock_load_pem_into_ckch() was
replaced by ssl_sock_load_files_into_ckch() in the crt-store loading.

But the side effect was that we always try to autodetect, and this is
not what we want. This patch reverse this, and add specific code in the
crt-list loading, so we could autodetect in crt-list like it was done
before, but still try to load files when a crt-store filename keyword is
specified.

Example:

These crt-list lines won't autodetect files:

    foobar.crt [key foobar.key issuer foobar.issuer ocsp-update on] *.foo.bar
    foobar.crt [key foobar.key] *.foo.bar

These crt-list lines will autodect files:

    foobar.pem [ocsp-update on] *.foo.bar
    foobar.pem
2024-05-21 18:30:45 +02:00
William Lallemand
d33a5f8e14 REGTESTS: ocsp-update: change the reg-test to support the new crt-store mode
Update the ocsp-update tests for the recent changes:

- Incompatibilities check string changed to match the crt-store one
- The "good configurations" are not good anymore because the
  ckch_conf_cmp() does not compare anymore with a global value.
2024-05-17 17:35:51 +02:00
William Lallemand
55e9e95914 MEDIUM: ssl: temporarily load files by detecting their presence in crt-store
crt-store is maint to be stricter than your common crt argument on a
bind line, and is supposed to be a declarative format.

However, since the 'ocsp-update' was migrated from ssl_conf to
ckch_conf, the .issuer file is not autodetected anymore when adding a
ocsp-update keyword in a crt-list file, which breaks retro-compatibility.

This patch is a quick fix that will disappear once we are able to be
strict on a crt-store and autodetect on a crt-list.
2024-05-17 17:35:51 +02:00
William Lallemand
db09c2168f CLEANUP: ssl/ocsp: remove the deprecated parsing code for "ocsp-update"
Remove the "ocsp-update" keyword handling from the crt-list.

The code was made as an exception everywhere so we could activate the
ocsp-update for an individual certificate.

The feature will still exists but will be parsed as a "crt-store"
keyword which will still be usable in a "crt-list". This will appear in
future commits.

This commit also disable the reg-tests for now.
2024-05-17 17:35:51 +02:00
William Lallemand
ef943c186d REGTESTS: update the ocsp-update tests
Update the ocsp-update tests for the recent changes:

- "tune.ssl.ocsp-update.mode" was renamed iin "ocsp-update.mode"
2024-05-17 14:50:00 +02:00
Christopher Faulet
25bcdb1d95 BUG/MAJOR: h1: Be stricter on request target validation during message parsing
As stated in issue #2565, checks on the request target during H1 message
parsing are not good enough. Invalid paths, not starting by a slash are in
fact parsed as authorities. The same error is repeated at the sample fetch
level. This last point is annoying because routing rules may be fooled. It
is also an issue when the URI or the Host header are updated.

Because the error is repeated at different places, it must be fixed. We
cannot be lax by arguing it is the server's job to accept or reject invalid
request targets. With this patch, we strengthen the checks performed on the
request target during H1 parsing. Idea is to reject invalid requests at this
step to be sure it is safe to manipulate the path or the authority at other
places.

So now, the asterisk-form is only allowed for OPTIONS and OTHER methods.
This last point was added to not reject the H2 preface. In addition, we take
care to have only one asterisk and nothing more. For the CONNECT method, we
take care to have a valid authority-form. All other form are rejected. The
authority-form is now only supported for CONNECT method. No specific check
is performed on the origin-form (except for the CONNECT method). For the
absolute-form, we take care to have a scheme and a valid authority.

These checks are not perfect but should be good enough to properly identify
each part of the request target for a relative small cost. But, it is a
breaking change. Some requests are now be rejected while they was not on
older versions. However, nowadays, it is most probably not an issue.  If it
turns out it's really an issue for legitimate use-cases, an option would be
to supports these kinds of requests when the "accept-invalid-http-request"
option is set, with the consequence of seeing some sample fetches having an
unexpected behavior.

This patch should fix the issue #2665. It MUST NOT be backported. First
because it is a breaking change. And then because by avoiding backporting
it, it remains possible to relax the parsing with the
"accept-invalid-http-request" option.
2024-05-15 21:20:37 +02:00
William Lallemand
8c6f43d382 REGTESTS: ssl: be more verbose with ocsp_compat_check.vtc
the ocsp_compat_check.vtc reg-test is difficult to debug given than the
haproxy output is piped in `grep -q`.

This patch helps by showing the haproxy output as well as the return
code.
2024-05-15 10:36:02 +02:00
Patrick Hemmer
28489021b3 BUG/MINOR: cfgparse: use curproxy global var from config post validation
Previously check_config_validity() had its own curproxy variable. This
resulted in the acl() sample fetch being unable to determine which
proxy was in use when used from within log-format statements. This
change addresses the issue by having the check_config_validity()
function use the global variable instead.
2024-05-06 18:45:47 +02:00
Patrick Hemmer
93d4e99714 BUG/MINOR: acl: support built-in ACLs with acl() sample
Built-in ACLs were not being searched by the acl() sample fetch. This
fixes that so they are searched if no other match is found.
2024-05-06 18:42:54 +02:00
Patrick Hemmer
7c6b410b35 REGTEST: add tests for acl() sample fetch
This adds reg tests for the recently added acl() sample fetch
2024-05-06 18:41:57 +02:00
Amaury Denoyelle
437062255e REGTESTS: replace REQUIRE_VERSION by version_atleast
REQUIRE_VERSION usage is deprecated in regtests for version >= 2.5. This
allows to specify full correct 3.0-dev9 version.
2024-05-03 16:35:12 +02:00
Ilia Shipitsin
a65c6d3574 CLEANUP: assorted typo fixes in the code and comments
This is 42nd iteration of typo fixes
2024-05-03 09:01:36 +02:00
Amaury Denoyelle
e3c976bb81 REGTESTS: stats: add test stats-file counters preload
Define a simple regtest to check stats-file loading on startup. A sample
stats-file is written with some invalid values which should be silently
ignored.
2024-05-02 16:44:28 +02:00
Remi Tricot-Le Breton
83639edd24 REGTESTS: cache: Add test on 'vary' other than accept-encoding
A bug related to vary and the 'accept-encoding' header was fixed in
"BUG/MEDIUM: cache: Vary not working properly on anything other than
accept-encoding". This patch adds tests specific to this bug.
2024-04-29 10:41:48 +02:00
Remi Tricot-Le Breton
d330e3ade5 REGTESTS: ssl: Remove "sleep" calls from ocsp auto update test
Instead of relying on the http client logs for synchronization, use the
specific OCSP logs that are emitted after the newly updated response is
inserted in the tree. This removes the need to wait between the syslog
reception and the insertion that was managed thanks to "sleep" calls.
This regtest can now be switched back to "devel" type instead of "slow".
2024-04-25 17:26:42 +02:00
Amaury Denoyelle
8fc0b18087 MINOR: stats: rename proxy stats
This commit is the first one of a serie which adjust naming convention
for stats module. The objective is to remove ambiguity and better
reflect how stats are implemented, especially since the introduction of
stats module.

This patch renames elements related to proxies statistics. One of the
main change is to rename ST_F_* statistics indexes prefix with the new
name ST_I_PX_*. This remove the reference to field which represents
another concept in the stats module. In the same vein, global
stat_fields variable is renamed metrics_px.
2024-04-22 16:25:18 +02:00
Ilya Shipitsin
ab7f05daba CLEANUP: assorted typo fixes in the code and comments
This is 41st iteration of typo fixes
2024-04-17 11:14:44 +02:00
William Lallemand
860b562b92 REGTESTS: ssl: test simple case of crt-store
Test the crt-store loading with some basic cases.
2024-04-12 15:38:54 +02:00
William Lallemand
4c361a9d8a REGTESTS: ssl: disable ssl/ocsp_auto_update.vtc
Test is broken, keep it disable for now.

  Add test: reg-tests/ssl/ocsp_auto_update.vtc
Testing with haproxy version: 3.0-dev6-9dd928-35
1 tests failed, 0 tests skipped, 0 tests passed
*    diag  0.0 /usr/bin/openssl
*    diag  0.0 /usr/bin/socat
make: *** [Makefile:1177: reg-tests] Error 1
2024-04-02 14:20:39 +02:00
Tim Duesterhus
296e9b8de1 REGTESTS: Remove REQUIRE_VERSION=1.9 from all tests (2)
see also:

2a5fb62ad REGTESTS: Remove REQUIRE_VERSION=1.9 from all tests
2024-04-02 07:27:33 +02:00
Tim Duesterhus
34b13378b5 REGTESTS: Do not use REQUIRE_VERSION for HAProxy 2.5+ (4)
Introduced in:

dfb1cea69 REGTESTS: promex: Adapt script to be less verbose
36d936dd1 REGTESTS: write a full reverse regtest
b57f15158 REGTESTS: provide a reverse-server test with name argument
f0bff2947 REGTESTS: provide a reverse-server test

see also:

fbbbc33df REGTESTS: Do not use REQUIRE_VERSION for HAProxy 2.5+
2024-04-02 07:27:33 +02:00
Christopher Faulet
e91da1dd8e REGTESTS: Fix script about OCSP update compatibility tests
There were two occurrences of the seventh test. I don't know really why, but
this triggered a VTC error:

---- h7    Assert error in _assert_VSB_state(), lib/vsb.c line 104:  Condition((s->s_flags & 0x00020000) == state) not true.  Errno=0 Success

Renumbering tests fixes the script.
2024-03-28 17:23:53 +01:00
Remi Tricot-Le Breton
28dcb7bb64 REGTESTS: ssl: Add functional test for global ocsp-update option
Add tests for the 'tune.ssl.ocsp-update.mode' global option that can be
used to enable ocsp auto update on all certificates.
2024-03-27 11:38:28 +01:00
Remi Tricot-Le Breton
c42132b3d5 REGTESTS: ssl: Add OCSP update compatibility tests
Add tests that focus on the incompatibility checks on ocsp-update mode.
This test will only call "haproxy -c" on multiple configurations that
combine the crt-list 'ocsp-update' option and the global
'tune.ssl.ocsp-update.mode'.
2024-03-27 11:38:28 +01:00
Aurelien DARRAGON
db1cd8f881 OPTIM: http_ext: avoid useless copy in http_7239_extract_{ipv4,ipv6}
In http_7239_extract_{ipv4,ipv6}, we declare a local buffer in order to
use inet_pton() since it requires a valid destination argument (cannot be
NULL). Then, if the caller provided <ip> argument, we copy inet_pton()
result (from local buffer to <ip>).

In fact when the caller provides <ip>, we may directly use <ip> as
inet_pton() dst argument to avoid an useless copy. Thus the local buffer
is only relevant when the user doesn't provide <ip>.

While at it, let's add a missing testcase for the rfc7239_n2nn converter
(to check that http_7239_extract_ipv4() with <ip> provided works properly)

This could be backported in 2.8 with b2bb925 ("MINOR: proxy/http_ext:
introduce proxy forwarded option")
2024-03-25 16:24:15 +01:00
Remi Tricot-Le Breton
10ece2cf66 REGTESTS: ssl: Add checks on ocsp-update log format
Add checks on the ocsp-update's dedicated log format.
2024-03-20 16:12:11 +01:00
Remi Tricot-Le Breton
3f04568ab1 REGTESTS: ssl: Add OCSP related tests
Add tests that combine the OCSP update mechanism and the various
preexisting commands that allow to manipulate certificates and
crt-lists.
2024-03-20 16:12:10 +01:00
Christopher Faulet
885e40494c MINOR: spoe: Add SPOE filters in the exposed deprecated directives
It is the first deprecated directive exposed via the
'expose-deprecated-directives' global option. This way, it is possible to
silent the warning about the SPOE uses.
2024-03-15 11:31:48 +01:00
William Lallemand
4895fdac5a BUG/MAJOR: ssl/ocsp: crash with ocsp when old process exit or using ocsp CLI
This patch reverts 2 fixes that were made in an attempt to fix the
ocsp-update feature used with the 'commit ssl cert' command.

The patches crash the worker when doing a soft-stop when the 'set ssl
ocsp-response' command was used, or during runtime if the ocsp-update
was used.

This was reported in issue #2462 and #2442.

The last patch reverted is the associated reg-test.

Revert "BUG/MEDIUM: ssl: Fix crash when calling "update ssl ocsp-response" when an update is ongoing"
This reverts commit 5e66bf26ec.

Revert "BUG/MEDIUM: ocsp: Separate refcount per instance and per store"
This reverts commit 04b77f84d1b52185fc64735d7d81137479d68b00.

Revert "REGTESTS: ssl: Add OCSP related tests"
This reverts commit acd1b85d3442fc58164bd0fb96e72f3d4b501d15.
2024-02-26 18:04:25 +01:00
Willy Tarreau
5d0dd88ac6 BUG/MINOR: vars/cli: fix missing LF after "get var" output
"get var" on the CLI was also missing an LF, and the vtest as well, so
that fixing only the code breaks the vtest. This must be backported to
2.4 as the issue was brought with commit c35eb38f1d ("MINOR: vars/cli:
add a "get var" CLI command to retrieve global variables").
2024-02-08 18:22:01 +01:00
Willy Tarreau
870e2d3f1f MEDIUM: mux-h2: update session trackers with number of glitches
We now update the session's tracked counters with the observed glitches.
In order to avoid incurring a high cost, e.g. if many small frames contain
issues, we batch the updates around h2_process_demux() by directly passing
the difference. Indeed, for now all functions that increment glitches are
called from h2_process_demux(). If that were to change, we'd just need to
keep the value of the last synced counter in the h2c struct instead of the
stack.

The regtest was updated to verify that the 3rd client that does not cause
issue still sees the counter resulting from client 2's mistakes. The rate
is also verified, considering it shouldn't fail since the period is very
long (1m).
2024-02-08 15:51:49 +01:00
Remi Tricot-Le Breton
66b20aada4 REGTESTS: ssl: Fix empty line in cli command input
The 'set ssl cert' command was failing because of empty lines in the
contents of the PEM file used to perform the update.
We were also missing the issuer in the newly created ckch_store, which
then raised an error when committing the transaction.
2024-02-07 17:10:49 +01:00
Remi Tricot-Le Breton
79d526f6da REGTESTS: ssl: Add OCSP related tests
Add tests that combine the OCSP update mechanism and the various
preexisting commands that allow to manipulate certificates and
crt-lists.
2024-02-07 17:10:49 +01:00
Christopher Faulet
dfb1cea693 REGTESTS: promex: Adapt script to be less verbose
First, checks on the resolver scope were added. Then, because of the recent
changes, the logs emitted by vtest are now too big and this makes the script
fails. So tests on NaN values are now performed on a smaller request. This
reduces enough the logs to pass.

In fact some checks were removed
2024-02-02 09:11:34 +01:00
Willy Tarreau
6e5aa16145 MINOR: connection: add sample fetches to report per-connection glitches
Now with fc_glitches and bc_glitches we can retrieve the number of
detected glitches on a front or back connection. On the backend it
can indicate a bug in a server that may induce frequent reconnections
hence CPU usage in TLS reconnections, and on the frontend it may
indicate an abusive client that may be trying to attack the stack
or to fingerprint it. Small non-zero values are definitely expected
and can be caused by network glitches for example, as well as rare
bugs in the other component (or maybe even in haproxy). These should
never be considered as alarming as long as they remain low (i.e.
much less than one per request). A reg-test is provided.
2024-01-18 17:21:44 +01:00
Willy Tarreau
4cc25f26f9 MEDIUM: http: add the ability to redefine http-err-codes and http-fail-codes
The new global keywords "http-err-codes" and "http-fail-codes" allow to
redefine which HTTP status codes indicate a client-induced error or a
server error, as tracked by stick-table counters. This is only done
globally, though everything was done so that it could easily be extended
to a per-proxy mechanism if there was a real need for this (but it would
eat quite more RAM then).

A simple reg-test was added (http-err-fail.vtc).
2024-01-11 15:10:08 +01:00
Aurelien DARRAGON
3b0bf5097b MINOR: map: mapfile ordering also matters for tree-based match types
Willy made me realize that tree-based matching may also suffer from
out-of-order mapfile loading, as opposed to what's being said in
b546bb6d ("BUG/MINOR: map: list-based matching potential ordering
regression") and the associated REGTEST.

Indeed, in case of duplicated keys, we want to be sure that only the key
that was first seen in the file will be returned (as long as it is not
removed). The above fix is still valid, and the list-based match regtest
will also prevent regressions for tree-based match since mapfile loading
logic is currently match-type agnostic.

But let's clarify that by making both the code comment and the regtest
more precise.
2024-01-11 11:13:54 +01:00
Aurelien DARRAGON
1088f0b969 REGTESTS: add a test to ensure map-ordering is preserved
As shown in "BUG/MINOR: map: list-based matching potential ordering
regression", list-based matching types such as dom are affected by the
order in which elements are loaded from the map.

Since this is historical behavior and existing usages depend on it, we
add a test to prevent future regressions.
2024-01-10 18:02:54 +01:00
Amaury Denoyelle
0627f470df REGTESTS: check attach-srv out of order declaration
Previous patch fixed a regression which caused some config with
attach-srv to be rejected if the rule was declared before the target
server itself. To better detect this kind of error, mix the declaration
order in the corresponding regtest.
2024-01-02 15:52:47 +01:00
Tim Duesterhus
1dcc6a8a96 BUG/MINOR: sample: Make the word converter compatible with -m found
Previously an expression like:

    path,word(2,/) -m found

always returned `true`.

Bug exists since the `word` converter exists. That is:
c9a0f6d023

The same bug was previously fixed for the `field` converter in commit
4381d26edc.

The fix should be backported to 1.6+.
2023-12-01 14:35:47 +01:00
Tim Duesterhus
8f8349fbef REGTESTS: sample: Test the behavior of consecutive delimiters for the field converter
This is in preparation of a follow-up patch to fix the word converter.
2023-12-01 14:35:47 +01:00
Amaury Denoyelle
86e5c607d1 MINOR: rhttp: mark reverse HTTP as experimental
Mark the reverse HTTP feature as experimental. This will allow to adjust
if needed the configuration mechanism with future developments without
maintaining retro-compatibility.

Concretely, each config directives linked to it now requires to specify
first global expose-experimental-directives before. This is the case for
the following directives :
- rhttp@ prefix uses in bind and server lines
- nbconn bind keyword
- attach-srv tcp rule

Each documentation section refering to these keywords are updated to
highlight this new requirement.

Note that this commit has duplicated on several places the code from the
global function check_kw_experimental(). This is because the latter only
work with cfg_keyword type. This is not adapted with bind_kw or
action_kw types. This should be improve in a future patch.
2023-11-30 15:04:27 +01:00
Aurelien DARRAGON
3a1b8ad1b5 REGTESTS: connection: disable http_reuse_be_transparent.vtc if !TPROXY
http_reuse_be_transparent.vtc relies on "transparent" proxy option which
is guarded by the USE_TPROXY ifdef at multiple places in the code.

Hence, executing the above test when haproxy was compiled without the
USE_TPROXY feature (ie: generic target) results in this kind of error:

  ***  h1    debug|[NOTICE]   (1189756) : haproxy version is 2.9-dev1-8fc21e-807
  ***  h1    debug|[NOTICE]   (1189756) : path to executable is ./haproxy
  ***  h1    debug|[ALERT]    (1189756) : config : parsing [/tmp/vtc.1189751.18665e7b/h1/cfg:11]: option 'transparent' is not supported due to build options.
  ***  h1    debug|[ALERT]    (1189756) : config : Error(s) found in configuration file : /tmp/vtc.1189751.18665e7b/h1/cfg

Now we skip the regtest if TPROXY feature is missing.
2023-11-24 16:27:55 +01:00
Willy Tarreau
cd352c0dbe MINOR: log/balance: rename "log-sticky" to "sticky"
After giving it some thought, it could pretty well happen that other
protocols benefit from the sticky algorithm that some used to emulate
using a "stick-on int(0)" or things like this previously. So better
rename it to "sticky" right now instead of having to keep that "log-"
prefix forever. It's still limited to logs, of course, only the algo
is renamed in the config.
2023-11-23 18:21:31 +01:00
Ilya Shipitsin
80813cdd2a CLEANUP: assorted typo fixes in the code and comments
This is 37th iteration of typo fixes
2023-11-23 16:23:14 +01:00
Willy Tarreau
42faad48f6 REGTESTS: http: add a test to validate chunked responses delivery
I've had this test here never committed over the last 2.5 years, that
works fine and I didn't notice it was not part of the tree. It makes a
server return odd-sized chunked responses with short pauses between half
of thems and verifies they're not truncated on the client. It may detect
eventually state machine breakages, so better commit it.
2023-11-18 11:16:41 +01:00
Aurelien DARRAGON
b61147fd2a MEDIUM: log/balance: merge tcp/http algo with log ones
"log-balance" directive was recently introduced to configure the
balancing algorithm to use when in a log backend. However, it is
confusing and it causes issues when used in default section.

In this patch, we take another approach: first we remove the
"log-balance" directive, and instead we rely on existing "balance"
directive to configure log load balancing in log backend.

Some algorithms such as roundrobin can be used as-is in a log backend,
and for log-only algorithms, they are implemented as "log-$name" inside
the "backend" directive.

The documentation was updated accordingly.
2023-11-18 11:16:21 +01:00
William Lallemand
3379441e26 REGTESTS: try to activate again the seamless reload test with the master CLI
Since the reload is now synchronous over the master CLI, try to reload
with it. This was a problem before with the signals because it wasn't
possible to wait for the end of the reload before sending the requests.

This activate again this test, we will see if it's more stable or we
will deactivate it again..
2023-11-17 17:11:42 +01:00
Christopher Faulet
af7db3a43c REGTESTS: http: Improve script testing abortonclose option
We now take care to properly handle the abortonclose close option if it is
set on the backend and be sure we ignore it when it is set on the frontend
(inherited from the defaults section).
2023-11-14 11:01:51 +01:00
William Lallemand
59b313832a REGTESTS: startup: -conf-OK requires -V with current VTest
Current version of VTest tests the output of "haproxy -c" instead of the
return code. Since we don't output anymore when the configuration is
valid, this broke the test. (a06f621).

This fixes the issue by adding the -V when doing a -conf-OK. But this
must fixed in VTest.
2023-11-13 14:57:26 +01:00
Alexander Stephan
ce7501de79 MINOR: connection: Send out generic, user-defined server TLVs
To follow-up the implementation of the new set-proxy-v2-tlv-fmt
keyword in the server, the connection is updated to use the previously
allocated TLVs. If no value was specified, we send out an empty TLV.
As the feature is fully working with this commit, documentation and a
test for the server and default-server are added as well.
2023-11-04 04:56:59 +01:00
Ruei-Bang Chen
7a1ec235cd MINOR: sample: Add fetcher for getting all cookie names
This new fetcher can be used to extract the list of cookie names from
Cookie request header or from Set-Cookie response header depending on
the stream direction. There is an optional argument that can be used
as the delimiter (which is assumed to be the first character of the
argument) between cookie names. The default delimiter is comma (,).

Note that we will treat the Cookie request header as a semi-colon
separated list of cookies and each Set-Cookie response header as
a single cookie and extract the cookie names accordingly.
2023-11-03 09:57:06 +01:00
William Lallemand
9496e7e888 REGTESTS: ssl: update the filters test for TLSv1.3 and sigalgs
Signature algorithms allows us to select the right certificates when
using TLSv1.3. This patch update the ssl_crt-list_filters.vtc regtest to
do more precise testing with TLSv1.3 in addition to TLSv1.2.

This allow us to test correctly bug #2300.

It could be backported to 2.8 with the previous fix for certificate
selection.
2023-10-26 19:23:04 +02:00
Jens Popp
f66b9f6018 MINOR: sample: Added support for Arrays in sample_conv_json_query in sample.c
Method now returns the content of Json Arrays, if it is specified in
Json Path as String. The start and end character is a square bracket. Any
complex object in the array is returned as Json, so that you might get Arrays
of Array or objects. Only recommended for Arrays of simple types (e.g.,
String or int) which will be returned as CSV String. Also updated
documentation and fixed issue with parenthesis and other changes from
comments.

This patch was discussed in issue #2281.

Signed-off-by: William Lallemand <wlallemand@haproxy.com>
2023-10-20 18:42:05 +02:00
Amaury Denoyelle
11ccd8d7cb REGTESTS: remove maxconn from rhttp bind line
The maxconn keyword is not used anymore for reverse HTTP bind. It has
been replaced recently by the new keyword nbconn. As it's default value
is 1, it can be safely removed from the regtest without affecting its
behavior.
2023-10-20 17:28:08 +02:00
Amaury Denoyelle
e05edf71df MINOR: cfgparse: rename "rev@" prefix to "rhttp@"
'rev@' was used to specify a bind/server used with reverse HTTP
transport. This notation was deemed not explicit enough. Rename it
'rhttp@' instead.
2023-10-20 14:44:37 +02:00
Amaury Denoyelle
9d4c7c1151 MINOR: server: convert @reverse to rev@ standard format
Remove the recently introduced '@reverse' notation for HTTP reverse
servers. Instead, reuse the 'rev@' prefix already defined for bind
lines.
2023-10-20 14:44:37 +02:00
Christopher Faulet
d1eab38131 REGTESTS: Reenable HTTP tests about splicing 2023-10-17 18:51:13 +02:00
Christopher Faulet
8b89fe3d8f MINOR: stconn: Temporarily remove kernel splicing support
mux-to-mux fast-forwarding will be added. To avoid mix with the splicing and
simplify the commits, the kernel splicing support is removed from the
stconn. CF_KERN_SPLICING flag is removed and the support is no longer tested
in process_stream().

In the stconn part, rcv_pipe() callback function is no longer called.

Reg-tests scripts testing the kernel splicing are temporarly marked as
broken.
2023-10-17 18:51:13 +02:00
Vladimir Vdovin
70d2d9aefc MINOR: support for http-response set-timeout
Added set-timeout action for http-response. Adapted reg-tests and
documentation.
2023-10-17 08:27:33 +02:00
Aurelien DARRAGON
7e70b8858f REGTEST: add a test for log-backend used as a log target
This regtest declares and uses 3 log backends, one of which has TCP syslog
servers declared in it and other ones UDP syslog servers.

Some tests aims at testing log distribution reliability by leveraging the
log-balance hash algorithm with a key extracted from the request URL, and
the dummy vtest syslog servers ensure that messages are sent to the
correct endpoint. Overall this regtest covers essential parts of the log
message distribution and log-balancing logic involved with log backends.

It also leverages the log-forward section to perform the TCP->UDP
translation required to test UDP endpoints since vtest syslog servers
work in UDP mode.

Finally, we have some tests to ensure that the server queuing/dequeuing
and failover (backup) logics work properly.
2023-10-13 10:05:06 +02:00
William Lallemand
fde517ba66 REGTESTS: wolfssl: temporarly disable some failing reg-tests
Temporarly disable the last failing reg-tests with WolfSSL in order to
be able to setup a CI.
2023-10-09 23:05:18 +02:00
William Lallemand
c24cc33c0f REGTESTS: ssl: disable ssl_dh.vtc for WolfSSL
Skip the ssl_dh reg-tests which is not working for WolfSSL.
2023-10-09 22:11:37 +02:00
William Lallemand
1105524cbe REGTESTS: ssl: update common.pem with the new pki
Update the SSL reg-test in order to use the new pki.
2023-10-09 21:58:21 +02:00
William Lallemand
8c1464098b REGTESTS: pki: add a pki for SSL tests
Add a PKI generated with cfssl in order to generated easily certificates
for the reg-tests.
2023-10-09 21:54:31 +02:00
Remi Tricot-Le Breton
a5e96425a2 MEDIUM: cache: Add "Origin" header to secondary cache key
This patch add a hash of the Origin header to the cache's secondary key.
This enables to manage store responses that have a "Vary: Origin" header
in the cache when vary is enabled.
This cannot be considered as a means to manage CORS requests though, it
only processes the Origin header and hashes the presented value without
any form of URI normalization.

This need was expressed by Philipp Hossner in GitHub issue #251.

Co-Authored-by: Philipp Hossner <philipp.hossner@posteo.de>
2023-10-05 10:53:54 +02:00
Christopher Faulet
6f6aed92b7 REGTESTS: filters: Don't set C-L header in the successful response to CONNECT
in random-forwarding.vtc script, adding "Content-Lnegth; 0" header in the
successful response to the CONNECT request is invalid but it may also lead
to wrong check on the response. "rxresp" directive don"t handle CONNECT
response. Thus "-no_obj" must be added instead, to be sure the payload won't
be retrieved or expected.
2023-10-04 15:34:18 +02:00
Vladimir Vdovin
f8b81f6eb7 MINOR: support for http-request set-timeout client
Added set-timeout for frontend side of session, so it can be used to set
custom per-client timeouts if needed. Added cur_client_timeout to fetch
client timeout samples.
2023-09-28 08:49:22 +02:00
Lokesh Jindal
915e48675a MEDIUM: sample: Enhances converter "bytes" to take variable names as arguments
Prior to this commit, converter "bytes" takes only integer values as
arguments.  After this commit, it can take variable names as inputs.
This allows us to dynamically determine the offset/length and capture
them in variables.  These variables can then be used with the converter.
Example use case: parsing a token present in a request header.
2023-09-22 08:48:51 +02:00
William Lallemand
61b6a4da6c REGTESTS: ssl: skip generate-certificates test w/ wolfSSL
WolfSSL does not seem to work correctly with the generate-certificates
features. This patch disables it temporarly.

    ssl-max-ver TLSv1.2 seems to be a problem in the reg-test and
    wolfSSL but without it it's not able to generate correctly the cert:

    ***  h1    debug|00000004:clear-lst.accept(0007)=0028 from [127.0.0.1:35956] ALPN=<none>
    ***  h1    debug|00000004:clear-lst.clireq[0028:ffffffff]: GET / HTTP/1.1
    ***  h1    debug|00000004:clear-lst.clihdr[0028:ffffffff]: x-sni: unknown-sni.com
    ***  h1    debug|00000004:clear-lst.clihdr[0028:ffffffff]: host: 127.0.0.1
    ***  h1    debug|fd[0x29] OpenSSL error[0x13d] : need the private key
    ***  h1    debug|<134>Sep 20 15:42:58 haproxy[165743]: unix:1 [20/Sep/2023:15:42:58.042] ssl-lst/1: SSL handshake failure (need the private key)
    **** dT    1.072
    ***  h1    debug|fd[0x2a] OpenSSL error[0x13d] : need the private key
    ***  h1    debug|<134>Sep 20 15:42:59 haproxy[165743]: unix:1 [20/Sep/2023:15:42:59.044] ssl-lst/1: SSL handshake failure (need the private key)
    **** dT    2.075
    ***  h1    debug|fd[0x29] OpenSSL error[0x13d] : need the private key
    ***  h1    debug|<134>Sep 20 15:43:00 haproxy[165743]: unix:1 [20/Sep/2023:15:43:00.046] ssl-lst/1: SSL handshake failure (need the private key)
    **** dT    3.079
    ***  h1    debug|fd[0x29] OpenSSL error[0x13d] : need the private key
    ***  h1    debug|<134>Sep 20 15:43:01 haproxy[165743]: unix:1 [20/Sep/2023:15:43:01.050] ssl-lst/1: SSL handshake failure (need the private key)
    **** dT    3.080
    ***  h1    debug|00000004:default_backend.clicls[0028:0023]
    ***  h1    debug|00000004:default_backend.closed[0028:0023]
    ***  h1    debug|<134>Sep 20 15:43:01 haproxy[165743]: 127.0.0.1:35956 [20/Sep/2023:15:42:58.042] clear-lst default_backend/s1 0/0/-1/-1/+3009 503 +217 - - SC-- 3/1/0/0/3 0/0 "GET / HTTP/1.1" 0/-/-/-/0 -/-/-
    **** c3    rxhdr|HTTP/1.1 503 Service Unavailable\r
    **** c3    rxhdr|content-length: 107\r
    **** c3    rxhdr|cache-control: no-cache\r
    **** c3    rxhdr|content-type: text/html\r
    **** c3    rxhdr|\r
2023-09-20 16:02:16 +02:00
William Lallemand
64a4b44a44 REGTESTS: ssl: skip OCSP test w/ WolfSSL
The OCSP test does not seem to be working correctly with wolfSSL.

i2d_OCSP_CERTID(data->ocsp_cid, NULL); always returns 0.

Skip it for now.
2023-09-20 15:23:32 +02:00
William Lallemand
15e591b6e0 MINOR: ssl: add support for 'curves' keyword on server lines
This patch implements the 'curves' keyword on server lines as well as
the 'ssl-default-server-curves' keyword in the global section.

It also add the keyword on the server line in the ssl_curves reg-test.

These keywords allow the configuration of the curves list for a server.
2023-09-07 23:29:10 +02:00