Commit Graph

686 Commits

Author SHA1 Message Date
Christopher Faulet
eb89a7da33 MAJOR: spoe: Let the SPOE back into the game
This reverts commits 885e40494c and
dff9807188.

We decided to spend some time to refactor and rationnalize the SPOE for the
3.1. Thus there is no reason to still consider it as deprecated for the
3.0. Compatibility between the both versions will be maintained.

See #2502 for more info.
2024-05-22 09:04:38 +02:00
William Lallemand
e6657fd108 MEDIUM: ssl: don't load file by discovering them in crt-store
In commit 55e9e9591 ("MEDIUM: ssl: temporarily load files by detecting
their presence in crt-store"), ssl_sock_load_pem_into_ckch() was
replaced by ssl_sock_load_files_into_ckch() in the crt-store loading.

But the side effect was that we always try to autodetect, and this is
not what we want. This patch reverse this, and add specific code in the
crt-list loading, so we could autodetect in crt-list like it was done
before, but still try to load files when a crt-store filename keyword is
specified.

Example:

These crt-list lines won't autodetect files:

    foobar.crt [key foobar.key issuer foobar.issuer ocsp-update on] *.foo.bar
    foobar.crt [key foobar.key] *.foo.bar

These crt-list lines will autodect files:

    foobar.pem [ocsp-update on] *.foo.bar
    foobar.pem
2024-05-21 18:30:45 +02:00
William Lallemand
d33a5f8e14 REGTESTS: ocsp-update: change the reg-test to support the new crt-store mode
Update the ocsp-update tests for the recent changes:

- Incompatibilities check string changed to match the crt-store one
- The "good configurations" are not good anymore because the
  ckch_conf_cmp() does not compare anymore with a global value.
2024-05-17 17:35:51 +02:00
William Lallemand
55e9e95914 MEDIUM: ssl: temporarily load files by detecting their presence in crt-store
crt-store is maint to be stricter than your common crt argument on a
bind line, and is supposed to be a declarative format.

However, since the 'ocsp-update' was migrated from ssl_conf to
ckch_conf, the .issuer file is not autodetected anymore when adding a
ocsp-update keyword in a crt-list file, which breaks retro-compatibility.

This patch is a quick fix that will disappear once we are able to be
strict on a crt-store and autodetect on a crt-list.
2024-05-17 17:35:51 +02:00
William Lallemand
db09c2168f CLEANUP: ssl/ocsp: remove the deprecated parsing code for "ocsp-update"
Remove the "ocsp-update" keyword handling from the crt-list.

The code was made as an exception everywhere so we could activate the
ocsp-update for an individual certificate.

The feature will still exists but will be parsed as a "crt-store"
keyword which will still be usable in a "crt-list". This will appear in
future commits.

This commit also disable the reg-tests for now.
2024-05-17 17:35:51 +02:00
William Lallemand
ef943c186d REGTESTS: update the ocsp-update tests
Update the ocsp-update tests for the recent changes:

- "tune.ssl.ocsp-update.mode" was renamed iin "ocsp-update.mode"
2024-05-17 14:50:00 +02:00
Christopher Faulet
25bcdb1d95 BUG/MAJOR: h1: Be stricter on request target validation during message parsing
As stated in issue #2565, checks on the request target during H1 message
parsing are not good enough. Invalid paths, not starting by a slash are in
fact parsed as authorities. The same error is repeated at the sample fetch
level. This last point is annoying because routing rules may be fooled. It
is also an issue when the URI or the Host header are updated.

Because the error is repeated at different places, it must be fixed. We
cannot be lax by arguing it is the server's job to accept or reject invalid
request targets. With this patch, we strengthen the checks performed on the
request target during H1 parsing. Idea is to reject invalid requests at this
step to be sure it is safe to manipulate the path or the authority at other
places.

So now, the asterisk-form is only allowed for OPTIONS and OTHER methods.
This last point was added to not reject the H2 preface. In addition, we take
care to have only one asterisk and nothing more. For the CONNECT method, we
take care to have a valid authority-form. All other form are rejected. The
authority-form is now only supported for CONNECT method. No specific check
is performed on the origin-form (except for the CONNECT method). For the
absolute-form, we take care to have a scheme and a valid authority.

These checks are not perfect but should be good enough to properly identify
each part of the request target for a relative small cost. But, it is a
breaking change. Some requests are now be rejected while they was not on
older versions. However, nowadays, it is most probably not an issue.  If it
turns out it's really an issue for legitimate use-cases, an option would be
to supports these kinds of requests when the "accept-invalid-http-request"
option is set, with the consequence of seeing some sample fetches having an
unexpected behavior.

This patch should fix the issue #2665. It MUST NOT be backported. First
because it is a breaking change. And then because by avoiding backporting
it, it remains possible to relax the parsing with the
"accept-invalid-http-request" option.
2024-05-15 21:20:37 +02:00
William Lallemand
8c6f43d382 REGTESTS: ssl: be more verbose with ocsp_compat_check.vtc
the ocsp_compat_check.vtc reg-test is difficult to debug given than the
haproxy output is piped in `grep -q`.

This patch helps by showing the haproxy output as well as the return
code.
2024-05-15 10:36:02 +02:00
Patrick Hemmer
28489021b3 BUG/MINOR: cfgparse: use curproxy global var from config post validation
Previously check_config_validity() had its own curproxy variable. This
resulted in the acl() sample fetch being unable to determine which
proxy was in use when used from within log-format statements. This
change addresses the issue by having the check_config_validity()
function use the global variable instead.
2024-05-06 18:45:47 +02:00
Patrick Hemmer
93d4e99714 BUG/MINOR: acl: support built-in ACLs with acl() sample
Built-in ACLs were not being searched by the acl() sample fetch. This
fixes that so they are searched if no other match is found.
2024-05-06 18:42:54 +02:00
Patrick Hemmer
7c6b410b35 REGTEST: add tests for acl() sample fetch
This adds reg tests for the recently added acl() sample fetch
2024-05-06 18:41:57 +02:00
Amaury Denoyelle
437062255e REGTESTS: replace REQUIRE_VERSION by version_atleast
REQUIRE_VERSION usage is deprecated in regtests for version >= 2.5. This
allows to specify full correct 3.0-dev9 version.
2024-05-03 16:35:12 +02:00
Ilia Shipitsin
a65c6d3574 CLEANUP: assorted typo fixes in the code and comments
This is 42nd iteration of typo fixes
2024-05-03 09:01:36 +02:00
Amaury Denoyelle
e3c976bb81 REGTESTS: stats: add test stats-file counters preload
Define a simple regtest to check stats-file loading on startup. A sample
stats-file is written with some invalid values which should be silently
ignored.
2024-05-02 16:44:28 +02:00
Remi Tricot-Le Breton
83639edd24 REGTESTS: cache: Add test on 'vary' other than accept-encoding
A bug related to vary and the 'accept-encoding' header was fixed in
"BUG/MEDIUM: cache: Vary not working properly on anything other than
accept-encoding". This patch adds tests specific to this bug.
2024-04-29 10:41:48 +02:00
Remi Tricot-Le Breton
d330e3ade5 REGTESTS: ssl: Remove "sleep" calls from ocsp auto update test
Instead of relying on the http client logs for synchronization, use the
specific OCSP logs that are emitted after the newly updated response is
inserted in the tree. This removes the need to wait between the syslog
reception and the insertion that was managed thanks to "sleep" calls.
This regtest can now be switched back to "devel" type instead of "slow".
2024-04-25 17:26:42 +02:00
Amaury Denoyelle
8fc0b18087 MINOR: stats: rename proxy stats
This commit is the first one of a serie which adjust naming convention
for stats module. The objective is to remove ambiguity and better
reflect how stats are implemented, especially since the introduction of
stats module.

This patch renames elements related to proxies statistics. One of the
main change is to rename ST_F_* statistics indexes prefix with the new
name ST_I_PX_*. This remove the reference to field which represents
another concept in the stats module. In the same vein, global
stat_fields variable is renamed metrics_px.
2024-04-22 16:25:18 +02:00
Ilya Shipitsin
ab7f05daba CLEANUP: assorted typo fixes in the code and comments
This is 41st iteration of typo fixes
2024-04-17 11:14:44 +02:00
William Lallemand
860b562b92 REGTESTS: ssl: test simple case of crt-store
Test the crt-store loading with some basic cases.
2024-04-12 15:38:54 +02:00
William Lallemand
4c361a9d8a REGTESTS: ssl: disable ssl/ocsp_auto_update.vtc
Test is broken, keep it disable for now.

  Add test: reg-tests/ssl/ocsp_auto_update.vtc
Testing with haproxy version: 3.0-dev6-9dd928-35
1 tests failed, 0 tests skipped, 0 tests passed
*    diag  0.0 /usr/bin/openssl
*    diag  0.0 /usr/bin/socat
make: *** [Makefile:1177: reg-tests] Error 1
2024-04-02 14:20:39 +02:00
Tim Duesterhus
296e9b8de1 REGTESTS: Remove REQUIRE_VERSION=1.9 from all tests (2)
see also:

2a5fb62ad REGTESTS: Remove REQUIRE_VERSION=1.9 from all tests
2024-04-02 07:27:33 +02:00
Tim Duesterhus
34b13378b5 REGTESTS: Do not use REQUIRE_VERSION for HAProxy 2.5+ (4)
Introduced in:

dfb1cea69 REGTESTS: promex: Adapt script to be less verbose
36d936dd1 REGTESTS: write a full reverse regtest
b57f15158 REGTESTS: provide a reverse-server test with name argument
f0bff2947 REGTESTS: provide a reverse-server test

see also:

fbbbc33df REGTESTS: Do not use REQUIRE_VERSION for HAProxy 2.5+
2024-04-02 07:27:33 +02:00
Christopher Faulet
e91da1dd8e REGTESTS: Fix script about OCSP update compatibility tests
There were two occurrences of the seventh test. I don't know really why, but
this triggered a VTC error:

---- h7    Assert error in _assert_VSB_state(), lib/vsb.c line 104:  Condition((s->s_flags & 0x00020000) == state) not true.  Errno=0 Success

Renumbering tests fixes the script.
2024-03-28 17:23:53 +01:00
Remi Tricot-Le Breton
28dcb7bb64 REGTESTS: ssl: Add functional test for global ocsp-update option
Add tests for the 'tune.ssl.ocsp-update.mode' global option that can be
used to enable ocsp auto update on all certificates.
2024-03-27 11:38:28 +01:00
Remi Tricot-Le Breton
c42132b3d5 REGTESTS: ssl: Add OCSP update compatibility tests
Add tests that focus on the incompatibility checks on ocsp-update mode.
This test will only call "haproxy -c" on multiple configurations that
combine the crt-list 'ocsp-update' option and the global
'tune.ssl.ocsp-update.mode'.
2024-03-27 11:38:28 +01:00
Aurelien DARRAGON
db1cd8f881 OPTIM: http_ext: avoid useless copy in http_7239_extract_{ipv4,ipv6}
In http_7239_extract_{ipv4,ipv6}, we declare a local buffer in order to
use inet_pton() since it requires a valid destination argument (cannot be
NULL). Then, if the caller provided <ip> argument, we copy inet_pton()
result (from local buffer to <ip>).

In fact when the caller provides <ip>, we may directly use <ip> as
inet_pton() dst argument to avoid an useless copy. Thus the local buffer
is only relevant when the user doesn't provide <ip>.

While at it, let's add a missing testcase for the rfc7239_n2nn converter
(to check that http_7239_extract_ipv4() with <ip> provided works properly)

This could be backported in 2.8 with b2bb925 ("MINOR: proxy/http_ext:
introduce proxy forwarded option")
2024-03-25 16:24:15 +01:00
Remi Tricot-Le Breton
10ece2cf66 REGTESTS: ssl: Add checks on ocsp-update log format
Add checks on the ocsp-update's dedicated log format.
2024-03-20 16:12:11 +01:00
Remi Tricot-Le Breton
3f04568ab1 REGTESTS: ssl: Add OCSP related tests
Add tests that combine the OCSP update mechanism and the various
preexisting commands that allow to manipulate certificates and
crt-lists.
2024-03-20 16:12:10 +01:00
Christopher Faulet
885e40494c MINOR: spoe: Add SPOE filters in the exposed deprecated directives
It is the first deprecated directive exposed via the
'expose-deprecated-directives' global option. This way, it is possible to
silent the warning about the SPOE uses.
2024-03-15 11:31:48 +01:00
William Lallemand
4895fdac5a BUG/MAJOR: ssl/ocsp: crash with ocsp when old process exit or using ocsp CLI
This patch reverts 2 fixes that were made in an attempt to fix the
ocsp-update feature used with the 'commit ssl cert' command.

The patches crash the worker when doing a soft-stop when the 'set ssl
ocsp-response' command was used, or during runtime if the ocsp-update
was used.

This was reported in issue #2462 and #2442.

The last patch reverted is the associated reg-test.

Revert "BUG/MEDIUM: ssl: Fix crash when calling "update ssl ocsp-response" when an update is ongoing"
This reverts commit 5e66bf26ec.

Revert "BUG/MEDIUM: ocsp: Separate refcount per instance and per store"
This reverts commit 04b77f84d1b52185fc64735d7d81137479d68b00.

Revert "REGTESTS: ssl: Add OCSP related tests"
This reverts commit acd1b85d3442fc58164bd0fb96e72f3d4b501d15.
2024-02-26 18:04:25 +01:00
Willy Tarreau
5d0dd88ac6 BUG/MINOR: vars/cli: fix missing LF after "get var" output
"get var" on the CLI was also missing an LF, and the vtest as well, so
that fixing only the code breaks the vtest. This must be backported to
2.4 as the issue was brought with commit c35eb38f1d ("MINOR: vars/cli:
add a "get var" CLI command to retrieve global variables").
2024-02-08 18:22:01 +01:00
Willy Tarreau
870e2d3f1f MEDIUM: mux-h2: update session trackers with number of glitches
We now update the session's tracked counters with the observed glitches.
In order to avoid incurring a high cost, e.g. if many small frames contain
issues, we batch the updates around h2_process_demux() by directly passing
the difference. Indeed, for now all functions that increment glitches are
called from h2_process_demux(). If that were to change, we'd just need to
keep the value of the last synced counter in the h2c struct instead of the
stack.

The regtest was updated to verify that the 3rd client that does not cause
issue still sees the counter resulting from client 2's mistakes. The rate
is also verified, considering it shouldn't fail since the period is very
long (1m).
2024-02-08 15:51:49 +01:00
Remi Tricot-Le Breton
66b20aada4 REGTESTS: ssl: Fix empty line in cli command input
The 'set ssl cert' command was failing because of empty lines in the
contents of the PEM file used to perform the update.
We were also missing the issuer in the newly created ckch_store, which
then raised an error when committing the transaction.
2024-02-07 17:10:49 +01:00
Remi Tricot-Le Breton
79d526f6da REGTESTS: ssl: Add OCSP related tests
Add tests that combine the OCSP update mechanism and the various
preexisting commands that allow to manipulate certificates and
crt-lists.
2024-02-07 17:10:49 +01:00
Christopher Faulet
dfb1cea693 REGTESTS: promex: Adapt script to be less verbose
First, checks on the resolver scope were added. Then, because of the recent
changes, the logs emitted by vtest are now too big and this makes the script
fails. So tests on NaN values are now performed on a smaller request. This
reduces enough the logs to pass.

In fact some checks were removed
2024-02-02 09:11:34 +01:00
Willy Tarreau
6e5aa16145 MINOR: connection: add sample fetches to report per-connection glitches
Now with fc_glitches and bc_glitches we can retrieve the number of
detected glitches on a front or back connection. On the backend it
can indicate a bug in a server that may induce frequent reconnections
hence CPU usage in TLS reconnections, and on the frontend it may
indicate an abusive client that may be trying to attack the stack
or to fingerprint it. Small non-zero values are definitely expected
and can be caused by network glitches for example, as well as rare
bugs in the other component (or maybe even in haproxy). These should
never be considered as alarming as long as they remain low (i.e.
much less than one per request). A reg-test is provided.
2024-01-18 17:21:44 +01:00
Willy Tarreau
4cc25f26f9 MEDIUM: http: add the ability to redefine http-err-codes and http-fail-codes
The new global keywords "http-err-codes" and "http-fail-codes" allow to
redefine which HTTP status codes indicate a client-induced error or a
server error, as tracked by stick-table counters. This is only done
globally, though everything was done so that it could easily be extended
to a per-proxy mechanism if there was a real need for this (but it would
eat quite more RAM then).

A simple reg-test was added (http-err-fail.vtc).
2024-01-11 15:10:08 +01:00
Aurelien DARRAGON
3b0bf5097b MINOR: map: mapfile ordering also matters for tree-based match types
Willy made me realize that tree-based matching may also suffer from
out-of-order mapfile loading, as opposed to what's being said in
b546bb6d ("BUG/MINOR: map: list-based matching potential ordering
regression") and the associated REGTEST.

Indeed, in case of duplicated keys, we want to be sure that only the key
that was first seen in the file will be returned (as long as it is not
removed). The above fix is still valid, and the list-based match regtest
will also prevent regressions for tree-based match since mapfile loading
logic is currently match-type agnostic.

But let's clarify that by making both the code comment and the regtest
more precise.
2024-01-11 11:13:54 +01:00
Aurelien DARRAGON
1088f0b969 REGTESTS: add a test to ensure map-ordering is preserved
As shown in "BUG/MINOR: map: list-based matching potential ordering
regression", list-based matching types such as dom are affected by the
order in which elements are loaded from the map.

Since this is historical behavior and existing usages depend on it, we
add a test to prevent future regressions.
2024-01-10 18:02:54 +01:00
Amaury Denoyelle
0627f470df REGTESTS: check attach-srv out of order declaration
Previous patch fixed a regression which caused some config with
attach-srv to be rejected if the rule was declared before the target
server itself. To better detect this kind of error, mix the declaration
order in the corresponding regtest.
2024-01-02 15:52:47 +01:00
Tim Duesterhus
1dcc6a8a96 BUG/MINOR: sample: Make the word converter compatible with -m found
Previously an expression like:

    path,word(2,/) -m found

always returned `true`.

Bug exists since the `word` converter exists. That is:
c9a0f6d023

The same bug was previously fixed for the `field` converter in commit
4381d26edc.

The fix should be backported to 1.6+.
2023-12-01 14:35:47 +01:00
Tim Duesterhus
8f8349fbef REGTESTS: sample: Test the behavior of consecutive delimiters for the field converter
This is in preparation of a follow-up patch to fix the word converter.
2023-12-01 14:35:47 +01:00
Amaury Denoyelle
86e5c607d1 MINOR: rhttp: mark reverse HTTP as experimental
Mark the reverse HTTP feature as experimental. This will allow to adjust
if needed the configuration mechanism with future developments without
maintaining retro-compatibility.

Concretely, each config directives linked to it now requires to specify
first global expose-experimental-directives before. This is the case for
the following directives :
- rhttp@ prefix uses in bind and server lines
- nbconn bind keyword
- attach-srv tcp rule

Each documentation section refering to these keywords are updated to
highlight this new requirement.

Note that this commit has duplicated on several places the code from the
global function check_kw_experimental(). This is because the latter only
work with cfg_keyword type. This is not adapted with bind_kw or
action_kw types. This should be improve in a future patch.
2023-11-30 15:04:27 +01:00
Aurelien DARRAGON
3a1b8ad1b5 REGTESTS: connection: disable http_reuse_be_transparent.vtc if !TPROXY
http_reuse_be_transparent.vtc relies on "transparent" proxy option which
is guarded by the USE_TPROXY ifdef at multiple places in the code.

Hence, executing the above test when haproxy was compiled without the
USE_TPROXY feature (ie: generic target) results in this kind of error:

  ***  h1    debug|[NOTICE]   (1189756) : haproxy version is 2.9-dev1-8fc21e-807
  ***  h1    debug|[NOTICE]   (1189756) : path to executable is ./haproxy
  ***  h1    debug|[ALERT]    (1189756) : config : parsing [/tmp/vtc.1189751.18665e7b/h1/cfg:11]: option 'transparent' is not supported due to build options.
  ***  h1    debug|[ALERT]    (1189756) : config : Error(s) found in configuration file : /tmp/vtc.1189751.18665e7b/h1/cfg

Now we skip the regtest if TPROXY feature is missing.
2023-11-24 16:27:55 +01:00
Willy Tarreau
cd352c0dbe MINOR: log/balance: rename "log-sticky" to "sticky"
After giving it some thought, it could pretty well happen that other
protocols benefit from the sticky algorithm that some used to emulate
using a "stick-on int(0)" or things like this previously. So better
rename it to "sticky" right now instead of having to keep that "log-"
prefix forever. It's still limited to logs, of course, only the algo
is renamed in the config.
2023-11-23 18:21:31 +01:00
Ilya Shipitsin
80813cdd2a CLEANUP: assorted typo fixes in the code and comments
This is 37th iteration of typo fixes
2023-11-23 16:23:14 +01:00
Willy Tarreau
42faad48f6 REGTESTS: http: add a test to validate chunked responses delivery
I've had this test here never committed over the last 2.5 years, that
works fine and I didn't notice it was not part of the tree. It makes a
server return odd-sized chunked responses with short pauses between half
of thems and verifies they're not truncated on the client. It may detect
eventually state machine breakages, so better commit it.
2023-11-18 11:16:41 +01:00
Aurelien DARRAGON
b61147fd2a MEDIUM: log/balance: merge tcp/http algo with log ones
"log-balance" directive was recently introduced to configure the
balancing algorithm to use when in a log backend. However, it is
confusing and it causes issues when used in default section.

In this patch, we take another approach: first we remove the
"log-balance" directive, and instead we rely on existing "balance"
directive to configure log load balancing in log backend.

Some algorithms such as roundrobin can be used as-is in a log backend,
and for log-only algorithms, they are implemented as "log-$name" inside
the "backend" directive.

The documentation was updated accordingly.
2023-11-18 11:16:21 +01:00
William Lallemand
3379441e26 REGTESTS: try to activate again the seamless reload test with the master CLI
Since the reload is now synchronous over the master CLI, try to reload
with it. This was a problem before with the signals because it wasn't
possible to wait for the end of the reload before sending the requests.

This activate again this test, we will see if it's more stable or we
will deactivate it again..
2023-11-17 17:11:42 +01:00
Christopher Faulet
af7db3a43c REGTESTS: http: Improve script testing abortonclose option
We now take care to properly handle the abortonclose close option if it is
set on the backend and be sure we ignore it when it is set on the frontend
(inherited from the defaults section).
2023-11-14 11:01:51 +01:00
William Lallemand
59b313832a REGTESTS: startup: -conf-OK requires -V with current VTest
Current version of VTest tests the output of "haproxy -c" instead of the
return code. Since we don't output anymore when the configuration is
valid, this broke the test. (a06f621).

This fixes the issue by adding the -V when doing a -conf-OK. But this
must fixed in VTest.
2023-11-13 14:57:26 +01:00
Alexander Stephan
ce7501de79 MINOR: connection: Send out generic, user-defined server TLVs
To follow-up the implementation of the new set-proxy-v2-tlv-fmt
keyword in the server, the connection is updated to use the previously
allocated TLVs. If no value was specified, we send out an empty TLV.
As the feature is fully working with this commit, documentation and a
test for the server and default-server are added as well.
2023-11-04 04:56:59 +01:00
Ruei-Bang Chen
7a1ec235cd MINOR: sample: Add fetcher for getting all cookie names
This new fetcher can be used to extract the list of cookie names from
Cookie request header or from Set-Cookie response header depending on
the stream direction. There is an optional argument that can be used
as the delimiter (which is assumed to be the first character of the
argument) between cookie names. The default delimiter is comma (,).

Note that we will treat the Cookie request header as a semi-colon
separated list of cookies and each Set-Cookie response header as
a single cookie and extract the cookie names accordingly.
2023-11-03 09:57:06 +01:00
William Lallemand
9496e7e888 REGTESTS: ssl: update the filters test for TLSv1.3 and sigalgs
Signature algorithms allows us to select the right certificates when
using TLSv1.3. This patch update the ssl_crt-list_filters.vtc regtest to
do more precise testing with TLSv1.3 in addition to TLSv1.2.

This allow us to test correctly bug #2300.

It could be backported to 2.8 with the previous fix for certificate
selection.
2023-10-26 19:23:04 +02:00
Jens Popp
f66b9f6018 MINOR: sample: Added support for Arrays in sample_conv_json_query in sample.c
Method now returns the content of Json Arrays, if it is specified in
Json Path as String. The start and end character is a square bracket. Any
complex object in the array is returned as Json, so that you might get Arrays
of Array or objects. Only recommended for Arrays of simple types (e.g.,
String or int) which will be returned as CSV String. Also updated
documentation and fixed issue with parenthesis and other changes from
comments.

This patch was discussed in issue #2281.

Signed-off-by: William Lallemand <wlallemand@haproxy.com>
2023-10-20 18:42:05 +02:00
Amaury Denoyelle
11ccd8d7cb REGTESTS: remove maxconn from rhttp bind line
The maxconn keyword is not used anymore for reverse HTTP bind. It has
been replaced recently by the new keyword nbconn. As it's default value
is 1, it can be safely removed from the regtest without affecting its
behavior.
2023-10-20 17:28:08 +02:00
Amaury Denoyelle
e05edf71df MINOR: cfgparse: rename "rev@" prefix to "rhttp@"
'rev@' was used to specify a bind/server used with reverse HTTP
transport. This notation was deemed not explicit enough. Rename it
'rhttp@' instead.
2023-10-20 14:44:37 +02:00
Amaury Denoyelle
9d4c7c1151 MINOR: server: convert @reverse to rev@ standard format
Remove the recently introduced '@reverse' notation for HTTP reverse
servers. Instead, reuse the 'rev@' prefix already defined for bind
lines.
2023-10-20 14:44:37 +02:00
Christopher Faulet
d1eab38131 REGTESTS: Reenable HTTP tests about splicing 2023-10-17 18:51:13 +02:00
Christopher Faulet
8b89fe3d8f MINOR: stconn: Temporarily remove kernel splicing support
mux-to-mux fast-forwarding will be added. To avoid mix with the splicing and
simplify the commits, the kernel splicing support is removed from the
stconn. CF_KERN_SPLICING flag is removed and the support is no longer tested
in process_stream().

In the stconn part, rcv_pipe() callback function is no longer called.

Reg-tests scripts testing the kernel splicing are temporarly marked as
broken.
2023-10-17 18:51:13 +02:00
Vladimir Vdovin
70d2d9aefc MINOR: support for http-response set-timeout
Added set-timeout action for http-response. Adapted reg-tests and
documentation.
2023-10-17 08:27:33 +02:00
Aurelien DARRAGON
7e70b8858f REGTEST: add a test for log-backend used as a log target
This regtest declares and uses 3 log backends, one of which has TCP syslog
servers declared in it and other ones UDP syslog servers.

Some tests aims at testing log distribution reliability by leveraging the
log-balance hash algorithm with a key extracted from the request URL, and
the dummy vtest syslog servers ensure that messages are sent to the
correct endpoint. Overall this regtest covers essential parts of the log
message distribution and log-balancing logic involved with log backends.

It also leverages the log-forward section to perform the TCP->UDP
translation required to test UDP endpoints since vtest syslog servers
work in UDP mode.

Finally, we have some tests to ensure that the server queuing/dequeuing
and failover (backup) logics work properly.
2023-10-13 10:05:06 +02:00
William Lallemand
fde517ba66 REGTESTS: wolfssl: temporarly disable some failing reg-tests
Temporarly disable the last failing reg-tests with WolfSSL in order to
be able to setup a CI.
2023-10-09 23:05:18 +02:00
William Lallemand
c24cc33c0f REGTESTS: ssl: disable ssl_dh.vtc for WolfSSL
Skip the ssl_dh reg-tests which is not working for WolfSSL.
2023-10-09 22:11:37 +02:00
William Lallemand
1105524cbe REGTESTS: ssl: update common.pem with the new pki
Update the SSL reg-test in order to use the new pki.
2023-10-09 21:58:21 +02:00
William Lallemand
8c1464098b REGTESTS: pki: add a pki for SSL tests
Add a PKI generated with cfssl in order to generated easily certificates
for the reg-tests.
2023-10-09 21:54:31 +02:00
Remi Tricot-Le Breton
a5e96425a2 MEDIUM: cache: Add "Origin" header to secondary cache key
This patch add a hash of the Origin header to the cache's secondary key.
This enables to manage store responses that have a "Vary: Origin" header
in the cache when vary is enabled.
This cannot be considered as a means to manage CORS requests though, it
only processes the Origin header and hashes the presented value without
any form of URI normalization.

This need was expressed by Philipp Hossner in GitHub issue #251.

Co-Authored-by: Philipp Hossner <philipp.hossner@posteo.de>
2023-10-05 10:53:54 +02:00
Christopher Faulet
6f6aed92b7 REGTESTS: filters: Don't set C-L header in the successful response to CONNECT
in random-forwarding.vtc script, adding "Content-Lnegth; 0" header in the
successful response to the CONNECT request is invalid but it may also lead
to wrong check on the response. "rxresp" directive don"t handle CONNECT
response. Thus "-no_obj" must be added instead, to be sure the payload won't
be retrieved or expected.
2023-10-04 15:34:18 +02:00
Vladimir Vdovin
f8b81f6eb7 MINOR: support for http-request set-timeout client
Added set-timeout for frontend side of session, so it can be used to set
custom per-client timeouts if needed. Added cur_client_timeout to fetch
client timeout samples.
2023-09-28 08:49:22 +02:00
Lokesh Jindal
915e48675a MEDIUM: sample: Enhances converter "bytes" to take variable names as arguments
Prior to this commit, converter "bytes" takes only integer values as
arguments.  After this commit, it can take variable names as inputs.
This allows us to dynamically determine the offset/length and capture
them in variables.  These variables can then be used with the converter.
Example use case: parsing a token present in a request header.
2023-09-22 08:48:51 +02:00
William Lallemand
61b6a4da6c REGTESTS: ssl: skip generate-certificates test w/ wolfSSL
WolfSSL does not seem to work correctly with the generate-certificates
features. This patch disables it temporarly.

    ssl-max-ver TLSv1.2 seems to be a problem in the reg-test and
    wolfSSL but without it it's not able to generate correctly the cert:

    ***  h1    debug|00000004:clear-lst.accept(0007)=0028 from [127.0.0.1:35956] ALPN=<none>
    ***  h1    debug|00000004:clear-lst.clireq[0028:ffffffff]: GET / HTTP/1.1
    ***  h1    debug|00000004:clear-lst.clihdr[0028:ffffffff]: x-sni: unknown-sni.com
    ***  h1    debug|00000004:clear-lst.clihdr[0028:ffffffff]: host: 127.0.0.1
    ***  h1    debug|fd[0x29] OpenSSL error[0x13d] : need the private key
    ***  h1    debug|<134>Sep 20 15:42:58 haproxy[165743]: unix:1 [20/Sep/2023:15:42:58.042] ssl-lst/1: SSL handshake failure (need the private key)
    **** dT    1.072
    ***  h1    debug|fd[0x2a] OpenSSL error[0x13d] : need the private key
    ***  h1    debug|<134>Sep 20 15:42:59 haproxy[165743]: unix:1 [20/Sep/2023:15:42:59.044] ssl-lst/1: SSL handshake failure (need the private key)
    **** dT    2.075
    ***  h1    debug|fd[0x29] OpenSSL error[0x13d] : need the private key
    ***  h1    debug|<134>Sep 20 15:43:00 haproxy[165743]: unix:1 [20/Sep/2023:15:43:00.046] ssl-lst/1: SSL handshake failure (need the private key)
    **** dT    3.079
    ***  h1    debug|fd[0x29] OpenSSL error[0x13d] : need the private key
    ***  h1    debug|<134>Sep 20 15:43:01 haproxy[165743]: unix:1 [20/Sep/2023:15:43:01.050] ssl-lst/1: SSL handshake failure (need the private key)
    **** dT    3.080
    ***  h1    debug|00000004:default_backend.clicls[0028:0023]
    ***  h1    debug|00000004:default_backend.closed[0028:0023]
    ***  h1    debug|<134>Sep 20 15:43:01 haproxy[165743]: 127.0.0.1:35956 [20/Sep/2023:15:42:58.042] clear-lst default_backend/s1 0/0/-1/-1/+3009 503 +217 - - SC-- 3/1/0/0/3 0/0 "GET / HTTP/1.1" 0/-/-/-/0 -/-/-
    **** c3    rxhdr|HTTP/1.1 503 Service Unavailable\r
    **** c3    rxhdr|content-length: 107\r
    **** c3    rxhdr|cache-control: no-cache\r
    **** c3    rxhdr|content-type: text/html\r
    **** c3    rxhdr|\r
2023-09-20 16:02:16 +02:00
William Lallemand
64a4b44a44 REGTESTS: ssl: skip OCSP test w/ WolfSSL
The OCSP test does not seem to be working correctly with wolfSSL.

i2d_OCSP_CERTID(data->ocsp_cid, NULL); always returns 0.

Skip it for now.
2023-09-20 15:23:32 +02:00
William Lallemand
15e591b6e0 MINOR: ssl: add support for 'curves' keyword on server lines
This patch implements the 'curves' keyword on server lines as well as
the 'ssl-default-server-curves' keyword in the global section.

It also add the keyword on the server line in the ssl_curves reg-test.

These keywords allow the configuration of the curves list for a server.
2023-09-07 23:29:10 +02:00
Andrew Hopkins
88988bb06c REGTESTS: ssl: skip ssl_dh test with AWS-LC
skip ssl_dh test when HAProxy is built with AWS-LC which does not support FFDH ciphersuites.
2023-09-04 18:21:01 +02:00
Alexander Stephan
f773ef721c MEDIUM: sample: Add fetch for arbitrary TLVs
Based on the new, generic allocation infrastructure, a new sample
fetch fc_pp_tlv is introduced. It is an abstraction for existing
PPv2 TLV sample fetches. It takes any valid TLV ID as argument and
returns the value as a string, similar to fc_pp_authority and
fc_pp_unique_id.
2023-08-29 15:31:28 +02:00
William Lallemand
e7d9082315 BUG/MINOR: ssl/cli: can't find ".crt" files when replacing a certificate
Bug was introduced by commit 26654 ("MINOR: ssl: add "crt" in the
cert_exts array").

When looking for a .crt directly in the cert_exts array, the
ssl_sock_load_pem_into_ckch() function will be called with a argument
which does not have its ".crt" extensions anymore.

If "ssl-load-extra-del-ext" is used this is not a problem since we try
to add the ".crt" when doing the lookup in the tree.

However when using directly a ".crt" without this option it will failed
looking for the file in the tree.

The fix removes the "crt" entry from the array since it does not seem to
be really useful without a rework of all the lookups.

Should fix issue #2265

Must be backported as far as 2.6.
2023-08-28 18:20:39 +02:00
Amaury Denoyelle
36d936dd15 REGTESTS: write a full reverse regtest
This test instantiates two haproxy instances :
* first one uses a reverse server with two bind pub and priv
* second one uses a reverse bind to initiate connection to priv endpoint

On startup, only first haproxy instance is up. A client send a request
to pub endpoint and should receive a HTTP 503 as no connection are
available on the reverse server.

Second haproxy instance is started. A delay of 3 seconds is inserted to
wait for the connection between the two LBs. Then a client retry the
request and this time should receive a HTTP 200 reusing the bootstrapped
connection.
2023-08-24 17:03:08 +02:00
Amaury Denoyelle
b57f151586 REGTESTS: provide a reverse-server test with name argument
This regtest is similar to the previous one, except the optional name
argument is specified.

An extra haproxy instance is used as a gateway for clear/TLS as vtest
does not support TLS natively.

A first request is done by specifying a name which does not match the
idle connection SNI. This must result in a HTTP 503. Then the correct
name is used which must result in a 200.
2023-08-24 17:02:37 +02:00
Amaury Denoyelle
f0bff29473 REGTESTS: provide a reverse-server test
Test support for reverse server. This can be test without the opposite
haproxy reversal support though a combination of VTC clients used to
emit HTTP/2 responses after connection.

This test ensures that first we get a 503 when connecting on a reverse
server with no idle connection. Then a dummy VTC client is connected to
act as as server. It is then expected that the same request is achieved
with a 200 this time.
2023-08-24 15:02:56 +02:00
Tim Duesterhus
c21b98a6d3 REGTESTS: Do not use REQUIRE_VERSION for HAProxy 2.5+ (3)
Introduced in:

424981cde REGTEST: add ifnone-forwardfor test
b015b3eb1 REGTEST: add RFC7239 forwarded header tests

see also:

fbbbc33df REGTESTS: Do not use REQUIRE_VERSION for HAProxy 2.5+
2023-08-15 11:29:13 +02:00
Willy Tarreau
22731762d9 BUG/MINOR: http: skip leading zeroes in content-length values
Ben Kallus also noticed that we preserve leading zeroes on content-length
values. While this is totally valid, it would be safer to at least trim
them before passing the value, because a bogus server written to parse
using "strtol(value, NULL, 0)" could inadvertently take a leading zero
as a prefix for an octal value. While there is not much that can be done
to protect such servers in general (e.g. lack of check for overflows etc),
at least it's quite cheap to make sure the transmitted value is normalized
and not taken for an octal one.

This is not really a bug, rather a missed opportunity to sanitize the
input, but is marked as a bug so that we don't forget to backport it to
stable branches.

A combined regtest was added to h1or2_to_h1c which already validates
end-to-end syntax consistency on aggregate headers.
2023-08-09 11:28:48 +02:00
Willy Tarreau
6492f1f29d BUG/MAJOR: http: reject any empty content-length header value
The content-length header parser has its dedicated function, in order
to take extreme care about invalid, unparsable, or conflicting values.
But there's a corner case in it, by which it stops comparing values
when reaching the end of the header. This has for a side effect that
an empty value or a value that ends with a comma does not deserve
further analysis, and it acts as if the header was absent.

While this is not necessarily a problem for the value ending with a
comma as it will be cause a header folding and will disappear, it is a
problem for the first isolated empty header because this one will not
be recontructed when next ones are seen, and will be passed as-is to the
backend server. A vulnerable HTTP/1 server hosted behind haproxy that
would just use this first value as "0" and ignore the valid one would
then not be protected by haproxy and could be attacked this way, taking
the payload for an extra request.

In field the risk depends on the server. Most commonly used servers
already have safe content-length parsers, but users relying on haproxy
to protect a known-vulnerable server might be at risk (and the risk of
a bug even in a reputable server should never be dismissed).

A configuration-based work-around consists in adding the following rule
in the frontend, to explicitly reject requests featuring an empty
content-length header that would have not be folded into an existing
one:

    http-request deny if { hdr_len(content-length) 0 }

The real fix consists in adjusting the parser so that it always expects a
value at the beginning of the header or after a comma. It will now reject
requests and responses having empty values anywhere in the C-L header.

This needs to be backported to all supported versions. Note that the
modification was made to functions h1_parse_cont_len_header() and
http_parse_cont_len_header(). Prior to 2.8 the latter was in
h2_parse_cont_len_header(). One day the two should be refused but the
former is also used by Lua.

The HTTP messaging reg-tests were completed to test these cases.

Thanks to Ben Kallus of Dartmouth College and Narf Industries for
reporting this! (this is in GH #2237).
2023-08-09 09:27:38 +02:00
Willy Tarreau
4d0175b54b REGTESTS: http-rules: verify that we block '#' by default for normalize-uri
Since we now block fragments by default, let's add an extra test there
to confirm that it's blocked even when stripping it.
2023-08-08 19:56:41 +02:00
Willy Tarreau
069d0e221e REGTESTS: http-rules: add accept-invalid-http-request for normalize-uri tests
We'll soon block the '#' by default so let's prepare the test to continue
to work.
2023-08-08 19:55:51 +02:00
Christopher Faulet
2efc9e0fb4 REGTESTS: Test SPLICE feature is enabled to execute script about splicing
There are 3 scripts relying on the splicing. We must take care the feature
is not explicitly disabled to execute them.
2023-08-04 15:08:06 +02:00
Christopher Faulet
28916a61c9 REGTESTS: http: Create a dedicated script to test spliced bodyless responses
Splicing is not available on all platform. Thus a dedicated script is used
to check we properly skip payload for bodyless response when splicing is
used. This way, we are still able to test the feature with the original
script on all platform.

This patch fixes an issue on the CI introduced by commit ef2b15998
("BUG/MINOR: htx/mux-h1: Properly handle bodyless responses when splicing is
used"). It must be backported with the above commit.
2023-08-04 15:02:19 +02:00
Christopher Faulet
ef2b15998c BUG/MINOR: htx/mux-h1: Properly handle bodyless responses when splicing is used
There is a mechanisme in the H1 and H2 multiplexer to skip the payload when
a response is returned to the client when it must not contain any payload
(response to a HEAD request or a 204/304 response). However, this does not
work when the splicing is used. The H2 multiplexer does not support the
splicing, so there is no issue. But with the mux-h1, when data are sent
using the kernel splicing, the mux on the server side is not aware the
client side should skip the payload. And once the data are put in a pipe,
there is no way to stop the sending.

It is a defect of the current design. This will be easier to deal with this
case when the mux-to-mux forwarding will be implemented. But for now, to fix
the issue, we should add an HTX flag on the start-line to pass the info from
the client side to the server side and be able to disable the splicing in
necessary.

The associated reg-test was improved to be sure it does not fail when the
splicing is configured.

This patch should be backported as far as 2.4..
2023-08-02 12:05:05 +02:00
Mariam John
00b7b49a46 MEDIUM: ssl: new sample fetch method to get curve name
Adds a new sample fetch method to get the curve name used in the
key agreement to enable better observability. In OpenSSLv3, the function
`SSL_get_negotiated_group` returns the NID of the curve and from the NID,
we get the curve name by passing the NID to OBJ_nid2sn. This was not
available in v1.1.1. SSL_get_curve_name(), which returns the curve name
directly was merged into OpenSSL master branch last week but will be available
only in its next release.
2023-07-17 15:45:41 +02:00
Remi Tricot-Le Breton
ca4fd73938 BUG/MINOR: cache: A 'max-age=0' cache-control directive can be overriden by a s-maxage
When a s-maxage cache-control directive is present, it overrides any
other max-age or expires value (see section 5.2.2.9 of RFC7234). So if
we have a max-age=0 alongside a strictly positive s-maxage, the response
should be cached.

This bug was raised in GitHub issue #2203.
The fix can be backported to all stable branches.
2023-07-04 22:15:00 +02:00
Aurelien DARRAGON
b973980853 REGTEST: add an extra testcase for ifnone-forwardfor
In GH #2187 it was mentioned that the ifnone-forwardfor regtest
did not cover the case where forwardfor ifnone is explicitly set in
the frontend but forwardfor option is not used in the backend.

Expected behavior in this case is that the frontend takes the precedence
because the backend did not specify the option.

Adding this missing case to prevent regressions in the future.
2023-06-20 15:33:01 +02:00
Christopher Faulet
75806f2fe8 REGTESTS: Add a script to test the kernel splicing with chunked messages
Support of the kernel splicing for chunked messages was re-introduced. This
script should validate it properly works.
2023-06-20 13:34:50 +02:00
Christopher Faulet
8fe3a33328 REGTESTS: h1_host_normalization : Add a barrier to not mix up log messages
Depending on the timing, time to time, the log messages can be mixed. A
client can start and be fully handled by HAProxy (including its log message)
before the log message of the previous client was emitted or received.  To
fix the issue, a barrier was added to be sure to eval the "expect" rule on
logs before starting the next client.
2023-06-20 12:49:39 +02:00
Christopher Faulet
1e5d1b0b0f REG-TESTS: stickiness: Delay haproxys start to properly resolv variables
Because of the commit 5cb8d7b8f ("BUG/MINOR: peers: Improve detection of
config errors in peers sections"), 2 scripts now report errors during
startup because some variables are not set and the remote peer server is
thus malformed. To perform a peer synchro between 2 haproxys in these
scripts, the startup must be delayed to properly resolve addresses.

In addidiotn, we must wait (2s) to be sure the connection between peers is
properly established. These scripts are now flagged as slow.
2023-06-05 08:24:34 +02:00
Willy Tarreau
ae0f8be011 MINOR: stats: protect against future stats fields omissions
As seen in commits 33a4461fa ("BUG/MINOR: stats: Fix Lua's `get_stats`
function") and a46b142e8 ("BUG/MINOR: Missing stat_field_names (since
f21d17bb)") it seems frequent to omit to update stats_fields[] when
adding a new ST_F_xxx entry. This breaks Lua's get_stats() and shows
a "(null)" in the header of "show stat", but that one is not detectable
to the naked eye anymore.

Let's add a reminder above the enum declaration about this, and a small
reg tests checking for the absence of "(null)". It was verified to fail
before the last patch above.
2023-06-02 08:39:53 +02:00
Christopher Faulet
535dd920df MINOR: compression: Improve the way Vary header is added
When a message is compressed, A "Vary" header is added with
"accept-encoding" value. However, a new header is always added, regardless
there is already a Vary header or not. In addition, if there is already a
Vary header, there is no check on values to be sure "accept-encoding" value
is not already there. So it is possible to have it twice.

To improve this part, we now test Vary header values and "accept-encoding"
is only added if it was not found. In addition, "accept-encoding" value is
appended to the last Vary header found, if any. Otherwise, a new header is
added.
2023-05-25 11:25:31 +02:00
Christopher Faulet
292619fc90 REGTESTS: log: Reduce again response inspect-delay for last_rule.vtc
It was previously reduced from 10s to 1s but it remains too high, espeically
for the CI. It may be drastically reduced to 100ms. Idea is to just be sure
we will wait for the response before evaluating the TCP rules.
2023-05-17 11:12:25 +02:00
Christopher Faulet
4ab27a2403 REGTESTS: log: Reduce response inspect-delay for last_rule.vtc
Because of the previous fix, log/last_rule.vtc script is failing. The
inspect-delay is no longer shorten when the end of the message is
reached. Thus WAIT_END acl is trully respected. 10s is too high and hit the
Vtext timeout, making the script fails.
2023-05-17 09:21:01 +02:00
William Lallemand
6e0c39d7ac BUILD: ssl: ssl_c_r_dn fetches uses functiosn only available since 1.1.1
Fix the openssl build with older openssl version by disabling the new
ssl_c_r_dn fetch.

This also disable the ssl_client_samples.vtc file for OpenSSL version
older than 1.1.1
2023-05-15 12:07:52 +02:00
Abhijeet Rastogi
df97f472fa MINOR: ssl: add new sample ssl_c_r_dn
This patch addresses #1514, adds the ability to fetch DN of the root
ca that was in the chain when client certificate was verified during SSL
handshake.
2023-05-15 10:48:05 +02:00
Aurelien DARRAGON
7963fb5669 REGTESTS: use lua mailer script for mailers tests
Since mailers/healthcheckmail.vtc already requires lua to emulate the
SMTP server for the test, force it to use lua mailers example script
to send email-alerts so we don't rely anymore on legacy tcpcheck
mailers implementation.

This is done by simply loading examples/mailers.lua (as a symlink) from
haproxy config file.
2023-05-05 16:28:32 +02:00