Commit Graph

453 Commits

Author SHA1 Message Date
Alex
51c8ad45ce MINOR: sample: converter: Add json_query converter
With the json_query can a JSON value be extacted from a header
or body of the request and saved to a variable.

This converter makes it possible to handle some JSON workload
to route requests to different backends.
2021-04-15 17:07:03 +02:00
Moemen MHEDHBI
92f7d43c5d MINOR: sample: add ub64dec and ub64enc converters
ub64dec and ub64enc are the base64url equivalent of b64dec and base64
converters. base64url encoding is the "URL and Filename Safe Alphabet"
variant of base64 encoding. It is also used in in JWT (JSON Web Token)
standard.
RFC1421 mention in base64.c file is deprecated so it was replaced with
RFC4648 to which existing converters, base64/b64dec, still apply.

Example:
  HAProxy:
    http-request return content-type text/plain lf-string %[req.hdr(Authorization),word(2,.),ub64dec]
  Client:
    Token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjoiZm9vIiwia2V5IjoiY2hhZTZBaFhhaTZlIn0.5VsVj7mdxVvo1wP5c0dVHnr-S_khnIdFkThqvwukmdg
    $ curl -H "Authorization: Bearer ${TOKEN}" http://haproxy.local
    {"user":"foo","key":"chae6AhXai6e"}
2021-04-13 17:28:13 +02:00
William Lallemand
a1e832b867 REGTESTS: ssl: mark set_ssl_cert_bundle.vtc as broken
set_ssl_cert_bundle.vtc requires at least OpenSSL 1.1.0 and we don't
have any way to check this when launching the reg-tests suite.

Mark the reg-test as broken since it will fails on old versions of
openSSL and libreSSL.
2021-04-02 17:13:55 +02:00
William Lallemand
35201833aa REGTESTS: ssl: "set ssl cert" and multi-certificates bundle
This test loads a configuration which uses multi-certificates bundle and
tries to change them over the CLI.

Could be backported as far as 2.2, however the 2.2 version must be
adapted to commit the bundle and not each certificate individually.
2021-04-02 15:47:17 +02:00
Christopher Faulet
d57d9fe3d8 REGTESTS: Add script to tests the wait-for-body HTTP action
This script tests the wait-for-body HTTP action, on the request side and the
response side.
2021-04-01 16:45:34 +02:00
Christopher Faulet
e1b9e1bb1e REGTESTS: Add script to tests TCP to HTTP upgrades
This script tests various TCP>HTTP upgrade scenarios and also performs some
checks for invalid/forbidden upgrades.
2021-04-01 13:51:22 +02:00
Willy Tarreau
b8bd1ee893 MEDIUM: cli: add a new experimental "set var" command
set var <name> <expression>
  Allows to set or overwrite the process-wide variable 'name' with the result
  of expression <expression>. Only process-wide variables may be used, so the
  name must begin with 'proc.' otherwise no variable will be set. The
  <expression> may only involve "internal" sample fetch keywords and converters
  even though the most likely useful ones will be str('something') or int().
  Note that the command line parser doesn't know about quotes, so any space in
  the expression must be preceeded by a backslash. This command requires levels
  "operator" or "admin". This command is only supported on a CLI connection
  running in experimental mode (see "experimental-mode on").

Just like for "set-var" in the global section, the command uses a temporary
dummy proxy to create a temporary "set-var(name)" rule to assign the value.

The reg test was updated to verify that an updated global variable is properly
reflected in subsequent HTTP responses.
2021-03-26 16:57:43 +01:00
Willy Tarreau
c35eb38f1d MINOR: vars/cli: add a "get var" CLI command to retrieve global variables
Process-wide variables can now be displayed from the CLI using "get var"
followed by the variable name. They must all start with "proc." otherwise
they will not be found. The output is very similar to the one of the
debug converter, with a type and value being reported for the embedded
sample.

This command is limited to clients with the level "operator" or higher,
since it can possibly expose traffic-related data.
2021-03-26 16:52:13 +01:00
Willy Tarreau
9fdf342ca9 REGTESTS: add a basic reg-test for some "set-var" commands
This reg-test tests "set-var" in the global section, with some overlapping
variables and using a few samples and converters, then at the TCP and HTTP
levels using proc/sess/req variables.
2021-03-26 16:34:53 +01:00
Remi Tricot-Le Breton
bc2c386992 BUG/MINOR: ssl: Prevent removal of crt-list line if the instance is a default one
If the first active line of a crt-list file is also the first mentioned
certificate of a frontend that does not have the strict-sni option
enabled, then its certificate will be used as the default one. We then
do not want this instance to be removable since it would make a frontend
lose its default certificate.
Considering that a crt-list file can be used by multiple frontends, and
that its first mentioned certificate can be used as default certificate
for only a subset of those frontends, we do not want the line to be
removable for some frontends and not the others. So if any of the ckch
instances corresponding to a crt-list line is a default instance, the
removal of the crt-list line will be forbidden.

It can be backported as far as 2.2.
2021-03-26 13:06:39 +01:00
Remi Tricot-Le Breton
8218aed90e BUG/MINOR: ssl: Fix update of default certificate
The default SSL_CTX used by a specific frontend is the one of the first
ckch instance created for this frontend. If this instance has SNIs, then
the SSL context is linked to the instance through the list of SNIs
contained in it. If the instance does not have any SNIs though, then the
SSL_CTX is only referenced by the bind_conf structure and the instance
itself has no link to it.
When trying to update a certificate used by the default instance through
a cli command, a new version of the default instance was rebuilt but the
default SSL context referenced in the bind_conf structure would not be
changed, resulting in a buggy behavior in which depending on the SNI
used by the client, he could either use the new version of the updated
certificate or the original one.

This patch adds a reference to the default SSL context in the default
ckch instances so that it can be hot swapped during a certificate
update.

This should fix GitHub issue #1143.

It can be backported as far as 2.2.
2021-03-26 13:06:29 +01:00
Remi Tricot-Le Breton
fb00f31af4 BUG/MINOR: ssl: Prevent disk access when using "add ssl crt-list"
If an unknown CA file was first mentioned in an "add ssl crt-list" CLI
command, it would result in a call to X509_STORE_load_locations which
performs a disk access which is forbidden during runtime. The same would
happen if a "ca-verify-file" or "crl-file" was specified. This was due
to the fact that the crt-list file parsing and the crt-list related CLI
commands parsing use the same functions.
The patch simply adds a new parameter to all the ssl_bind parsing
functions so that they know if the call is made during init or by the
CLI, and the ssl_store_load_locations function can then reject any new
cafile_entry creation coming from a CLI call.

It can be backported as far as 2.2.
2021-03-23 19:29:46 +01:00
Amaury Denoyelle
79e6d2a7ad REGTESTS: wait for proper return of enable server in cli add server test
Add an empty expect statement after the 'enable server' cli command.
This ensures that the command has been properly handled by haproxy and
its processing is over.

It should fix the unstable behavior of the test which causes reports of
503 even after the server has been enabled.

This should fix the github issue #1188.
2021-03-22 11:52:48 +01:00
Amaury Denoyelle
318c008c0d REGTESTS: remove unneeded experimental-mode in cli add server test
The experimental-mode cli command is only required for the moment for
'add server'. Remove it for the 'enable server' command.
2021-03-22 11:52:06 +01:00
Ilya Shipitsin
19d14710e9 REGTESTS: revert workaround for a crash with recent libressl on http-reuse sni
LibreSSL-3.2.5 has fixed "use-after-free" in tls session resumption,
let us enable session resumption back
2021-03-20 09:32:57 +01:00
Amaury Denoyelle
304672320e MINOR: server: support keyword proto in 'add server' cli
Allow to specify the mux proto for a dynamic server. It must be
compatible with the backend mode to be accepted. The reg-tests has been
extended for this error case.
2021-03-18 16:22:10 +01:00
Amaury Denoyelle
65273abf41 REGTESTS: implement test for 'add server' cli
Write a regtest for the cli command 'add server'. This test will execute
some invalid commands and validates the reported error. A client will
then try to connect to a dynamic server just created and activated.
2021-03-18 16:22:10 +01:00
Daniel Corbett
befef70e23 BUG/MINOR: sample: Rename SenderComID/TargetComID to SenderCompID/TargetCompID
The recently introduced Financial Information eXchange (FIX)
converters have some hard coded tags based on the specification that
were misspelled. Specifically, SenderComID and TargetComID should
be SenderCompID and TargetCompID according to the specification [1][2].

This patch updates all references, which includes the converters
themselves, the regression test, and the documentation.

[1] https://fiximate.fixtrading.org/en/FIX.5.0SP2_EP264/tag49.html
[2] https://fiximate.fixtrading.org/en/FIX.5.0SP2_EP264/tag56.html
2021-03-10 10:44:20 +01:00
Amaury Denoyelle
a47dfab673 REGTESTS: test http-reuse if no server target
Add two new regtests which check the behavior of http-reuse when the
connection target is not a server. More specifically check the dispatch
and transparent backend. In these cases, the behavior should be similar
to http-reuse never mode.
2021-03-03 11:40:25 +01:00
Christopher Faulet
3820ff8d2f REGTESTS: Add script to test except param for fowardedfor/originalto options
IPv6 support was added for these options. This script test different IPv4
and IPv6 combinations.
2021-02-26 13:53:26 +01:00
William Dauchy
98ad35f826 REGTESTS: contrib/prometheus-exporter: test well known labels
as we previously briefly broke labels handling, test them to make sure
we don't introduce regressions in the future.
see also commit 040b1195f7 ("BUG/MINOR:
contrib/prometheus-exporter: Restart labels dump at the right pos") for
reference

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-02-19 18:03:59 +01:00
William Dauchy
b45674433b REGTESTS: contrib/prometheus-exporter: test NaN values
In order to make sure we detect when we change default behaviour for
some metrics, test the NaN value when it is expected.

Those metrics were listed since our last rework as their default value
changed, unless the appropriate config is set.

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-02-19 18:03:59 +01:00
Amaury Denoyelle
d8ea188058 REGTESTS: workaround for a crash with recent libressl on http-reuse sni
Disable the ssl-reuse for the sni test on http_reuse_conn_hash vtc. This
seems to be the origin of a crash with libressl environment from 3.2.2
up to 3.3.1 included.

For now, it is not determined if the root cause is in haproxy or
libressl.

Please look for the github issue #1115 for all the details.
2021-02-19 16:47:20 +01:00
Amaury Denoyelle
4cce7088d1 REGTESTS: reorder reuse conn proxy protocol test
Try to fix http_reuse_conn_hash proxy protocol for both single and
multi-thread environment. Schedule a new set of requests to be sure that
takeover will be functional even with pool-low-count set to 2.
2021-02-18 16:07:17 +01:00
Amaury Denoyelle
ffcd290ddd REGTESTS: fix http_reuse_conn_hash proxy test
Define pool-low-conn to 2 to force to have at least 2 idle connections
on the server side for the purpose of the test.
2021-02-17 09:11:52 +01:00
William Dauchy
e3f7bd5ae9 MEDIUM: contrib/prometheus-exporter: add listen stats
this was a missing piece for a while now even though it was planned. This
patch adds listen stats.
Nothing in particular but we make use of the status helper previously
added.  `promex_st_metrics` diff also looks scary, but I had to realign
all lines.

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-02-15 14:14:59 +01:00
William Dauchy
d1a7b85a40 MEDIUM: server: support {check,agent}_addr, agent_port in server state
logical followup from cli commands addition, so that the state server
file stays compatible with the changes made at runtime; use previously
added helper to load server attributes.

also alloc a specific chunk to avoid mixing with other called functions
using it

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-02-12 16:04:52 +01:00
Amaury Denoyelle
7f583be822 REGTESTS: fix sni used in http_reuse_conn_hash for libressl 3.3.0
libressl 3.3.0 is stricter on the sni field and fails if it contains
illegal characters such as the underscore. Replace sni field with proper
name to pass the test on the CI environment.
2021-02-12 15:27:33 +01:00
Amaury Denoyelle
49aa5e9eca REGTESTS: deactivate a failed test on CI in http_reuse_conn_hash
The last request with proxy protocol fails on the CI, reporting
first_http_request as 1. Currently this is not reproducible on my
local machine.
2021-02-12 14:18:39 +01:00
Amaury Denoyelle
8abbd2dcc3 REGTESTS: adjust http_reuse_conn_hash requirements
This test is only valid for 2.4+ as the tested feature was introduced in
this version. Besides, it can only run with ssl support to test sni
reuse.
2021-02-12 14:18:16 +01:00
Amaury Denoyelle
7ef06c8253 MINOR: reg-tests: test http-reuse with proxy protocol
Complete the http-reuse test with the proxy protocol usage. A server
connection can be reused if the proxy protocol fields are the same.
2021-02-12 12:54:04 +01:00
Amaury Denoyelle
1921d20fff MINOR: connection: use proxy protocol as parameter for srv conn hash
Use the proxy protocol frame if proxy protocol is activated on the
server line. Do not add anymore these connections in the private list.
If some requests are made with the same proxy fields, they can reuse
the idle connection.

The reg-tests proxy_protocol_send_unique_id must be adapted has it
relied on the side effect behavior that every requests from a same
connection reused a private server connection. Now, a new connection is
created as expected if the proxy protocol fields differ.
2021-02-12 12:54:04 +01:00
Amaury Denoyelle
f679d9bbf3 MINOR: reg-test: test http-reuse with specific dst addr
Add a reg-test for connection reuse with a specific destination address.
It is based on the set-dst-port http rule.
2021-02-12 12:54:04 +01:00
Amaury Denoyelle
e676239922 MINOR: reg-tests: test http-reuse with sni
Add a reg-test for connection reuse with the sni parameter. Ensure that
only connections that use the same sni are reused.
2021-02-12 12:48:11 +01:00
Christopher Faulet
475fab397e REGTESTS: Add a script to test payload skipping for bodyless HTTP responses
This script is used to be sure unexpected payload in bodyless HTTP responses
is properly ignored on outgoing side, both by H1 and H2 muxes.
2021-02-10 16:25:42 +01:00
Willy Tarreau
826f3ab5e6 MINOR: stick-tables/counters: add http_fail_cnt and http_fail_rate data types
Historically we've been counting lots of client-triggered events in stick
tables to help detect misbehaving ones, but we've been missing the same on
the server side, and there's been repeated requests for being able to count
the server errors per URL in order to precisely monitor the quality of
service or even to avoid routing requests to certain dead services, which
is also called "circuit breaking" nowadays.

This commit introduces http_fail_cnt and http_fail_rate, which work like
http_err_cnt and http_err_rate in that they respectively count events and
their frequency, but they only consider server-side issues such as network
errors, unparsable and truncated responses, and 5xx status codes other
than 501 and 505 (since these ones are usually triggered by the client).
Note that retryable errors are purposely not accounted for, so that only
what the client really sees is considered.

With this it becomes very simple to put some protective measures in place
to perform a redirect or return an excuse page when the error rate goes
beyond a certain threshold for a given URL, and give more chances to the
server to recover from this condition. Typically it could look like this
to bypass a URL causing more than 10 requests per second:

  stick-table type string len 80 size 4k expire 1m store http_fail_rate(1m)
  http-request track-sc0 base       # track host+path, ignore query string
  http-request return status 503 content-type text/html \
      lf-file excuse.html if { sc0_http_fail_rate gt 10 }

A more advanced mechanism using gpt0 could even implement high/low rates
to disable/enable the service.

Reg-test converteers_ref_cnt_never_dec.vtc was updated to test it.
2021-02-10 12:27:01 +01:00
William Dauchy
69164228b8 MEDIUM: contrib/prometheus-exporter: export base stick table stats
I saw some people falling back to unix socket to collect some data they
could not find in prometheus exporter. One of them is base info from
stick tables (used/size).
I do not plan to extend it more for now; keys are quite a mess to
handle.

This should resolve github issue #1008.

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-02-08 10:49:08 +01:00
Willy Tarreau
2da742933d REGTESTS: unbreak http-check-send.vtc
As noticed by Christopher, I messed up the version fix in commit
cb4ed02ef ("REGTESTS: mark http-check-send.vtc as 2.4-only"), as while
looking up the commit introducing the change I accidently reverted it.
Let's reinsert the contents of the file prior to that fix, except the
version, of course.
2021-02-05 10:13:15 +01:00
Willy Tarreau
23296f92f4 REGTESTS: mark sample_fetches/hashes.vtc as 2.4-only
Commit 9eea56009 ("REGTESTS: add tests for the xxh3 converter") introduced
the xxh3 to the tests thus made it incompatible with 2.3 and older, let's
upgrade the version requirement.
2021-02-04 18:07:59 +01:00
Willy Tarreau
cb4ed02ef0 REGTESTS: mark http-check-send.vtc as 2.4-only
Since commit 39ff8c519 ("REGTESTS: complete http-check test"), it breaks
on pre-2.4, let's update the required version.
2021-02-04 18:06:13 +01:00
William Dauchy
fe03e7d045 MEDIUM: server: adding support for check_port in server state
We can currently change the check-port using the cli command `set server
check-port` but there is a consistency issue when using server state.
This patch aims to fix this problem but will be also a good preparation
work to get rid of checkport flag, so we are able to know when checkport
was set by config.

I am fully aware this is not making github #953 moving forward, I
however think this might be acceptable while waiting for a proper
solution and resolve consistency problem faced with port settings.

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-02-04 10:46:52 +01:00
William Lallemand
e0de0a6b32 MINOR: ssl/cli: flush the server session cache upon 'commit ssl cert'
Flush the SSL session cache when updating a certificate which is used on a
server line. This prevent connections to be established with a cached
session which was using the previous SSL_CTX.

This patch also replace the ha_barrier with a thread_isolate() since there
are more operations to do. The reg-test was also updated to remove the
'no-ssl-reuse' keyword which is now uneeded.
2021-02-03 18:51:01 +01:00
William Lallemand
8695ce0bae BUG/MEDIUM: ssl/cli: abort ssl cert is freeing the old store
The "abort ssl cert" command is buggy and removes the current ckch store,
and instances, leading to SNI removal. It must only removes the new one.

This patch also adds a check in set_ssl_cert.vtc and
set_ssl_server_cert.vtc.

Must be backported as far as 2.2.
2021-02-01 17:58:21 +01:00
William Lallemand
ff97edac3e REGTESTS: set_ssl_server_cert: cleanup the SSL caching option
Replace the tune.ssl.cachesize 0 and the no-tls-tickets by a
no-ssl-reuse option on the server line.
2021-02-01 14:57:31 +01:00
William Lallemand
a870a9cfdb REGTESTS: set_ssl_server_cert.vtc: remove SSL caching and set as working
In a previous commit this test was disabled because I though the
feature was broken, but in fact this is the test which is broken.

Indeed the connection between the server and the client was not
renegociated and was using the SSL cache or a ticket. To be work
correctly these 2 features must be disabled or a new connection must be
established after the ticket timeout, which is too long for a regtest.

Also a "nbthread 1" was added as it was easier to reproduce the problem
with it.
2021-02-01 14:50:17 +01:00
William Lallemand
e814321287 REGTESTS: set_ssl_server_cert.vtc: set as broken
It looks like this test is broken with a low nbthread value (1 for
example). Disable this test in the CI until the problem is solved.
2021-01-28 18:08:36 +01:00
Christopher Faulet
d2dcd8a88f REGTEST: Don't use the websocket to validate http-check
Now, some conformance tests are performed when an HTTP connection is
upgraded to websocket. This make the http-check-send.vtc script failed for
the backend <be6_ws>. Because the purpose of this health-check is to pass a
"Connection: Upgrade" header on an http-check send rule, we may use a dummy
protocal instead.
2021-01-28 16:37:14 +01:00
Christopher Faulet
85a813676f REGTESTS: Fix required versions for several scripts
The following scripts require HAProxy 2.4 :

 * cache/caching_rules.vtc
 * cache/post_on_entry.vtc
 * cache/vary.vtc
 * checks/1be_40srv_odd_health_checks.vtc
 * checks/40be_2srv_odd_health_checks.vtc
 * checks/4be_1srv_health_checks.vtc
 * converter/fix.vtc
 * converter/mqtt.vtc
 * http-messaging/protocol_upgrade.vtc
 * http-messaging/websocket.vtc
 * http-set-timeout/set_timeout.vtc
 * log/log_uri.vtc

However it may change is features are backported.
2021-01-28 16:37:14 +01:00
Amaury Denoyelle
8969c32d7e MINOR: vtc: add websocket test
Test the conformance of websocket rfc6455 in haproxy. In particular, if
a missing key is detected on a h1 message, haproxy must close the
connection.

Note that the case h2 client/h1 srv is not tested as I did not find a
way to calculate the key on the server side.
2021-01-28 16:37:14 +01:00
Amaury Denoyelle
bda34302d7 MINOR: vtc: add test for h1/h2 protocol upgrade translation
This test send HTTP/1.1 Get+Upgrade or HTTP/2 Extended Connect through a
haproxy instance.
2021-01-28 16:37:14 +01:00
Christopher Faulet
d8b44f0d9f REGTESTS: Don't run http_msg_full_on_eom script on the 2.4 anymore
The EOM block will be removed on the 2.4, thus this script will be broken on
this version. Now it is skipped for this version. It remains valid for 2.3
and 2.2.
2021-01-28 16:37:14 +01:00
William Lallemand
7b79424c05 REGTESTS: set_ssl_server_cert.vtc: check the sha1 from the server
Check the sha1 from the server side with the sample ssl_c_sha1 sample
fetch in order to evict a possible problem with "show/set ssl cert".
2021-01-28 16:00:22 +01:00
William Lallemand
7e69637ac5 REGTESTS: set_ssl_server_cert.vtc: check the Sha1 Fingerprint
Check the sha1 fingerprint once the certificate was changed with "show
ssl cert". This way the test is more reliable.
2021-01-28 15:11:59 +01:00
William Lallemand
8788c6ff13 REGTESTS: set_ssl_server_cert.vtc: remove the abort command
Temporarily remove the abort command as it seems to cause problems when
trying to do a "show ssl cert" after it.
2021-01-28 15:04:03 +01:00
Remi Tricot-Le Breton
f3eedfe195 MEDIUM: ssl: Enable backend certificate hot update
When trying to update a backend certificate, we should find a
server-side ckch instance thanks to which we can rebuild a new ssl
context and a new ckch instance that replace the previous ones in the
server structure. This way any new ssl session will be built out of the
new ssl context and the newly updated certificate.

This resolves a subpart of GitHub issue #427 (the certificate part)
2021-01-26 15:19:36 +01:00
Amaury Denoyelle
d86ae4bd03 MINOR: reg-tests: add http-reuse test
Add a serie of 4 tests for the various http-reuse modes : never, safe,
aggressive and always.
2021-01-26 14:48:39 +01:00
Christopher Faulet
6071c2d12d BUG/MEDIUM: filters/htx: Fix data forwarding when payload length is unknown
It is only a problem on the response path because the request payload length
it always known. But when a filter is registered to analyze the response
payload, the filtering may hang if the server closes just after the headers.

The root cause of the bug comes from an attempt to allow the filters to not
immediately forward the headers if necessary. A filter may choose to hold
the headers by not forwarding any bytes of the payload. For a message with
no payload but a known payload length, there is always a EOM block to
forward. Thus holding the EOM block for bodyless messages is a good way to
also hold the headers. However, messages with an unknown payload length,
there is no EOM block finishing the message, but only a SHUTR flag on the
channel to mark the end of the stream. If there is no payload when it
happens, there is no payload at all to forward. In the filters API, it is
wrongly detected as a condition to not forward the headers.

Because it is not the most used feature and not the obvious one, this patch
introduces another way to hold the message headers at the begining of the
forwarding. A filter flag is added to explicitly says the headers should be
hold. A filter may choose to set the STRM_FLT_FL_HOLD_HTTP_HDRS flag and not
forwad anything to hold the headers. This flag is removed at each call, thus
it must always be explicitly set by filters. This flag is only evaluated if
no byte has ever been forwarded because the headers are forwarded with the
first byte of the payload.

reg-tests/filters/random-forwarding.vtc reg-test is updated to also test
responses with unknown payload length (with and without payload).

This patch must be backported as far as 2.0.
2021-01-26 09:53:52 +01:00
Remi Tricot-Le Breton
6ca89162dc MINOR: cache: Do not store responses with an unknown encoding
If a server varies on the accept-encoding header and it sends a response
with an encoding we do not know (see parse_encoding_value function), we
will not store it. This will prevent unexpected errors caused by
cache collisions that could happen in accept_encoding_hash_cmp.
2021-01-15 22:33:05 +01:00
William Dauchy
1704efee89 MINOR: contrib/prometheus-exporter: avoid connection close header
it does not seem to have a reason to close connections after each
request; reflect that in tests by doing all requests within the same
client.

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-01-13 15:11:38 +01:00
William Dauchy
76603f2552 MINOR: reg-tests: add base prometheus test
Add a base test to start with something, even though this is not
necessarily complete.
Also make use of the recent REQUIRE_SERVICE option to exclude it from
test list of it was not build with prometheus included.

note: I thought it was possible to send multiple requests within the
same client, but I'm getting "HTTP header is incomplete" from the second
request

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-01-11 14:16:06 +01:00
William Dauchy
5417e898ff CLEANUP: sample: remove uneeded check in json validation
- check functions are never called with a NULL args list, it is always
  an array, so first check can be removed
- the expression parser guarantees that we can't have anything else,
  because we mentioned json converter takes a mandatory string argument.
  Thus test on `ARGT_STR` can be removed as well
- also add breaking line between enum and function declaration

In order to validate it, add a simple json test testing very simple
cases but can be improved in the future:

- default json converter without args
- json converter failing on error (utf8)
- json converter with error being removed (utf8s)

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-01-10 07:39:58 +01:00
William Dauchy
888b0ae8cf MINOR: converter: adding support for url_enc
add base support for url encode following RFC3986, supporting `query`
type only.

- add test checking url_enc/url_dec/url_enc
- update documentation
- leave the door open for future changes

this should resolve github issue #941

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-01-06 23:43:04 +01:00
Ilya Shipitsin
1e9a66603f CLEANUP: assorted typo fixes in the code and comments
This is 14th iteration of typo fixes
2021-01-06 16:26:50 +01:00
Willy Tarreau
fae9a4cd2c REGTESTS: add unresolvable servers to srvkey-addr
This ensures that the previous fix remains operational, as without it
an unresolvable server causes a crash when it's been indexed.
2021-01-06 09:20:22 +01:00
Thayne McCombs
23cc52d34b REGTESTS: add test for stickiness using "srvkey addr"
This tests the new "srvkey addr" stick-table statement, which sticks based
on the server's address.
2020-12-31 10:08:54 +01:00
Tim Duesterhus
dc38bc4a1a BUG/MEDIUM: cache: Fix hash collision in accept-encoding handling for Vary
This patch fixes GitHub Issue #988. Commit ce9e7b2521
was not sufficient, because it fell back to a hash comparison if the bitmap
of known encodings was not acceptable instead of directly returning the the
cached response is not compatible.

This patch also extends the reg-test to test the hash collision that was
mentioned in #988.

Vary handling is 2.4, no backport needed.
2020-12-31 09:39:08 +01:00
Remi Tricot-Le Breton
e6cc5b5974 MINOR: cache: Replace the "process-vary" option's expected values
Replace the <0/1> expected values of the process-vary option by a more
usual <on/off> pair.
2020-12-24 17:18:00 +01:00
Remi Tricot-Le Breton
14df1e1526 REGTESTS: cache: Add a specific test for the accept-encoding normalizer
Those tests check the explicit support of a subset of encodings added to
the accept-encoding normalizer.
2020-12-24 17:18:00 +01:00
Remi Tricot-Le Breton
b054b6dec6 REGTESTS: cache: Simplify vary.vtc file
This test used chunked responses but it is not needed. All the chunked
responses are replaced by more simple 'bodylen' directives.
2020-12-24 17:18:00 +01:00
Remi Tricot-Le Breton
e4421dec7e BUG/MINOR: cache: Manage multiple headers in accept-encoding normalization
The accept-encoding part of the secondary key (vary) was only built out
of the first occurrence of the header. So if a client had two
accept-encoding headers, gzip and br for instance, the key would have
been built out of the gzip string. So another client that only managed
gzip would have been sent the cached resource, even if it was a br resource.
The http_find_header function is now called directly by the normalizers
so that they can manage multiple headers if needed.
A request that has more than 16 encodings will be considered as an
illegitimate request and its response will not be stored.

This fixes GitHub issue #987.

It does not need any backport.
2020-12-24 17:18:00 +01:00
Dragan Dosen
9eea56009d REGTESTS: add tests for the xxh3 converter 2020-12-23 06:39:21 +01:00
Amaury Denoyelle
39ff8c519c REGTESTS: complete http-check test
Add a new check for a pseudo-websocket handshake, specifying the
Connection header to verify if it is properly handled by http-check send
directive. Also check that default http/1.1 checks have the header
Connection: close.
2020-12-22 14:22:44 +01:00
Ilya Shipitsin
f38a01884a CLEANUP: assorted typo fixes in the code and comments
This is 13n iteration of typo fixes
2020-12-21 11:24:48 +01:00
Amaury Denoyelle
78ea5c99a9 REGTESTS: add regtest for http-request set-timeout
This test compares the timeout value for requests using the sample
fetches in accordance with the application of set-timeout rules.
2020-12-11 12:01:07 +01:00
Christopher Faulet
6ade861305 REGTESTS: Fix proxy_protocol_tlv_validation
Remaining data after the PROXYv2 header now trigger a parsing error in the
H1 multiplexer Thus the required version for this test is now set to 2.4.
2020-12-04 14:41:49 +01:00
Remi Tricot-Le Breton
72cffaf440 MEDIUM: cache: Remove cache entry in case of POST on the same resource
In case of successful unsafe method on a stored resource, the cached entry
must be invalidated (see RFC7234#4.4).
A "non-error response" is one with a 2xx (Successful) or 3xx (Redirection)
status code.
This implies that the primary hash must now be calculated on requests
that have an unsafe method (POST or PUT for instance) so that we can
disable the corresponding entries when we process the response.
2020-12-04 10:21:56 +01:00
Remi Tricot-Le Breton
795e1412b0 MINOR: cache: Do not store stale entry
When a response has an Age header (filled in by another cache on the
message's path) that is greater than its defined maximum age (extracted
either from cache-control directives or an expires header), it is
already stale and should not be cached.
2020-12-04 10:21:56 +01:00
Willy Tarreau
cc794b91f6 REGTESTS: add a test for the threaded Lua code
This is simply txn_get_priv.vtc with the loading made using
lua-load-per-thread, allowing all threads to run independently.
There's nothing global nor thread-specific in this test, which makes
an excellent example of something that should work out of the box.
Four concurrent clients are initialized with 4 loops each so as to
give a little chance to various threads to run concurrently.
2020-12-02 21:53:16 +01:00
Remi Tricot-Le Breton
8bb72aa82f MINOR: cache: Improve accept_encoding_normalizer
Turn the "Accept-Encoding" value to lower case before processing it.
Calculate the CRC on every token instead of a sorted concatenation of
them all (in order to avoir copying them) then XOR all the CRCs into a
single hash (while ignoring duplicates).
2020-12-02 16:32:54 +01:00
Maciej Zdeb
fcdfd857b3 MINOR: log: Logging HTTP path only with %HPO
This patch adds a new logging variable '%HPO' for logging HTTP path only
(without query string) from relative or absolute URI.

For example:
log-format "hpo=%HPO hp=%HP hu=%HU hq=%HQ"

GET /r/1 HTTP/1.1
=>
hpo=/r/1 hp=/r/1 hu=/r/1 hq=

GET /r/2?q=2 HTTP/1.1
=>
hpo=/r/2 hp=/r/2 hu=/r/2?q=2 hq=?q=2

GET http://host/r/3 HTTP/1.1
=>
hpo=/r/3 hp=http://host/r/3 hu=http://host/r/3 hq=

GET http://host/r/4?q=4 HTTP/1.1
=>
hpo=/r/4 hp=http://host/r/4 hu=http://host/r/4?q=4 hq=?q=4
2020-12-01 09:32:44 +01:00
Christopher Faulet
b381a505c1 BUG/MAJOR: tcpcheck: Allocate input and output buffers from the buffer pool
Historically, the input and output buffers of a check are allocated by hand
during the startup, with a specific size (not necessarily the same than
other buffers). But since the recent refactoring of the checks to rely
exclusively on the tcp-checks and to use the underlying mux layer, this part
is totally buggy. Indeed, because these buffers are now passed to a mux,
they maybe be swapped if a zero-copy is possible. In fact, for now it is
only possible in h2_rcv_buf(). Thus the bug concretely only exists if a h2
health-check is performed. But, it is a latent bug for other muxes.

Another problem is the size of these buffers. because it may differ for the
other buffer size, it might be source of bugs.

Finally, for configurations with hundreds of thousands of servers, having 2
buffers per check always allocated may be an issue.

To fix the bug, we now allocate these buffers when required using the buffer
pool. Thus not-running checks don't waste memory and muxes may swap them if
possible. The only drawback is the check buffers have now always the same
size than buffers used by the streams. This deprecates indirectly the
"tune.chksize" global option.

In addition, the http-check regtest have been update to perform some h2
health-checks.

Many thanks to @VigneshSP94 for its help on this bug.

This patch should solve the issue #936. It relies on the commit "MINOR:
tcpcheck: Don't handle anymore in-progress send rules in tcpcheck_main".
Both must be backport as far as 2.2.

bla
2020-11-27 10:29:41 +01:00
Remi Tricot-Le Breton
754b2428d3 MINOR: cache: Add a process-vary option that can enable/disable Vary processing
The cache section's process-vary option takes a 0 or 1 value to disable
or enable the vary processing.
When disabled, a response containing such a header will never be cached.
When enabled, we will calculate a preliminary hash for a subset of request
headers on all the incoming requests (which might come with a cpu cost) which
will be used to build a secondary key for a given request (see RFC 7234#4.1).
The default value is 0 (disabled).
2020-11-24 16:52:57 +01:00
Remi Tricot-Le Breton
1785f3dd96 MEDIUM: cache: Add the Vary header support
Calculate a preliminary secondary key for every request we see so that
we can have a real secondary key if the response is cacheable and
contains a manageable Vary header.
The cache's ebtree is now allowed to have multiple entries with the same
primary key. Two of those entries will be distinguished thanks to
secondary keys stored in the cache_entry (based on hashes of a subset of
their headers).
When looking for an entry in the cache (cache_use), we still use the
primary key (built the same way as before), but in case of match, we
also need to check if the entry has a vary signature. If it has one, we
need to perform an extra check based on the newly built secondary key.
We will only be able to forge a response out of the cache if both the
primary and secondary keys match with one of our entries. Otherwise the
request will be forwarder to the server.
2020-11-24 16:52:57 +01:00
Maciej Zdeb
ebdd4c55da MINOR: http_act: Add -m flag for del-header name matching method
This patch adds -m flag which allows to specify header name
matching method when deleting headers from http request/response.
Currently beg, end, sub, str and reg are supported.

This is related to GitHub issue #909
2020-11-21 15:54:30 +01:00
Willy Tarreau
4137889911 REGTESTS: mark proxy_protocol_random_fail as broken
As indicated in issue #907 it very frequently fails on FreeBSD, and
looking at it, it's broken in multiple ways. It relies on log ordering
between two layers, the first one being allowed to support h2. Given
that on FreeBSD it usually ends up in timeouts, it's very likely that
for some reason one frontend logs before the other one or that curl
uses h2 instead of h1 there, and that the log instance waits forever
for its lines.

Usually it works fine when run locally though, so let's not remove it
and mark it as broken instead so that it can still be used when relevant.
2020-11-21 15:33:03 +01:00
William Lallemand
a1ef754a6d REGTEST: server/cli_set_ssl.vtc requires OpenSSL
Add OpenSSL as a requirement for this test.
2020-11-18 17:41:28 +01:00
William Dauchy
f63704488e MEDIUM: cli/ssl: configure ssl on server at runtime
in the context of a progressive backend migration, we want to be able to
activate SSL on outgoing connections to the server at runtime without
reloading.
This patch adds a `set server ssl` command; in order to allow that:

- add `srv_use_ssl` to `show servers state` command for compatibility,
  also update associated parsing
- when using default-server ssl setting, and `no-ssl` on server line,
  init SSL ctx without activating it
- when triggering ssl API, de/activate SSL connections as requested
- clean ongoing connections as it is done for addr/port changes, without
  checking prior server state

example config:

backend be_foo
  default-server ssl
  server srv0 127.0.0.1:6011 weight 1 no-ssl

show servers state:

  5 be_foo 1 srv0 127.0.0.1 2 0 1 1 15 1 0 4 0 0 0 0 - 6011 - -1

where srv0 can switch to ssl later during the runtime:

  set server be_foo/srv0 ssl on

  5 be_foo 1 srv0 127.0.0.1 2 0 1 1 15 1 0 4 0 0 0 0 - 6011 - 1

Also update existing tests and create a new one.

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2020-11-18 17:22:28 +01:00
William Dauchy
a2a46ee572 REGTESTS: converter: add url_dec test
while looking at `url_dec` implementation I realised there was not yet a
simple test to avoid future regressions.
This one is testing simple case, including the "+" behaviour depending
on the argument passed to `url_dec`

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2020-11-17 11:53:25 +01:00
Willy Tarreau
1bc898398b REGTESTS: mark the abns test as broken again
There is apparently a race in this test that would require relying on
haproxy's output to make it reliably work, but currently vtest doesn't
have this option. Let's mark it broken again to avoid polluting the CI.

This is discussed in github issue #950.
2020-11-17 11:45:10 +01:00
Christopher Faulet
4f33760fe0 REGTESTS: Add a script to test the random forwarding with several filters
Three filters are used. The compression filter is enclose by two trace filters,
both with the random forwarding enabled. An HTTP test is performed but also a
TCP test using an HTTP tunnel.
2020-11-17 11:34:36 +01:00
Tim Duesterhus
afe36e457f REGTESTS: Add sample_fetches/cook.vtc
Add a reg-test verifying the fix in dea7c209f8.

Some parts of the configuration used in the were taken from the initial bug
report from Maciej.

Should be backported together with dea7c209f8
(all stable versions).

Co-authored-by: Maciej Zdeb <maciej@zdeb.pl>
2020-11-13 19:46:15 +01:00
Christopher Faulet
c300747dec REGTEST: make ssl_client_samples and ssl_server_samples require to 2.2
Some missing sample fetches was backported to 2.2 making these tests compatible
with the 2.2.
2020-11-13 17:12:30 +01:00
Remi Tricot-Le Breton
cc9bf2e5fe MEDIUM: cache: Change caching conditions
Do not cache responses that do not have an explicit expiration time
(s-maxage or max-age Cache-Control directives or Expires header) or a
validator (ETag or Last-Modified headers) anymore, as suggested in
RFC 7234#3.
The TX_FLAG_IGNORE flag is used instead of the TX_FLAG_CACHEABLE so as
not to change the behavior of the checkcache option.
2020-11-12 11:22:05 +01:00
William Lallemand
8f04e1849d REGTEST: ssl: mark reg-tests/ssl/ssl_crt-list_filters.vtc as broken
This regtest requires a version of OpenSSL which supports the
ClientHello callback which is only the case of recents SSL libraries
(openssl 1.1.1).

This was reported in issue #944.
2020-11-10 22:40:24 +01:00
William Lallemand
50c03aac04 BUG/MEDIUM: ssl/crt-list: correctly insert crt-list line if crt already loaded
In issue #940, it was reported that the crt-list does not work correctly
anymore. Indeed when inserting a crt-list line which use a certificate
previously seen in the crt-list, this one won't be inserted in the SNI
list and will be silently ignored.

This bug was introduced by commit  47da821 "MEDIUM: ssl: emulates the
multi-cert bundles in the crtlist".

This patch also includes a reg-test which tests this issue.

This bugfix must be backported in 2.3.
2020-11-06 16:39:39 +01:00
William Lallemand
3ff9591ea2 REGTEST: ssl: test wildcard and multi-type + exclusions
This test checks that the bug #818 and #810 are fixed.

It test if there is no inconsistency with multiple certificate types and
that the exclusion of the certificate is correctly working with a negative
filter.
2020-11-06 14:59:36 +01:00
Christopher Faulet
32186472cb REGTEST: converter: Add a regtest for MQTT converters
This new script tests mqtt_is_valid() and mqtt_get_field_value() converters used
to validate and extract information from a MQTT (Message Queuing Telemetry
Transport) message.
2020-11-05 19:27:08 +01:00
Christopher Faulet
7983b8687e REGTEST: converter: Add a regtest for fix converters
This new script tests fix_is_valid() and fix_tag_value() converters used to
validate and extract information from a FIX (Financial Information eXchange)
message.
2020-11-05 19:26:40 +01:00
Willy Tarreau
501c99588e REGTESTS: mark abns_socket as working now
William noticed that the real issue in the abns test was that it was
failing to rebind some listeners, issues which were addressed in the
previous commits (in some cases, the master would even accept traffic
on the worker's socket which was not properly disabled, causing some
of the strange error messages).

The other issue documented in commit 2ea15a080 was the lack of
reliability when the test was run in parallel. This is caused by the
abns socket which uses a hard-coded address, making all tests in
parallel to step onto each others' toes. Since vtest cannot provide
abns sockets, we're instead concatenating the number of the listening
port that vtest allocated for another frontend to the abns path, which
guarantees to make them unique in the system.

The test works fine in all cases now, even with 100 in parallel.
2020-11-04 14:42:15 +01:00
Remi Tricot-Le Breton
a6476114ec MINOR: cache: Add Expires header value parsing
When no Cache-Control max-age or s-maxage information is present in a
cached response, we need to parse the Expires header value (RFC 7234#5.3).
An invalid Expires date value or a date earlier than the reception date
will make the cache_entry stale upon creation.
For now, the Cache-Control and Expires headers are parsed after the
insertion of the response in the cache so even if the parsing of the
Expires results in an already stale entry, the entry will exist in the
cache.
2020-10-30 11:08:38 +01:00
Remi Tricot-Le Breton
bf97121f1c MINOR: cache: Create res.cache_hit and res.cache_name sample fetches
Res.cache_hit sample fetch returns a boolean which is true when the HTTP
response was built out of a cache. The cache's name is returned by the
res.cache_name sample_fetch.

This resolves GitHub issue #900.
2020-10-27 18:25:43 +01:00
Remi Tricot-Le Breton
53161d81b8 MINOR: cache: Process the If-Modified-Since header in conditional requests
If a client sends a conditional request containing an If-Modified-Since
header (and no If-None-Match header), we try to compare the date with
the one stored in the cache entry (coming either from a Last-Modified
head, or a Date header, or corresponding to the first response's
reception time). If the request's date is earlier than the stored one,
we send a "304 Not Modified" response back. Otherwise, the stored is sent
(through a 200 OK response).

This resolves GitHub issue #821.
2020-10-27 18:10:25 +01:00
William Lallemand
1ac17682e5 REGTEST: ssl: test "set ssl cert" with separate key / crt
This reg-test tests the "set ssl cert" command the same way the
set_ssl_cert.vtc does, but with separate key/crt files and with the
ssl-load-extra-del-ext.

It introduces new key/.crt files that contains the same pair as the
existing .pem.
2020-10-23 18:41:08 +02:00
Remi Tricot-Le Breton
5bbdc81cf1 REGTEST: cache: Add if-none-match test case
Test that if-none-match header is properly taken into account and that
when the conditions are fulfilled, a "304 Not Modified" response can be
sent to the client.

Co-authored-by: Tim Duesterhus <tim@bastelstu.be>
2020-10-22 16:10:20 +02:00
Willy Tarreau
43ba3cf2b5 MEDIUM: proxy: remove start_proxies()
Its sole remaining purpose was to display "proxy foo started", which
has little benefit and pollutes output for those with plenty of proxies.
Let's remove it now.

The VTCs were updated to reflect this, because many of them had explicit
counts of dropped lines to match this message.

This is tagged as MEDIUM because some users may be surprized by the
loss of this quite old message.
2020-10-09 11:27:30 +02:00
Willy Tarreau
2ea15a0804 REGTESTS: mark abns_socket as broken
This test is inherently racy. It regularly pops up on the CI, and I've
spent one hour chasing a bug that apparently doesn't exist, just because
I'm running it 10 times in a row and it reports from 4 to 8 failures
when built at -O2 and generally even more at -O0. The logs are very
confusing, often reporting that it failed with status 0, with nothing
else wrong. I suspect it might sometimes be the shell command that fails
if it executes faster than haproxy finishes to start up, which would
also explain the relation with the optimization level. E.g:

>  Testing with haproxy version: 2.2.0
>  #    top  TEST reg-tests/seamless-reload/abns_socket.vtc FAILED (3.006) exit=2
>  #    top  TEST reg-tests/seamless-reload/abns_socket.vtc FAILED (3.006) exit=2
>  #    top  TEST reg-tests/seamless-reload/abns_socket.vtc FAILED (3.009) exit=2
>  #    top  TEST reg-tests/seamless-reload/abns_socket.vtc FAILED (3.008) exit=2
>  #    top  TEST reg-tests/seamless-reload/abns_socket.vtc FAILED (3.007) exit=2
>  #    top  TEST reg-tests/seamless-reload/abns_socket.vtc FAILED (3.007) exit=2
>  6 tests failed, 0 tests skipped, 4 tests passed

Some of the failures include this, suggesting that some barriers could
help:
  ---- h1   haproxy h1 PID file check failed:
       Could not read PID file '/tmp/haregtests-2020-10-09_11-19-40.kgsDB4/vtc.30539.04dbea7f/h1/pid

Since it has been causing false positives and consumed way more
troubleshooting time than it saved, let's mark it as broken so that it
doesn't waste more time. We can bring it back when someone manages to
figure what the problem is.
2020-10-09 11:26:42 +02:00
Christopher Faulet
f7177271f3 BUG/MINOR: tcpcheck: Set socks4 and send-proxy flags before the connect call
Since the health-check refactoring in the 2.2, the checks through a socks4 proxy
are broken. To fix this bug, CO_FL_SOCKS4 flag must be set on the connection
before calling the connect() callback function because this flags is checked to
use the right destination address. The same is done for the CO_FL_SEND_PROXY
flag for a consistency purpose.

A reg-test has been added to test the "check-via-socks4" directive.

This patch must be backported to 2.2.
2020-10-02 17:14:34 +02:00
Willy Tarreau
73cc5457cb REGTEST: make map_regm_with_backref require 1.7
map_regm was only introduced in 1.7, I don't know why the require field
was set to 1.6, probably tha the test evolved and didn't start with
map_regm.
2020-09-29 11:00:51 +02:00
Willy Tarreau
f8d46de806 REGTEST: make abns_socket.vtc require 1.8
It's not as much an abns test as it is a seamless reload test, which
appeared in 1.8 due to "expose-fd listeners".
2020-09-29 10:58:44 +02:00
Willy Tarreau
55da2b97be REGTEST: make agent-check.vtc require 1.8
This one looks at the output of "show server state", which change slightly
in 1.8 (shows DNS fields).
2020-09-29 10:58:04 +02:00
Willy Tarreau
0f08c37f0d REGTEST: the iif converter test requires 2.3
This one was recently added in 2.3.
2020-09-29 10:52:32 +02:00
Willy Tarreau
989fe79a4b REGTEST: make ssl_client_samples and ssl_server_samples requiret to 2.3
These ones added new sample fetches that are only available in 2.3 and
which fail on older versions.
2020-09-29 10:52:30 +02:00
Willy Tarreau
2aa3dd2b43 REGTEST: fix host part in balance-uri-path-only.vtc
We used to set it to ${h1_px_addr} but it randomly fails on certain
hosts (FreeBSD and OSX) where the address is surprisingly set to "::1"
while the Host field contains 127.0.0.1 (hence two different address
families). While this is likely a minor issue in vtest, we don't need
to depend on this and can easily hard-code 127.0.0.1 which is already
used in other tests.
2020-09-29 10:52:27 +02:00
Willy Tarreau
19490907b3 REGTESTS: add a few load balancing tests
This adds "balance-rr" to test round robin, "balance-uri" to test the
default balance-uri method, and "balance-uri-path-only" which mixes H1
and H2 through "balance uri path-only" and verifies that they reach
the same server.

Note that for the latter, "proto h2" explicitly had to be placed on
the listening socket otherwise it would timeout. This may indicate an
issue in the H1->H2 upgrade depending how the H2 preface is sent maybe.
2020-09-23 09:02:13 +02:00
Tim Duesterhus
3943e4fc3e MINOR: sample: Add iif(<true>,<false>) converter
iif() takes a boolean as input and returns one of the two argument
strings depending on whether the boolean is true.

This converter most likely is most useful to return the proper scheme
depending on the value returned by the `ssl_fc` fetch, e.g. for use within
the `x-forwarded-proto` request header.

However it can also be useful for use within a template that is sent to
the client using `http-request return` with a `lf-file`. It allows the
administrator to implement a simple condition, without needing to prefill
variables within the regular configuration using `http-request
set-var(req.foo)`.
2020-09-11 16:59:27 +02:00
Christopher Faulet
1c22048d66 REGTEST: Add a test for request path manipulations, with and without the QS
This new script tests set-path/replace-path and set-pathq/replace-pathq
rules. It also tests path and pathq sample fetches.

This patch should be backported to 2.2 if corresponding keywords are also
backported.
2020-09-04 11:41:47 +02:00
Willy Tarreau
db522b542e REGTEST: increase some short timeouts to make tests more reliable
A few regtests continue to regularly fail in highly loaded VMs because
they have very short timeouts. Actually the goal of running with short
timeouts was to make sure we do not uselessly wait during tests designed
to trigger them, but these timeouts here are never supposed to fire at
all, so they don't need to be kept in the 15-20ms range. They do not
pose any issue on any regular machine, but VMs are often suffering from
huge time jumps and cannot always produce responses in that short of a
time.

Just like with commit ce6fc25b1 ("REGTEST: increase timeouts on the
seamless-reload test"), let's raise these short timeouts to 1 second.
A few other ones remain set to 150-200ms and do not seem to cause any
issue. Some are actually expected to trigger so let's not touch them
for now.
2020-09-02 07:30:34 +02:00
Willy Tarreau
a1af38b74e REGTEST: remove stray leading spaces in converteers_ref_cnt_never_dec.vtc
Since commit f92afb732 ("MEDIUM: cfgparse: Emit hard error on truncated lines")
we now produce parsing errors on truncated lines, in an effort to clean
up dangerous or broken config files. And it turns out that one of our own
regression tests was suffering from this, as diagnosed by William and Tim.
The cause is the leading spaces in front of "} -start" that vtest makes
part of the output file, so the file finishes with a partial line made
of spaces.
2020-08-19 11:20:28 +02:00
William Dauchy
a598b500b4 MINOR: ssl: add ssl_{c,s}_chain_der fetch methods
Following work from Arjen and Mathilde, it adds ssl_{c,s}_chain_der
methods; it returns DER encoded certs from SSL_get_peer_cert_chain

Also update existing vtc tests to add random intermediate certificates

When getting the result through this header:
  http-response add-header x-ssl-chain-der %[ssl_c_chain_der,hex]
One can parse it with any lib accepting ASN.1 DER data, such as in go:
  bin, err := encoding/hex.DecodeString(cert)
  certs_parsed, err := x509.ParseCertificates(bin)

Cc: Arjen Nienhuis <arjen@zorgdoc.nl>
Signed-off-by: Mathilde Gilles <m.gilles@criteo.com>
Signed-off-by: William Dauchy <w.dauchy@criteo.com>
2020-08-07 15:38:40 +02:00
Christopher Faulet
9467f18d32 BUG/MINOR: http-rules: Fix ACLs parsing for http deny rules
The parsing of http deny rules with no argument or only the deny_status argument
is buggy if followed by an ACLs expression (starting with "if" or "unless"
keyword). Instead of using the proxy errorfiles, a dummy error is used. To fix
the bug, the parsing function must also check for "if" or "unless" keyword in
such cases.

This patch should fix the issue #720. No backport is needed.
2020-06-30 09:32:03 +02:00
William Lallemand
86bc9539f2 REGTEST: ssl: add some ssl_c_* sample fetches test
Test the following ssl sample fetches:

ssl_c_der, ssl_c_sha1,hex, ssl_c_notafter, ssl_c_notbefore,
ssl_c_sig_alg, ssl_c_i_dn, ssl_c_s_dn, ssl_c_serial,hex, ssl_c_key_alg,
ssl_c_version

This reg-test could be used as far as haproxy 1.6.
2020-06-26 12:11:39 +02:00
William Lallemand
6e56b2cd8a REGTEST: ssl: tests the ssl_f_* sample fetches
Test the following ssl sample fetches:

ssl_f_der, ssl_f_sha1,hex, ssl_f_notafter, ssl_f_notbefore,
ssl_f_sig_alg, ssl_f_i_dn, ssl_f_s_dn, ssl_f_serial,hex, ssl_f_key_alg,
ssl_f_version

This reg-test could be used as far as haproxy 1.5.
2020-06-26 11:33:59 +02:00
Ilya Shipitsin
47d17182f4 CLEANUP: assorted typo fixes in the code and comments
This is 10th iteration of typo fixes
2020-06-26 11:27:28 +02:00
William Lallemand
bfa3e81a7b MINOR: ssl: add the ssl_s_* sample fetches for server side certificate
This commit adds some sample fetches that were lacking on the server
side:

ssl_s_key_alg, ssl_s_notafter, ssl_s_notbefore, ssl_s_sig_alg,
ssl_s_i_dn, ssl_s_s_dn, ssl_s_serial, ssl_s_sha1, ssl_s_der,
ssl_s_version
2020-06-25 21:22:24 +02:00
William Lallemand
99cc21851f MINOR: cli/ssl: handle trailing slashes in crt-list commands
Trailing slashes were not handled in crt-list commands on CLI which can
be useful when you use the commands with a directory.

Strip the slashes before looking for the crtlist in the tree.
2020-06-25 15:40:10 +02:00
Christopher Faulet
60837d340c REGTEST: Add a simple script to tests errorfile directives in proxy sections
This script is compatible with all HAProxy versions. It does not depend on 2.2
features.
2020-06-22 10:35:38 +02:00
William Lallemand
5bb21b1d29 REGTEST: http-rules: test spaces in ACLs with master CLI
Do the tests for spaces on the CLI with the master CLI.

Could be backported as far as 2.0 once the required patches are applied.
2020-06-19 14:32:55 +02:00
William Lallemand
398c5f39ee REGTEST: http-rules: test spaces in ACLs
This reg-test tests the spaces in an ACL file, it tries to add new
entries with spaces from the CLI

This reg-test could backported in all stable branches if the fix for
spaces on the CLI was backported.
2020-06-19 14:32:55 +02:00
Tim Duesterhus
f38175cf6e MINOR: sample: Add secure_memcmp converter
secure_memcmp compares two binary strings in constant time. It's only
available when haproxy is compiled with USE_OPENSSL.
2020-06-09 22:04:13 +02:00
Tim Duesterhus
9947f77a59 REGTEST: Add connection/proxy_protocol_send_unique_id_alpn
This reg-test checks that sending unique IDs via PPv2 works for servers
with the `alpn` option specified (issue #640). As a side effect it also
checks that PPv2 works with ALPN (issue #651).

It has been verified that the test fails without the following commits
applied and succeeds with them applied.

   1f9a4ecea BUG/MEDIUM: backend: set the connection owner to the session when using alpn.
   083fd42d5 BUG/MEDIUM: connection: Ignore PP2 unique ID for stream-less connections
   eb9ba3cb2 BUG/MINOR: connection: Always get the stream when available to send PP2 line

Without the first two commits HAProxy crashes during execution of the
test. Without the last commit the test will fail, because no unique ID
is received.
2020-05-27 13:29:56 +02:00
Christopher Faulet
80f59e5db1 REGTESTS: Require the version 2.2 to execute lua/set_var
This script depends on LUA changes introduced in HAProxy 2.2 and not backported.
2020-05-26 13:36:30 +02:00
Christopher Faulet
185a64309c REGTESTS: Add missing OPENSSL to REQUIRE_OPTIONS for compression/lua_validation
The test uses the `ssl` keyword, add `OPENSSL` as a requirement.

Should be backported to all branches with that test.
2020-05-26 13:36:30 +02:00
Tim Duesterhus
4e172c93f9 MEDIUM: lua: Add ifexist parameter to set_var
As discussed in GitHub issue #624 Lua scripts should not use
variables that are never going to be read, because the memory
for variable names is never going to be freed.

Add an optional `ifexist` parameter to the `set_var` function
that allows a Lua developer to set variables that are going to
be ignored if the variable name was not used elsewhere before.

Usually this mean that there is no `var()` sample fetch for the
variable in question within the configuration.
2020-05-25 08:12:35 +02:00
Tim Duesterhus
84ebc136a1 MINOR: lua: Make set_var() and unset_var() return success
This patch makes `set_var()` and `unset_var()` return a boolean indicating
success.
2020-05-25 08:12:31 +02:00
Tim Duesterhus
de5021a133 REGTESTS: Add missing OPENSSL to REQUIRE_OPTIONS for lua/txn_get_priv
The test uses the `ssl` keyword, add `OPENSSL` as a requirement.

Should be backported to all branches with that test.
2020-05-25 08:12:10 +02:00
Christopher Faulet
ed48657e02 REGTESTS: checks: Fix tls_health_checks when IPv6 addresses are used
In tls_health_checks.vtc, when IPv6 addresses are used, A config error is
reported because of the "addr" server parameter. Because there is no specified
port, the IPv6 address must be enclosed into brackets to be properly parsed. It
also works with IPv4 addresses. But instead, a dummy port is added to the addr
parameter. This way, we also check the port parameter, when specified, is used
in priority over the port found in the addr parameter.

This patch should fix the issue #646.
2020-05-25 08:06:30 +02:00
Christopher Faulet
66163ec616 BUG/MINOR: checks: Respect check-ssl param when a port or an addr is specified
When a check port or a check address is specified, the check transport layer is
ignored. So it is impossible to do a SSL check in this case. This bug was
introduced by the commit 8892e5d30 ("BUG/MEDIUM: server/checks: Init server
check during config validity check").

This patch should fix the issue #643. It must be backported to all branches
where the above commit was backported.
2020-05-20 22:55:05 +02:00
Christopher Faulet
3b967c1210 MINOR: http-htx/proxy: Add http-error directive using http return syntax
The http-error directive can now be used instead of errorfile to define an error
message in a proxy section (including default sections). This directive uses the
same syntax that http return rules. The only real difference is the limitation
on status code that may be specified. Only status codes supported by errorfile
directives are supported for this new directive. Parsing of errorfile directive
remains independent from http-error parsing. But functionally, it may be
expressed in terms of http-errors :

  errorfile <status> <file> ==> http-errror status <status> errorfile <file>
2020-05-20 18:27:14 +02:00
Christopher Faulet
5cb513abeb MEDIUM: http-rules: Rely on http reply for http deny/tarpit rules
"http-request deny", "http-request tarpit" and "http-response deny" rules now
use the same syntax than http return rules and internally rely on the http
replies. The behaviour is not the same when no argument is specified (or only
the status code). For http replies, a dummy response is produced, with no
payload. For old deny/tarpit rules, the proxy's error messages are used. Thus,
to be compatible with existing configuration, the "default-errorfiles" parameter
is implied. For instance :

  http-request deny deny_status 404

is now an alias of

  http-request deny status 404 default-errorfiles
2020-05-20 18:27:13 +02:00
William Lallemand
8177ad9895 MINOR: ssl: split config and runtime variable for ssl-{min,max}-ver
In the CLI command 'show ssl crt-list', the ssl-min-ver and the
ssl-min-max arguments were always displayed because the dumped versions
were the actual version computed and used by haproxy, instead of the
version found in the configuration.

To fix the problem, this patch separates the variables to have one with
the configured version, and one with the actual version used. The dump
only shows the configured version.
2020-05-20 16:49:02 +02:00
Christopher Faulet
62f79fe68a MEDIUM: checks: Make post-41 the default mode for mysql checks
MySQL 4.1 is old enough to be the default mode for mysql checks. So now, once a
username is defined, post-41 mode is automatically used. To do mysql checks on
previous MySQL version, the argument "pre-41" must be used.

Note, it is a compatibility breakage for everyone using an antique and
unsupported MySQL version.
2020-05-18 18:32:09 +02:00
Patrick Gansterer
8e36651ed3 MINOR: sample: Add digest and hmac converters
Make the digest and HMAC function of OpenSSL accessible to the user via
converters. They can be used to sign and validate content.

Reviewed-by: Tim Duesterhus <tim@bastelstu.be>
2020-05-12 10:08:11 +02:00
Willy Tarreau
a0be8595c6 REGTESTS: make the http-check-send test require version 2.2
It causes failures when passing regtests on older releases.
2020-05-07 18:42:22 +02:00
Christopher Faulet
574e7bd7f3 MINOR: checks: Support log-format string to set the body for HTTP send rules
For http-check send rules, it is now possible to use a log-format string to set
the request's body. the keyword "body-lf" should be used instead of "body". If the
string eval fails, no body is added.
2020-05-06 18:04:05 +02:00
Christopher Faulet
7c95f5f22b MINOR: checks: Support log-format string to set the URI for HTTP send rules
For http-check send rules, it is now possible to use a log-format string to set
the request URI. the keyword "uri-lf" should be used instead of "uri". If the
string eval fails, we fall back on the default uri "/".
2020-05-06 18:04:05 +02:00
Christopher Faulet
8d38f0affd REGTEST: checks: Adapt SSL error message reported when connection is rejected
Depending on the SSL library version, the reported error may differ when the
connection is rejected during the handshake. An empty handshke may be detected
or just an generic handshake error. So tcp-check-ssl.vtc has been adapted to
support both error messages.
2020-05-06 12:45:11 +02:00
Christopher Faulet
b5594265d2 MINOR: checks: Simplify matching on HTTP headers in HTTP expect rules
Extra parameters on http-check expect rules, for the header matching method, to
use log-format string or to match full header line have been removed. There is
now separate matching methods to match a full header line or to match each
comma-separated values. "http-check expect fhdr" must be used in the first case,
and "http-check expect hdr" in the second one. In addition, to match log-format
header name or value, "-lf" suffix must be added to "name" or "value"
keyword. For intance:

   http-check expect hdr name "set-cookie" value-lf -m beg "sessid=%[var(check.cookie)]"

Thanks to this changes, each parameter may only be interpreted in one way.
2020-05-06 12:42:36 +02:00
Christopher Faulet
e596d184be MEDIUM: checks: Remove dedicated sample fetches and use response ones instead
All sample fetches in the scope "check." have been removed. Response sample
fetches must be used instead. It avoids keyword duplication. So, for instance,
res.hdr() must be now used instead of check.hdr().

To do so, following sample fetches have been added on the response :

  * res.body, res.body_len and res.body_size
  * res.hdrs and res.hdrs_bin

Sample feches dealing with the response's body are only useful in the health
checks context. When called from a stream context, there is no warranty on the
body presence. There is no option to wait the response's body.
2020-05-06 12:37:43 +02:00
Christopher Faulet
3970819a55 MEDIUM: checks: Support matching on headers for http-check expect rules
It is now possible to add http-check expect rules matching HTTP header names and
values. Here is the format of these rules:

  http-check expect header name [ -m <meth> ] <name> [log-format] \
                           [ value [ -m <meth> ] <value> [log-format] [full] ]

the name pattern (name ...) is mandatory but the value pattern (value ...) is
optionnal. If not specified, only the header presence is verified. <meth> is the
matching method, applied on the header name or the header value. Supported
matching methods are:

  * "str" (exact match)
  * "beg" (prefix match)
  * "end" (suffix match)
  * "sub" (substring match)
  * "reg" (regex match)

If not specified, exact matching method is used. If the "log-format" option is
used, the pattern (<name> or <value>) is evaluated as a log-format string. This
option cannot be used with the regex matching method. Finally, by default, the
header value is considered as comma-separated list. Each part may be tested. The
"full" option may be used to test the full header line. Note that matchings are
case insensitive on the header names.
2020-05-05 11:19:27 +02:00
Christopher Faulet
cb436f0c29 REGTEST: checks: Fix sync condition for agent-check
agent-check.vtc script fails time to time because the 2nd cli command is sent to
early. Waiting for the connection close in the s1 server should be enough to be
sure the server state is updated.
2020-05-05 11:07:00 +02:00
William Lallemand
a911548715 REGTEST: ssl: improve the "set ssl cert" test
Improve the test by removing the curl command and using the same proxy
chaining technique as in commit 3ed722f ("REGTEST: ssl: remove curl from
the "add ssl crt-list" test").

A 3rd request was added which must fail, to ensure that the SNI was
effectively removed from HAProxy.

This patch also adds timeouts in the default section, logs on stderr and
fix some indentation issues.
2020-04-30 10:28:34 +02:00
William Lallemand
3ed722f03c REGTEST: ssl: remove curl from the "add ssl crt-list" test
Using curl for SSL tests can be a problem if it wasn't compiled with the
right SSL library and if it didn't share any cipher with HAProxy. To
have more robust tests we now use HAProxy as an SSL client, so we are
sure that the client and the server share the same SSL requirements.

This patch also adds timeouts in the default section, logs on stderr and
fix some indentation issues.
2020-04-30 09:53:48 +02:00
Christopher Faulet
8aa825a356 REGTEST: http-rules: Require PCRE or PCRE2 option to run map_redirect script
Only PCRE was specified as required option to execute this script. But PCRE2 is
an valid alternative.
2020-04-29 14:32:28 +02:00
William Lallemand
2e0dbb7412 REGTEST: ssl: test the client certificate authentication
This reg-test tests the client auth feature of HAProxy for both the
backend and frontend section with a CRL list.

This reg-test uses 2 chained listeners because vtest does not handle the
SSL. Test the frontend client auth and the backend side at the same
time.

It sends 3 requests: one with a correct certificate, one with an expired
one and one which was revoked. The client then checks if we received the
right one with the right error.

Certificates, CA and CRL are expiring in 2050 so it should be fine for
the CI.

This test could be backported as far as HAProxy 1.6
2020-04-28 22:04:13 +02:00
Christopher Faulet
907701bcda MINOR: checks: Use ver keyword to specify the HTTP version for http checks
'ver' keyword is already used by sample fetches while 'vsn' is not used anywhere
else. So better to use 'ver' too for http-check send rules.
2020-04-28 09:37:01 +02:00
Christopher Faulet
f97f846413 REGTEST: Add a script to validate agent checks 2020-04-27 10:46:28 +02:00
Christopher Faulet
88d939c831 Revert "MEDIUM: checks: capture groups in expect regexes"
This reverts commit 1979943c30ef285ed04f07ecf829514de971d9b2.

Captures in comment was only used when a tcp-check expect based on a negative
regex matching failed to eventually report what was captured while it was not
expected. It is a bit far-fetched to be useable IMHO. on-error and on-success
log-format strings are far more usable. For now there is few check sample
fetches (in fact only one...). But it could be really powerful to report info in
logs.
2020-04-27 10:46:28 +02:00
Christopher Faulet
aec7f76af8 REGTEST: Add scripts to test based tcp-check health-checks
These scripts have been added to validate the health-checks based on tcp-check
rules (http, redis, MySQL...).
2020-04-27 10:46:28 +02:00
Christopher Faulet
a67ed43b71 REGTEST: Fix reg-tests about health-checks to adapt them to recent changes 2020-04-27 10:46:28 +02:00
Christopher Faulet
617780f99b REGTEST: Adapt regtests about checks to recent changes 2020-04-27 09:39:38 +02:00
Gaetan Rivet
9dcb09fc98 MEDIUM: checks: capture groups in expect regexes
Parse back-references in comments of tcp-check expect rules.  If references are
made, capture groups in the match and replace references to it within the
comment when logging the error. Both text and binary regex can caputre groups
and reference them in the expect rule comment.

[Cf: I slightly updated the patch. exp_replace() function is used instead of a
custom one. And if the trash buffer is too small to contain the comment during
the substitution, the comment is ignored.]
2020-04-27 09:39:37 +02:00
Gaetan Rivet
1afd826ae4 MINOR: checks: add min-recv tcp-check expect option
Some expect rules cannot be satisfied due to inherent ambiguity towards
the received data: in the absence of match, the current behavior is to
be forced to wait either the end of the connection or a buffer full,
whichever comes first. Only then does the matching diagnostic is
considered  conclusive. For instance :

    tcp-check connect
    tcp-check expect !rstring "^error"
    tcp-check expect string "valid"

This check will only succeed if the connection is closed by the server before
the check timeout. Otherwise the first expect rule will wait for more data until
"^error" regex matches or the check expires.

Allow the user to explicitly define an amount of data that will be
considered enough to determine the value of the check.

This allows succeeding on negative rstring rules, as previously
in valid condition no match happened, and the matching was repeated
until the end of the connection. This could timeout the check
while no error was happening.

[Cf: I slighly updated the patch. The parameter was renamed and the value is a
signed integer to support -1 as default value to ignore the parameter.]
2020-04-27 09:39:37 +02:00
Christopher Faulet
8acb1284bc MINOR: checks: Add a way to send custom headers and payload during http chekcs
The 'http-check send' directive have been added to add headers and optionnaly a
payload to the request sent during HTTP healthchecks. The request line may be
customized by the "option httpchk" directive but there was not official way to
add extra headers. An old trick consisted to hide these headers at the end of
the version string, on the "option httpchk" line. And it was impossible to add
an extra payload with an "http-check expect" directive because of the
"Connection: close" header appended to the request (See issue #16 for details).

So to make things official and fully support payload additions, the "http-check
send" directive have been added :

    option httpchk POST /status HTTP/1.1

    http-check send hdr Content-Type "application/json;charset=UTF-8" \
        hdr X-test-1 value1 hdr X-test-2 value2 \
        body "{id: 1, field: \"value\"}"

When a payload is defined, the Content-Length header is automatically added. So
chunk-encoded requests are not supported yet. For now, there is no special
validity checks on the extra headers.

This patch is inspired by Kiran Gavali's work. It should fix the issue #16 and
as far as possible, it may be backported, at least as far as 1.8.
2020-04-27 09:39:37 +02:00
William Lallemand
4fd9433fb0 REGTEST: ssl/cli: add other cases of 'add ssl crt-list'
Add some certificates in the crt-list with no filters and/or no options
so we can test more cases.
2020-04-02 15:40:19 +02:00
William Lallemand
785325141d REGTEST: ssl: pollute the crt-list file
Pollute localhost.crt-list with extra spaces, empty lines and comments
so the parser of the crt-list could be tested in a better way.
2020-04-01 20:10:53 +02:00
William Lallemand
fdb6db4850 REGTEST: ssl/cli: tests options and filters w/ add ssl crt-list
Now that the 'add ssl crt-list' command supports filters and options,
add some in the vtc file to test them.
2020-04-01 20:10:53 +02:00
Willy Tarreau
5e8017d53c REGTEST: make the unique-id test depend on version 2.0
Regtest unique-id.vtc was added by commit 5fcec84c58 ("REGTEST: Add
unique-id reg-test") but it relies on the "uuid" sample fetch which
is only available in version 2.0 and above. Let's reflect that in
the REQUIRE_VERSION tag.
2020-04-01 16:08:43 +02:00
Willy Tarreau
1d52c7b52b REGTEST: make the PROXY TLV validation depend on version 2.2
Regtest proxy_protocol_tlv_validation was added by commit 488ee7fb6e
("BUG/MAJOR: proxy_protocol: Properly validate TLV lengths") but it
relies on a trick involving http-after-response to append a header
after a 400-badreq response, which is not possible in earlier versions,
so make it depend on 2.2.
2020-03-31 16:37:58 +02:00
William Lallemand
4781fad407 REGTEST: ssl/cli: change test type to devel
Change the type of test from slow to devel for add_ssl-crt-list.vtc and
set_ssl_cert.vtc.
2020-03-31 14:52:22 +02:00
William Lallemand
2be4a2e02d REGTEST: ssl/cli: test the 'add ssl crt-list' command
Test the 'add ssl crt-list' feature by inserting the ecdsa.pem
certificate and verifying with curl and strict-sni that it works.
2020-03-31 12:32:18 +02:00
Jerome Magnin
eb421b2fe0 MINOR: listener: add so_name sample fetch
Add a sample fetch for the name of a bind. This can be useful to
take decisions when PROXY protocol is used and we can't rely on dst,
such as the sample config below.

  defaults
    mode http
  listen bar
    bind 127.0.0.1:1111
    server s1 127.0.1.1:1234 send-proxy

  listen foo
    bind 127.0.1.1:1234 name foo accept-proxy
    http-request return status 200 hdr dst %[dst] if { dst 127.0.1.1 }
2020-03-29 05:47:29 +02:00
Willy Tarreau
ce6fc25b17 REGTEST: increase timeouts on the seamless-reload test
The abns_socket in seamless-reload regtest regularly fails in Travis-CI
on smaller machines only (typically the ppc64le and sometimes s390x).
The error always reports an incomplete HTTP header as seen from the
client. And this can occasionally be reproduced on the minicloud ppc64le
image when setting a huge file descriptors limit (1 million).

What happens in fact is the following: depending on the binding order,
some connections from the client might reach the TCP listener on the
old instance and be forwarded to the ABNS listener of the second
instance just being prepared to start up. But due to the huge number
of FDs, setting them up takes slightly more time and the 20ms server
timeout may expire before the new instance finishes its startup. This
can result in an occasional 504, except that since the client timeout
is the same as the server timeout, both sides are closed at the same
time and the client doesn't receive the 504.

In addition a second problem plugs onto this: by default http-reuse is
enabled. Some requests being forwarded to the older instance will be
sent over an already established connection. But the CPU used by the
starting process using many FDs will be taken away from the older
process, whose abns listener will not see a request for more than 20ms,
and will decide to kill the idle client connection. At the same moment
the TCP proxy forwards a request over this closing connection, it
detects the close and silently closes the other side to let the
client retry, which is detected by the vtest client as another case
of empty header. This is easier to reproduce in VMs with few CPUs
(2 or less) and some noisy neighbors such as a few spinning loops in
background.

Let's just increase this tests' timeout to avoid this. While a few
ms are close to the scheduler's granularity, this test is never
supposed to trigger the timeouts so it's safe to go higher without
impacts on the test execution time. At one second the problem seems
impossible to reproduce on the minicloud VMs.
2020-03-23 09:11:51 +01:00
Ilya Shipitsin
77e3b4a2c4 CLEANUP: assorted typo fixes in the code and comments
These are mostly comments in the code. A few error messages were fixed
and are of low enough importance not to deserve a backport. Some regtests
were also fixed.
2020-03-14 09:42:07 +01:00
Tim Duesterhus
cf6e0c8a83 MEDIUM: proxy_protocol: Support sending unique IDs using PPv2
This patch adds the `unique-id` option to `proxy-v2-options`. If this
option is set a unique ID will be generated based on the `unique-id-format`
while sending the proxy protocol v2 header and stored as the unique id for
the first stream of the connection.

This feature is meant to be used in `tcp` mode. It works on HTTP mode, but
might result in inconsistent unique IDs for the first request on a keep-alive
connection, because the unique ID for the first stream is generated earlier
than the others.

Now that we can send unique IDs in `tcp` mode the `%ID` log variable is made
available in TCP mode.
2020-03-13 17:26:43 +01:00
Tim Duesterhus
d1b15b6e9b MINOR: proxy_protocol: Ingest PP2_TYPE_UNIQUE_ID on incoming connections
This patch reads a proxy protocol v2 provided unique ID and makes it
available using the `fc_pp_unique_id` fetch.
2020-03-13 17:25:23 +01:00
Tim Duesterhus
488ee7fb6e BUG/MAJOR: proxy_protocol: Properly validate TLV lengths
This patch fixes PROXYv2 parsing when the payload of the TCP connection is
fused with the PROXYv2 header within a single recv() call.

Previously HAProxy ignored the PROXYv2 header length when attempting to
parse the TLV, possibly interpreting the first byte of the payload as a
TLV type.

This patch adds proper validation. It ensures that:

1. TLV parsing stops when the end of the PROXYv2 header is reached.
2. TLV lengths cannot exceed the PROXYv2 header length.
3. The PROXYv2 header ends together with the last TLV, not allowing for
   "stray bytes" to be ignored.

A reg-test was added to ensure proper behavior.

This patch tries to find the sweat spot between a small and easily
backportable one, and a cleaner one that's more easily adaptable to
older versions, hence why it merges the "if" and "while" blocks which
causes a reindent of the whole block. It should be used as-is for
versions 1.9 to 2.1, the block about PP2_TYPE_AUTHORITY should be
dropped for 2.0 and the block about CRC32C should be dropped for 1.8.

This bug was introduced when TLV parsing was added. This happened in commit
b3e54fe387. This commit was first released
with HAProxy 1.6-dev1.

A similar issue was fixed in commit 7209c204bd.

This patch must be backported to HAProxy 1.6+.
2020-03-06 11:11:22 +01:00
Tim Duesterhus
5fcec84c58 REGTEST: Add unique-id reg-test
This reg-test verifies the following behavior:

1. That unique IDs are stable (i.e. the bug fixed in 530408f976)
2. That unique IDs can use values from the HTTP request (see https://www.mail-archive.com/haproxy@formilux.org/msg36436.html)
2020-03-05 07:23:00 +01:00
Christopher Faulet
6072beb214 MINOR: http-ana: Match on the path if the monitor-uri starts by a /
if the monitor-uri starts by a slash ('/'), the matching is performed against
the request's path instead of the request's uri. It is a workaround to let the
HTTP/2 requests match the monitor-uri. Indeed, in HTTP/2, clients are encouraged
to send absolute URIs only.

This patch is not tagged as a bug, because the previous behavior matched exactly
what the doc describes. But it may surprise that HTTP/2 requests don't match the
monitor-uri.

This patch may be backported to 2.1 because URIs of HTTP/2 are stored using the
absolute-form starting this version. For previous versions, this patch will only
helps explicitely absolute HTTP/1 requests (and only the HTX part because on the
legacy HTTP, all the URI is matched).

It should fix the issue #509.
2020-02-18 16:29:29 +01:00
Willy Tarreau
b5e62679aa REGTESTS: use "command -v" instead of "which"
Ilya reported that the "which" utility is not that much portable and is
absent from Fedora. "type -p" is not portable either, and the correct
solution appears to be "command -v", so let's use this for now, we can
change it again in the future in case of problems.

Link: https://www.mail-archive.com/haproxy@formilux.org/msg36332.html
2020-02-18 14:45:22 +01:00
Christopher Faulet
3610486e8c REGTESTS: Add a reg test for http-after-response rulesets
A reg test has been added to ensure the evaluation of http-after-responses rules
is functionnal for all kind of responses (server, applet and internal
responses).
2020-02-06 15:13:04 +01:00
Christopher Faulet
c5a0aeef85 REGTESTS: Add reg tests for the HTTP return action
2 reg tests have been added to ensure the HTTP return action is functionnal. A
reg test is about returning error files. The other one is about returning
default responses and responses based on string or file payloads.
2020-02-06 15:13:04 +01:00
Willy Tarreau
fa10ffdd12 REGTESTS: make the set_ssl_cert test require version 2.2
It doesn't only use "set ssl cert" but also relies on "show ssl cert"
which is only available in 2.2.
2020-01-24 14:29:08 +01:00
Christopher Faulet
a5afb0bf36 REGEST: Add reg tests about error files
2 reg tests are added. The first one ensures the declaration of errors in a
proxy is fonctionnal. It declares http-errors sections and declare error files
using the errorfile and the errorfiles directives, both in the default section
and the frontend sections. The second one ensures it is possible to use a custom
error file for an HTTP deny rule.
2020-01-20 15:19:55 +01:00
Christopher Faulet
cedf8fbcc6 REGTEST: Add a strict rewriting mode reg test
With this new reg test we ensure the strict rewriting mode of HTTP rules is
functional. The mode is tested for request and response rules. The default mode
(strict), the swtich off and the reset on new ruleset are tested for both.
2020-01-20 15:19:55 +01:00
Christopher Faulet
95d169ba9a REGTEST: Fix format of set-uri HTTP request rule in h1or2_to_h1c.vtc
First, concat() is a converter, not a sample fetch. So use str() sample fetch
with no string and call concat on it. Then, the argument of the set-uri rule
must be a log format string. So it must be inside %[] to be evaluated.
2020-01-20 15:18:45 +01:00
Christopher Faulet
01f44456e6 MINOR: http-htx: Move htx sample fetches in the scope "internal"
HTX sample fetches are now prefixed by "internal." to explicitly reserve their
uses for debugging or testing purposes.
2020-01-20 15:18:45 +01:00
Willy Tarreau
ec9ac54982 REGTEST: add sample_fetches/hashes.vtc to validate hashes
This regtest validates all hashes that we support, on all input bytes from
0x00 to 0xFF. Those supporting avalanche are tested as well. It also tests
len(), hex() and base64(). It purposely does not enable sha2() because this
one relies on OpenSSL and there's no point in validating that OpenSSL knows
how to hash, what matters is that we can test our hashing functions in all
cases. However since the tests were written, they're still present and
commented out in case that helps.

It may be backported to supported versions, possibly dropping a few algos
that were not supported (e.g. crc32c requires 1.9 minimum).

Note that this test will fail on crc32/djb2/sdbm/wt6 unless patches
"BUG/MINOR: stream: init variables when the list is empty" and
"BUG/MAJOR: hashes: fix the signedness of the hash inputs" are included.
2020-01-16 08:45:27 +01:00
William Lallemand
25b5693021 REGTEST: mcli/mcli_start_progs: start 2 programs
This regtest tests the issue #446 by starting 2 programs and checking if
they exist in the "show proc" of the master CLI.

Should be backported as far as 2.0.
2020-01-14 15:42:38 +01:00
Ilya Shipitsin
3e34ac7d09 REGTEST: set_ssl_cert.vtc: replace "echo" with "printf"
"echo -e" for some reason does not work on travis-ci, so let us switch
to "printf"

Fixes: #423
2020-01-09 05:56:04 +01:00
Willy Tarreau
8c4c1d4299 REGTEST: make the "set ssl cert" require version 2.1
This test fails on 2.0 and earlier since the feature was introduced in 2.1,
let's add the REQUIRE_VERSION tag.
2019-12-20 14:35:18 +01:00
William Lallemand
d5b464bfee REGTEST: run-regtests: implement #REQUIRE_BINARIES
Implement #REQUIRE_BINARIES for vtc files.

The run-regtests.sh script will check if the binary is available in the
environment, if not, it wil disable the vtc.
2019-12-19 14:36:46 +01:00
William Lallemand
9c1aa0a2a1 REGTEST: ssl: test the "set ssl cert" CLI command
Add a reg-test which test the update of a certificate over the CLI. This
test requires socat and curl.

This commit also adds an ECDSA certificate in the ssl directory.
2019-12-19 13:51:38 +01:00
Tim Duesterhus
cd3732456b MINOR: sample: Validate the number of bits for the sha2 converter
Instead of failing the conversion when an invalid number of bits is
given the sha2 converter now fails with an appropriate error message
during startup.

The sha2 converter was introduced in d437630237,
which is in 2.1 and higher.
2019-12-17 13:28:00 +01:00
Christopher Faulet
1eee6ca89e REGTEST: Add an HTX reg-test to check an edge case
This test checks that an HTTP message is properly processed when we failed to
add the HTX EOM block in an HTX message during the parsing because the buffer is
full. Some space must be released in the buffer to make it possible. This
requires an extra pass in the H1 multiplexer. Here, we must be sure the mux is
called while there is no more incoming data.

It is a "devel" test because conditions to run the test successfully is highly
dependent on the implementation. So if it fail, it is not necessarily a bug. It
may be due of an internal change. It relies on internal HTX sample fetches.
2019-12-11 16:46:16 +01:00
vkill
1dfd16536f MINOR: backend: Add srv_name sample fetche
The sample fetche can get srv_name without foreach
`core.backends["bk"].servers`.

Then we can get Server class quickly via
`core.backends[txn.f:be_name()].servers[txn.f:srv_name()]`.

Issue#342
2019-11-01 05:40:24 +01:00
Jerome Magnin
1dccfbeb5b REGTEST: vtest can now enable mcli with its own flag
VTest can now enable mworker and mcli with separate flags so lets
update vtc files that need it. This also allows to revert the change
made with 1545a59c ("REGTESTS: make seamless-reload depend on 1.9
and above").
2019-10-30 19:10:04 +01:00
Willy Tarreau
04068a1939 REGTESTS: server/cli_set_fqdn requires version 1.8 minimum
This test uses "set server <srv> fqdn" which is not available in 1.7.
All reg-tests now pass on 1.7.
2019-10-22 13:06:59 +02:00
Willy Tarreau
1545a59c7b REGTESTS: make seamless-reload depend on 1.9 and above
Since latest updates, vtest requires the master CLI when running in
master-worker mode, and this one is only available starting with 1.9.
The seamless reload test is the only one depending on this and now
fails on 1.8, so let's adjust it accordingly.
2019-10-22 10:42:10 +02:00
Tim Duesterhus
4381d26edc BUG/MINOR: sample: Make the field converter compatible with -m found
Previously an expression like:

    path,field(2,/) -m found

always returned `true`.

Bug exists since the `field` converter exists. That is:
f399b0debf

The fix should be backported to 1.6+.
2019-10-21 15:49:42 +02:00
William Lallemand
cd48277469 REGTEST: mcli/mcli_show_info: launch a 'show info' on the master CLI
This test launches a HAProxy process in master worker with 'nbproc 4'.
It sends a "show info" to the process 3 and verify that the right
process replied.

This regtest depends on the support of the master CLI for VTest.
2019-10-18 14:47:30 +02:00