Commit Graph

12807 Commits

Author SHA1 Message Date
Willy Tarreau
3b8f9b7b88 BUG/MEDIUM: lists: add missing store barrier in MT_LIST_ADD/MT_LIST_ADDQ
The torture test run for previous commit 787dc20 ("BUG/MEDIUM: lists: add
missing store barrier on MT_LIST_BEHEAD()") finally broke again after 34M
connections. It appeared that MT_LIST_ADD and MT_LIST_ADDQ were suffering
from the same missing barrier when restoring the original pointers before
giving up, when checking if the element was already added. This is indeed
something which seldom happens with the shared scheduler, in case two
threads simultaneously try to wake up the same tasklet.

With a store barrier there after reverting the pointers, the torture test
survived 750M connections on the NanoPI-Fire3, so it looks good this time.
Probably that MT_LIST_BEHEAD should be added to test-list.c since it seems
to be more sensitive to concurrent accesses with MT_LIST_ADDQ.

It's worth noting that there is no barrier between the last two pointers
update, while there is one in MT_LIST_POP and MT_LIST_BEHEAD, the latter
having shown to be needed, but I cannot demonstrate why we would need
one here. Given that the code seems solid here, let's stick to what is
shown to work.

This fix should be backported to 2.1, just for the sake of safety since
the issue couldn't be triggered there, but it could change with the
compiler or when backporting a fix for example.
2020-07-09 05:01:27 +02:00
Willy Tarreau
787dc20952 BUG/MEDIUM: lists: add missing store barrier on MT_LIST_BEHEAD()
When running multi-threaded tests on my NanoPI-Fire3 (8 A53 cores), I
managed to occasionally get either a bus error or a segfault in the
scheduler, but only when running at a high connection rate (injecting
on a tcp-request connection reject rule). The bug is rare and happens
around once per million connections. I could never reproduce it with
less than 4 threads nor on A72 cores.

Haproxy 2.1.0 would also fail there but not 2.1.7.

Every time the crash happened with the TL_URGENT task list corrupted,
though it was not immediately after the LIST_SPLICE() call, indicating
background activity survived the MT_LIST_BEHEAD() operation. This queue
is where the shared runqueue is transferred, and the shared runqueue
gets fast inter-thread tasklet wakeups from idle conn takeover and new
connections.

Comparing the MT_LIST_BEHEAD() and MT_LIST_DEL() implementations, it's
quite obvious that a few barriers are missing from the former, and these
will simply fail on weakly ordered caches.

Two store barriers were added before the break() on failure, to match
what is done on the normal path. Missing them almost always results in
a segfault which is quite rare but consistent (after ~3M connections).

The 3rd one before updating n->prev seems intuitively needed though I
coudln't make the code fail without it. It's present in MT_LIST_DEL so
better not be needlessly creative. The last one is the most important
one, and seems to be the one that helps a concurrent MT_LIST_ADDQ()
detect a late failure and try again. With this, the code survives at
least 30M connections.

Interestingly the exact same issue was addressed in 2.0-dev2 for MT_LIST_DEL
with commit 690d2ad4d ("BUG/MEDIUM: list: add missing store barriers when
updating elements and head").

This fix must be backported to 2.1 as MT_LIST_BEHEAD() is also used there.

It's only tagged as medium because it will only affect entry-level CPUs
like Cortex A53 (x86 are not affected), and requires load levels that are
very hard to achieve on such machines to trigger it. In practice it's
unlikely anyone will ever hit it.
2020-07-08 19:45:50 +02:00
Pierre Cheynier
1e36976352 CLEANUP: contrib/prometheus-exporter: typo fixes for ssl reuse metric
A typo I identified while having a look to our metric inventory.
(s/frontent/frontend)
2020-07-07 23:15:03 +02:00
Tim Duesterhus
d0c0ca2720 CLEANUP: Add static void hlua_deinit()
Compiling HAProxy with USE_LUA=1 and running a configuration check within
valgrind with a very simple configuration such as:

    listen foo
    	bind *:8080

Will report quite a few possible leaks afterwards:

    ==24048== LEAK SUMMARY:
    ==24048==    definitely lost: 0 bytes in 0 blocks
    ==24048==    indirectly lost: 0 bytes in 0 blocks
    ==24048==      possibly lost: 95,513 bytes in 1,209 blocks
    ==24048==    still reachable: 329,960 bytes in 71 blocks
    ==24048==         suppressed: 0 bytes in 0 blocks

Printing these possible leaks shows that all of them are caused by Lua.
Luckily Lua makes it *very* easy to free all used memory, so let's do
this on shutdown.

Afterwards this patch is applied the output looks much better:

    ==24199== LEAK SUMMARY:
    ==24199==    definitely lost: 0 bytes in 0 blocks
    ==24199==    indirectly lost: 0 bytes in 0 blocks
    ==24199==      possibly lost: 0 bytes in 0 blocks
    ==24199==    still reachable: 329,960 bytes in 71 blocks
    ==24199==         suppressed: 0 bytes in 0 blocks
2020-07-07 16:52:35 +02:00
Tim Duesterhus
bbdd5b8ca9 CLEANUP: Add static void vars_deinit()
vars_deinit() frees all var_names during deinit().
2020-07-07 16:52:35 +02:00
Tim Duesterhus
34bef074c6 CLEANUP: haproxy: Free post_server_check_list in deinit()
This allocation is technically always reachable and cannot leak, but so are
a few others that *are* freed.
2020-07-07 16:52:35 +02:00
Tim Duesterhus
0837eb11cf CLEANUP: haproxy: Free server_deinit_list in deinit()
This allocation is technically always reachable and cannot leak, but so are
a few others that *are* freed.
2020-07-07 16:52:35 +02:00
Tim Duesterhus
fdf904a297 CLEANUP: haproxy: Free post_deinit_list in deinit()
This allocation is technically always reachable and cannot leak, but so are
a few others that *are* freed.
2020-07-07 16:52:35 +02:00
Tim Duesterhus
17e363f751 CLEANUP: haproxy: Free proxy_deinit_list in deinit()
This allocation is technically always reachable and cannot leak, but so are
a few others that *are* freed.
2020-07-07 16:52:35 +02:00
Tim Duesterhus
041a626a8a BUG/MINOR: sample: Free str.area in smp_check_const_meth
Given the following example configuration:

    listen foo
    	mode http
    	bind *:8080
    	http-request set-var(txn.leak) meth(GET)
    	server x example.com:80

Running a configuration check with valgrind reports:

    ==25992== 4 bytes in 1 blocks are definitely lost in loss record 1 of 344
    ==25992==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==25992==    by 0x4E239D: my_strndup (tools.c:2261)
    ==25992==    by 0x581E20: make_arg_list (arg.c:253)
    ==25992==    by 0x4DE91D: sample_parse_expr (sample.c:890)
    ==25992==    by 0x58E304: parse_store (vars.c:772)
    ==25992==    by 0x566A3F: parse_http_req_cond (http_rules.c:95)
    ==25992==    by 0x4A4CE6: cfg_parse_listen (cfgparse-listen.c:1339)
    ==25992==    by 0x494C59: readcfgfile (cfgparse.c:2049)
    ==25992==    by 0x545145: init (haproxy.c:2029)
    ==25992==    by 0x421E42: main (haproxy.c:3175)

After this patch is applied the leak is gone as expected.

This is a fairly minor leak, but it can add up for many uses of the `bool()`
sample fetch. The bug most likely exists since the `bool()` sample fetch was
introduced in commit cc103299c7. The fix may
be backported to HAProxy 1.6+.
2020-07-07 16:52:35 +02:00
Tim Duesterhus
c7d8a86f2f BUG/MINOR: sample: Free str.area in smp_check_const_bool
Given the following example configuration:

    listen foo
    	mode http
    	bind *:8080
    	http-request set-var(txn.leak) bool(1)
    	server x example.com:80

Running a configuration check with valgrind reports:

    ==24233== 2 bytes in 1 blocks are definitely lost in loss record 1 of 345
    ==24233==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==24233==    by 0x4E238D: my_strndup (tools.c:2261)
    ==24233==    by 0x581E10: make_arg_list (arg.c:253)
    ==24233==    by 0x4DE90D: sample_parse_expr (sample.c:890)
    ==24233==    by 0x58E2F4: parse_store (vars.c:772)
    ==24233==    by 0x566A2F: parse_http_req_cond (http_rules.c:95)
    ==24233==    by 0x4A4CE6: cfg_parse_listen (cfgparse-listen.c:1339)
    ==24233==    by 0x494C59: readcfgfile (cfgparse.c:2049)
    ==24233==    by 0x545135: init (haproxy.c:2029)
    ==24233==    by 0x421E42: main (haproxy.c:3175)

After this patch is applied the leak is gone as expected.

This is a fairly minor leak, but it can add up for many uses of the `bool()`
sample fetch. The bug most likely exists since the `bool()` sample fetch was
introduced in commit cc103299c7. The fix may
be backported to HAProxy 1.6+.
2020-07-07 16:52:35 +02:00
Tim Duesterhus
826cf0729b BUG/MINOR: haproxy: Free srule->expr during deinit
Given the following example configuration:

    backend foo
    	mode http
    	use-server %[str(x)] if { always_true }
    	server x example.com:80

Running a configuration check with valgrind reports:

    ==19376== 170 (40 direct, 130 indirect) bytes in 1 blocks are definitely lost in loss record 281 of 347
    ==19376==    at 0x4C2FB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==19376==    by 0x5091AC: add_sample_to_logformat_list (log.c:511)
    ==19376==    by 0x50A5A6: parse_logformat_string (log.c:671)
    ==19376==    by 0x4957F2: check_config_validity (cfgparse.c:2588)
    ==19376==    by 0x54442D: init (haproxy.c:2129)
    ==19376==    by 0x421E42: main (haproxy.c:3169)

After this patch is applied the leak is gone as expected.

This is a very minor leak that can only be observed if deinit() is called,
shortly before the OS will free all memory of the process anyway. No
backport needed.
2020-07-07 16:52:35 +02:00
Tim Duesterhus
6fb74a1dc3 BUG/MINOR: haproxy: Free srule->file during deinit
Given the following example configuration:

    backend foo
    	mode http
    	use-server x if { always_true }
    	server x example.com:80

Running a configuration check with valgrind reports:

    ==18650== 14 bytes in 1 blocks are definitely lost in loss record 3 of 345
    ==18650==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==18650==    by 0x649E489: strdup (strdup.c:42)
    ==18650==    by 0x4A5438: cfg_parse_listen (cfgparse-listen.c:1548)
    ==18650==    by 0x494C59: readcfgfile (cfgparse.c:2049)
    ==18650==    by 0x5450B5: init (haproxy.c:2029)
    ==18650==    by 0x421E42: main (haproxy.c:3168)

After this patch is applied the leak is gone as expected.

This is a very minor leak that can only be observed if deinit() is called,
shortly before the OS will free all memory of the process anyway. No
backport needed.
2020-07-07 16:52:35 +02:00
Tim Duesterhus
18c63591f0 BUG/MINOR: haproxy: Free proxy->unique_id_header during deinit
Given the following example configuration:

    frontend foo
    	mode http
    	bind *:8080
    	unique-id-header x

Running a configuration check with valgrind reports:

    ==17621== 2 bytes in 1 blocks are definitely lost in loss record 1 of 341
    ==17621==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==17621==    by 0x649E489: strdup (strdup.c:42)
    ==17621==    by 0x4A87F1: cfg_parse_listen (cfgparse-listen.c:2747)
    ==17621==    by 0x494C59: readcfgfile (cfgparse.c:2049)
    ==17621==    by 0x545095: init (haproxy.c:2029)
    ==17621==    by 0x421E42: main (haproxy.c:3167)

After this patch is applied the leak is gone as expected.

This is a very minor leak that can only be observed if deinit() is called,
shortly before the OS will free all memory of the process anyway. No
backport needed.
2020-07-07 16:52:35 +02:00
Tim Duesterhus
cb8f13c26d BUG/MINOR: haproxy: Add missing free of server->(hostname|resolvers_id)
Given the following example configuration:

    resolvers test
    	nameserver test 127.0.0.1:53
    listen foo
    	bind *:8080
    	server foo example.com resolvers test

Running a configuration check within valgrind reports:

    ==21995== 5 bytes in 1 blocks are definitely lost in loss record 1 of 30
    ==21995==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==21995==    by 0x5726489: strdup (strdup.c:42)
    ==21995==    by 0x4B2CFB: parse_server (server.c:2163)
    ==21995==    by 0x4680C1: cfg_parse_listen (cfgparse-listen.c:534)
    ==21995==    by 0x459E33: readcfgfile (cfgparse.c:2167)
    ==21995==    by 0x50778D: init (haproxy.c:2021)
    ==21995==    by 0x418262: main (haproxy.c:3133)
    ==21995==
    ==21995== 12 bytes in 1 blocks are definitely lost in loss record 3 of 30
    ==21995==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==21995==    by 0x5726489: strdup (strdup.c:42)
    ==21995==    by 0x4AC666: srv_prepare_for_resolution (server.c:1606)
    ==21995==    by 0x4B2EBD: parse_server (server.c:2081)
    ==21995==    by 0x4680C1: cfg_parse_listen (cfgparse-listen.c:534)
    ==21995==    by 0x459E33: readcfgfile (cfgparse.c:2167)
    ==21995==    by 0x50778D: init (haproxy.c:2021)
    ==21995==    by 0x418262: main (haproxy.c:3133)

with one more leak unrelated to `struct server`. After applying this
patch the leak is gone as expected.

This is a very minor leak that can only be observed if deinit() is called,
shortly before the OS will free all memory of the process anyway. No
backport needed.
2020-07-07 16:52:35 +02:00
Tim Duesterhus
797657875f BUG/MINOR: haproxy: Free proxy->format_unique_id during deinit
Given the following example configuration:

    frontend foo
    	mode http
    	bind *:8080
    	unique-id-format x

Running a configuration check with valgrind reports:

    ==30712== 42 (40 direct, 2 indirect) bytes in 1 blocks are definitely lost in loss record 18 of 39
    ==30712==    at 0x4C2FB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==30712==    by 0x4ED7E9: add_to_logformat_list (log.c:462)
    ==30712==    by 0x4EEE28: parse_logformat_string (log.c:720)
    ==30712==    by 0x47B09A: check_config_validity (cfgparse.c:3046)
    ==30712==    by 0x52881D: init (haproxy.c:2121)
    ==30712==    by 0x41F382: main (haproxy.c:3126)

After this patch is applied the leak is gone as expected.

This is a very minor leak that can only be observed if deinit() is called,
shortly before the OS will free all memory of the process anyway. No
backport needed.
2020-07-07 16:52:35 +02:00
Tim Duesterhus
867cd98546 BUG/MINOR: sample: Fix freeing of conv_exprs in release_sample_expr
Instead of just calling release_sample_arg(conv_expr->arg_p) we also must
free() the conv_expr itself (after removing it from the list).

Given the following example configuration:

    frontend foo
    	bind *:8080
    	mode http
    	http-request set-var(txn.foo) str(bar)
    	acl is_match str(foo),strcmp(txn.hash) -m bool

Running a configuration check within valgrind reports:

    ==1431== 32 bytes in 1 blocks are definitely lost in loss record 20 of 43
    ==1431==    at 0x4C2FB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==1431==    by 0x4C39B5: sample_parse_expr (sample.c:982)
    ==1431==    by 0x56B410: parse_acl_expr (acl.c:319)
    ==1431==    by 0x56BA7F: parse_acl (acl.c:697)
    ==1431==    by 0x48D225: cfg_parse_listen (cfgparse-listen.c:816)
    ==1431==    by 0x4797C3: readcfgfile (cfgparse.c:2167)
    ==1431==    by 0x52943D: init (haproxy.c:2021)
    ==1431==    by 0x41F382: main (haproxy.c:3133)

After this patch is applied the leak is gone as expected.

This is a fairly minor leak that can only be observed if samples need to be
freed, which is not something that should occur during normal processing and
most likely only during shut down. Thus no backport should be needed.
2020-07-07 16:52:35 +02:00
Tim Duesterhus
9fa0df55cd BUG/MINOR: acl: Fix freeing of expr->smp in prune_acl_expr
Instead of simply calling free() in expr->smp->arg_p in certain cases
properly free the sample using release_sample_expr().

Given the following example configuration:

    frontend foo
    	bind *:8080
    	mode http
    	http-request set-var(txn.foo) str(bar)
    	acl is_match str(foo),strcmp(txn.hash) -m bool

Running a configuration check within valgrind reports:

    ==31371== 160 (48 direct, 112 indirect) bytes in 1 blocks are definitely lost in loss record 35 of 45
    ==31371==    at 0x4C2FB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==31371==    by 0x4C3832: sample_parse_expr (sample.c:876)
    ==31371==    by 0x56B3E0: parse_acl_expr (acl.c:319)
    ==31371==    by 0x56BA4F: parse_acl (acl.c:697)
    ==31371==    by 0x48D225: cfg_parse_listen (cfgparse-listen.c:816)
    ==31371==    by 0x4797C3: readcfgfile (cfgparse.c:2167)
    ==31371==    by 0x5293ED: init (haproxy.c:2021)
    ==31371==    by 0x41F382: main (haproxy.c:3126)

After this patch this leak is reduced. It will be fully removed in a
follow up patch:

    ==32503== 32 bytes in 1 blocks are definitely lost in loss record 20 of 43
    ==32503==    at 0x4C2FB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==32503==    by 0x4C39B5: sample_parse_expr (sample.c:982)
    ==32503==    by 0x56B410: parse_acl_expr (acl.c:319)
    ==32503==    by 0x56BA7F: parse_acl (acl.c:697)
    ==32503==    by 0x48D225: cfg_parse_listen (cfgparse-listen.c:816)
    ==32503==    by 0x4797C3: readcfgfile (cfgparse.c:2167)
    ==32503==    by 0x52943D: init (haproxy.c:2021)
    ==32503==    by 0x41F382: main (haproxy.c:3133)

This is a fairly minor leak that can only be observed if ACLs need to be
freed, which is not something that should occur during normal processing
and most likely only during shut down. Thus no backport should be needed.
2020-07-07 16:52:35 +02:00
William Dauchy
a5194607ab MINOR: config: make strict limits enabled by default
as agreed a few months ago, enable strict-limits for v2.3
update configuration manual accordingly

Signed-off-by: William Dauchy <w.dauchy@criteo.com>
2020-07-07 16:52:35 +02:00
Willy Tarreau
848dbdf918 [RELEASE] Released version 2.3-dev0
Released version 2.3-dev0 with the following main changes :
    - [RELEASE] Released version 2.3-dev0
    - MINOR: version: back to development, update status message
2020-07-07 16:39:18 +02:00
Willy Tarreau
e3cb9978c2 MINOR: version: back to development, update status message
Update the status message and update INSTALL again.
2020-07-07 16:38:51 +02:00
Willy Tarreau
33205c23a7 [RELEASE] Released version 2.3-dev0
Released version 2.3-dev0 with the following main changes :
    - exact copy of 2.2.0
2020-07-07 16:35:28 +02:00
Willy Tarreau
3a00c915fd [RELEASE] Released version 2.2.0
Released version 2.2.0 with the following main changes :
    - BUILD: mux-h2: fix typo breaking build when using DEBUG_LOCK
    - CLEANUP: makefile: update the outdated list of DEBUG_xxx options
    - BUILD: tools: make resolve_sym_name() return a const
    - CLEANUP: auth: fix useless self-include of auth-t.h
    - BUILD: tree-wide: cast arguments to tolower/toupper to unsigned char
    - CLEANUP: assorted typo fixes in the code and comments
    - WIP/MINOR: ssl: add sample fetches for keylog in frontend
    - DOC: fix tune.ssl.keylog sample fetches array
    - BUG/MINOR: ssl: check conn in keylog sample fetch
    - DOC: configuration: various typo fixes
    - MINOR: log: Remove unused case statement during the log-format string parsing
    - BUG/MINOR: mux-h1: Fix the splicing in TUNNEL mode
    - BUG/MINOR: mux-h1: Don't read data from a pipe if the mux is unable to receive
    - BUG/MINOR: mux-h1: Disable splicing only if input data was processed
    - BUG/MEDIUM: mux-h1: Disable splicing for the conn-stream if read0 is received
    - MINOR: mux-h1: Improve traces about the splicing
    - BUG/MINOR: backend: Remove CO_FL_SESS_IDLE if a client remains on the last server
    - BUG/MEDIUM: connection: Don't consider new private connections as available
    - BUG/MINOR: connection: See new connection as available only on reuse always
    - DOC: configuration: remove obsolete mentions of H2 being converted to HTTP/1.x
    - CLEANUP: ssl: remove unrelevant comment in smp_fetch_ssl_x_keylog()
    - DOC: update INSTALL with new compiler versions
    - DOC: minor update to coding style file
    - MINOR: version: mention that it's an LTS release now
2020-07-07 16:33:14 +02:00
Willy Tarreau
44c47de81a MINOR: version: mention that it's an LTS release now
The new version is going to be LTS up to around Q2 2025.
2020-07-07 16:31:52 +02:00
Willy Tarreau
bdc62096fc DOC: minor update to coding style file
It used to still refrence includes/{types,proto,common}/, now turned
to import/ and haproxy/.
2020-07-07 16:23:11 +02:00
Willy Tarreau
4ced4bd426 DOC: update INSTALL with new compiler versions
gcc is known to work up to 10.1. Also update the message about the
development version.
2020-07-07 16:23:11 +02:00
William Lallemand
bd97f1d758 CLEANUP: ssl: remove unrelevant comment in smp_fetch_ssl_x_keylog()
Remove a comment.
2020-07-07 16:14:08 +02:00
Willy Tarreau
253c2519c2 DOC: configuration: remove obsolete mentions of H2 being converted to HTTP/1.x
The first H2 implementation in version 1.8 used to turn HTTP/2 requests
to HTTP/1.1, causing many limitations. This is not true anymore and we
don't suffer from the lack of server-side H2 nor are we forced to close
mode anymore, so let's remove such obsolete mentions.

This could be backported to 2.0.
2020-07-07 15:55:23 +02:00
Christopher Faulet
2883fcf65b BUG/MINOR: connection: See new connection as available only on reuse always
When the multiplexer creation is delayed after the handshakes phase, the
connection is added in the available connection list if http-reuse never is not
configured for the backend. But it is a wrong statement. At this step, the
connection is not safe because it is a new connection. So it must be added in
the available connection list only if http-reuse always is used.

No backport needed, this is 2.2-dev.
2020-07-07 14:31:01 +02:00
Christopher Faulet
aa27853ce2 BUG/MEDIUM: connection: Don't consider new private connections as available
When a connection is created and the multiplexer is installed, if the connection
is marked as private, don't consider it as available, regardless the number of
available streams. This test is performed when the mux is installed when the
connection is created, in connect_server(), and when the mux is installed after
the handshakes stage.

No backport needed, this is 2.2-dev.
2020-07-07 14:30:38 +02:00
Christopher Faulet
e91a526c8f BUG/MINOR: backend: Remove CO_FL_SESS_IDLE if a client remains on the last server
When a connection is picked from the session server list because the proxy or
the session are marked to use the last requested server, if it is idle, we must
marked it as used removing the CO_FL_SESS_IDLE flag and decrementing the session
idle_conns counter.

This patch must be backported as far as 1.9.
2020-07-07 14:30:26 +02:00
Christopher Faulet
27182297c7 MINOR: mux-h1: Improve traces about the splicing
Trace messages have been added when the CS_FL_MAY_SPLICE flag is set or unset
and when the splicing is really enabled for the H1 connection.

This patch may be backpored to 2.1 to ease debugging.
2020-07-07 14:29:18 +02:00
Christopher Faulet
a131a8fe93 BUG/MEDIUM: mux-h1: Disable splicing for the conn-stream if read0 is received
The CS_FL_MAY_SPLICE flag must be unset for the conn-stream if a read0 is
received while reading on the kernel pipe. It is mandatory when some data was
also received. Otherwise, this flag prevent the call to the h1 rcv_buf()
callback. Thus the read0 will never be handled by the h1 multiplexer leading to
a freeze of the session until a timeout is reached.

This patch must be backported to 2.1 and 2.0.
2020-07-07 14:29:18 +02:00
Christopher Faulet
7b7016bf6e BUG/MINOR: mux-h1: Disable splicing only if input data was processed
In h1_rcv_buf(), the splicing is systematically disabled if it was previously
enabled. When it happens, if the splicing is enabled it means the channel's
buffer was empty before calling h1_rcv_buf(). Thus, the only reason to disable
the splicing at this step is when some input data have just been processed.

This patch may be backported to 2.1 and 2.0.
2020-07-07 14:29:18 +02:00
Christopher Faulet
0060be9ccf BUG/MINOR: mux-h1: Don't read data from a pipe if the mux is unable to receive
In h1_rcv_pipe(), if the mux is unable to receive data, for instance because the
multiplexer is blocked on input waiting the other side (BUSY mode), no receive
must be performed.

This patch must be backported to 2.1 and 2.0.
2020-07-07 14:29:18 +02:00
Christopher Faulet
2eaf30955f BUG/MINOR: mux-h1: Fix the splicing in TUNNEL mode
In the commit 17ccd1a35 ("BUG/MEDIUM: connection: add a mux flag to indicate
splice usability"), The CS_FL_MAY_SPLICE flags was added to notify the upper
layer that the mux is able to use the splicing. But this was only done for the
payload in a message, in HTTP_MSG_DATA state. But the splicing is also possible
in TUNNEL mode, in HTTP_MSG_TUNNEL state. In addition, the splicing ability is
always disabled for chunked messages.

This patch must be backported to 2.1 and 2.0.
2020-07-07 14:29:18 +02:00
Christopher Faulet
b4174d5a79 MINOR: log: Remove unused case statement during the log-format string parsing
Since the commit cd0d2ed6e ("MEDIUM: log-format: make the LF parser aware of
sample expressions' end"), the LF_STEXPR label in the last switch-case statement
at the end of the for loop in the parse_logformat_string() function cannot be
reached anymore.

This patch should fix the issue #723.
2020-07-07 14:28:43 +02:00
Daniel Corbett
67a8271cc7 DOC: configuration: various typo fixes
Quick round of typo corrections for configuration.txt
2020-07-07 11:55:16 +02:00
William Lallemand
eec1d45f9d BUG/MINOR: ssl: check conn in keylog sample fetch
Add a check on the conn pointer to avoid a NULL dereference in
smp_fetch_ssl_x_keylog().

The problem is not suppose to happen because the function is only used
for the frontend at the moment.

Introduced by 7d42ef5, 2.2 only.

Fix issue #733.
2020-07-07 10:48:13 +02:00
William Lallemand
d742b6caf7 DOC: fix tune.ssl.keylog sample fetches array
The labels EXPORTER_SECRET and EARLY_EXPORTER_SECRET were swapped in the
array.
2020-07-07 10:25:15 +02:00
William Lallemand
7d42ef5b22 WIP/MINOR: ssl: add sample fetches for keylog in frontend
OpenSSL 1.1.1 provides a callback registering function
SSL_CTX_set_keylog_callback, which allows one to receive a string
containing the keys to deciphers TLSv1.3.

Unfortunately it is not possible to store this data in binary form and
we can only get this information using the callback. Which means that we
need to store it until the connection is closed.

This patches add 2 pools, the first one, pool_head_ssl_keylog is used to
store a struct ssl_keylog which will be inserted as a ex_data in a SSL *.
The second one is pool_head_ssl_keylog_str which will be used to store
the hexadecimal strings.

To enable the capture of the keys, you need to set "tune.ssl.keylog on"
in your configuration.

The following fetches were implemented:

ssl_fc_client_early_traffic_secret,
ssl_fc_client_handshake_traffic_secret,
ssl_fc_server_handshake_traffic_secret,
ssl_fc_client_traffic_secret_0,
ssl_fc_server_traffic_secret_0,
ssl_fc_exporter_secret,
ssl_fc_early_exporter_secret
2020-07-06 19:08:03 +02:00
Ilya Shipitsin
46a030cdda CLEANUP: assorted typo fixes in the code and comments
This is 11th iteration of typo fixes
2020-07-06 14:34:32 +02:00
Willy Tarreau
f278eec37a BUILD: tree-wide: cast arguments to tolower/toupper to unsigned char
NetBSD apparently uses macros for tolower/toupper and complains about
the use of char for array subscripts. Let's properly cast all of them
to unsigned char where they are used.

This is needed to fix issue #729.
2020-07-05 21:50:02 +02:00
Willy Tarreau
b0be8ae2a8 CLEANUP: auth: fix useless self-include of auth-t.h
Since recent include cleanups auth-t.h ended up including itself.
2020-07-05 21:32:47 +02:00
Willy Tarreau
0c439d8956 BUILD: tools: make resolve_sym_name() return a const
Originally it was made to return a void* because some comparisons in the
code where it was used required a lot of casts. But now we don't need
that anymore. And having it non-const breaks the build on NetBSD 9 as
reported in issue #728.

So let's switch to const and adjust debug.c to accomodate this.
2020-07-05 20:26:04 +02:00
Willy Tarreau
4c442b08b3 CLEANUP: makefile: update the outdated list of DEBUG_xxx options
A few options didn't exist anymore (FSM, HASH) and quite a few ones were
added since last update (MEM_STATS, DONT_SHARE_POOLS, NO_LOCKLESS_POOLS,
NO_LOCAL_POOLS, FAIL_ALLOC, STRICT_NOCRASH, HPACK.
2020-07-04 12:43:46 +02:00
Willy Tarreau
46ac781779 BUILD: mux-h2: fix typo breaking build when using DEBUG_LOCK
A typo was accidently introduced in commit 48ce6a3 ("BUG/MEDIUM: muxes:
Make sure nobody stole the connection before using it."), a "&" was
placed in front of "OTHER_LOCK", which breaks DEBUG_LOCK. No backport
is needed.
2020-07-04 07:16:18 +02:00
Willy Tarreau
62f11a589f [RELEASE] Released version 2.2-dev12
Released version 2.2-dev12 with the following main changes :
    - BUG/MINOR: mux_h2: don't lose the leaving trace in h2_io_cb()
    - MINOR: cli: make "show sess" stop at the last known session
    - CLEANUP: buffers: remove unused buffer_wq_lock lock
    - BUG/MEDIUM: buffers: always allocate from the local cache first
    - MINOR: connection: align toremove_{lock,connections} and cleanup into idle_conns
    - CONTRIB: debug: add missing flags SI_FL_L7_RETRY & SI_FL_D_L7_RETRY
    - BUG/MEDIUM: connections: Don't increase curr_used_conns for shared connections.
    - BUG/MEDIUM: checks: Increment the server's curr_used_conns
    - REORG: buffer: rename buffer.c to dynbuf.c
    - REORG: includes: create tinfo.h for the thread_info struct
    - CLEANUP: pool: only include the type files from types
    - MINOR: pools: move the LRU cache heads to thread_info
    - BUG/MINOR: debug: fix "show fd" null-deref when built with DEBUG_FD
    - MINOR: stats: add 3 new output values for the per-server idle conn state
    - MINOR: activity: add per-thread statistics on FD takeover
    - BUG/MINOR: server: start cleaning idle connections from various points
    - MEDIUM: server: improve estimate of the need for idle connections
    - MINOR: stats: add the estimated need of concurrent connections per server
    - BUG/MINOR: threads: Don't forget to init each thread toremove_lock.
    - BUG/MEDIUM: lists: Lock the element while we check if it is in a list.
    - Revert "BUG/MEDIUM: lists: Lock the element while we check if it is in a list."
    - BUG/MINOR: haproxy: don't wake already stopping threads on exit
    - BUG/MINOR: server: always count one idle slot for current thread
    - MEDIUM: server: use the two thresholds for the connection release algorithm
    - BUG/MINOR: http-rules: Fix ACLs parsing for http deny rules
    - BUG/MINOR: sched: properly cover for a rare MT_LIST_ADDQ() race
    - MINOR: mux-h1: avoid taking the toremove_lock in on dying tasks
    - MINOR: mux-h2: avoid taking the toremove_lock in on dying tasks
    - MINOR: mux-fcgi: avoid taking the toremove_lock in on dying tasks
    - MINOR: pools: increase MAX_BASE_POOLS to 64
    - DOC: ssl: add "allow-0rtt" and "ciphersuites" in crt-list
    - BUG/MEDIUM: pattern: Add a trailing \0 to match strings only if possible
    - BUG/MEDIUM: log-format: fix possible endless loop in parse_logformat_string()
    - BUG/MINOR: proxy: fix dump_server_state()'s misuse of the trash
    - BUG/MINOR: proxy: always initialize the trash in show servers state
    - MINOR: cli/proxy: add a new "show servers conn" command
    - MINOR: server: skip servers with no idle conns earlier
    - BUG/MINOR: server: fix the connection release logic regarding nearly full conditions
    - MEDIUM: server: add a new pool-low-conn server setting
    - BUG/MEDIUM: backend: always search in the safe list after failing on the idle one
    - MINOR: backend: don't always takeover from the same threads
    - MINOR: sched: make sched->task_list_size atomic
    - MEDIUM: sched: create a new TASK_KILLED task flag
    - MEDIUM: sched: implement task_kill() to kill a task
    - MEDIUM: mux-h1: use task_kill() during h1_takeover() instead of task_wakeup()
    - MEDIUM: mux-h2: use task_kill() during h2_takeover() instead of task_wakeup()
    - MEDIUM: mux-fcgi: use task_kill() during fcgi_takeover() instead of task_wakeup()
    - MINOR: list: Add MT_LIST_DEL_SAFE_NOINIT() and MT_LIST_ADDQ_NOCHECK()
    - CLEANUP: connections: rename the toremove_lock to takeover_lock
    - MEDIUM: connections: Don't use a lock when moving connections to remove.
    - DOC: configuration: add missing index entries for tune.pool-{low,high}-fd-ratio
    - DOC: configuration: fix alphabetical ordering for tune.pool-{high,low}-fd-ratio
    - MINOR: config: add a new tune.idle-pool.shared global setting.
    - MINOR: 51d: silence a warning about null pointer dereference
    - MINOR: debug: add a new "debug dev memstats" command
    - MINOR: log-format: allow to preserve spacing in log format strings
    - BUILD: debug: avoid build warnings with DEBUG_MEM_STATS
    - BUG/MAJOR: sched: make sure task_kill() always queues the task
    - BUG/MEDIUM: muxes: Make sure nobody stole the connection before using it.
    - BUG/MEDIUM: cli/proxy: don't try to dump idle connection state if there's none
    - BUILD: haproxy: fix build error when RLIMIT_AS is not set
    - BUG/MAJOR: sched: make it work also when not building with DEBUG_STRICT
    - MINOR: log: add time second fraction field to rfc5424 log timestamp.
    - BUG/MINOR: log: missing timezone on iso dates.
    - BUG/MEDIUM: server: don't kill all idle conns when there are not enough
    - MINOR: sched: split tasklet_wakeup() into tasklet_wakeup_on()
    - BUG/MEDIUM: connections: Set the tid for the old tasklet on takeover.
    - BUG/MEDIUM: connections: Let the xprt layer know a takeover happened.
    - BUG/MINOR: http_act: don't check capture id in backend (2)
    - BUILD: makefile: disable threads by default on OpenBSD
    - BUILD: peers: fix build warning with gcc 4.2.1
    - CI: cirrus-ci: exclude slow reg-tests
2020-07-04 07:10:24 +02:00
Ilya Shipitsin
95d8af76c8 CI: cirrus-ci: exclude slow reg-tests
slow tests are racy, we already skip them on travis-ci, let us do the
same in cirrus-ci
2020-07-04 06:58:14 +02:00
Willy Tarreau
a898f0c6ff BUILD: peers: fix build warning with gcc 4.2.1
Building on OpenBSD 6.7 with gcc-4.2.1 yields the following warnings
which suggest that the initialization is not taken as expected but
that the container member is reset with each initialization:

  src/peers.c: In function 'peer_send_updatemsg':
  src/peers.c:1000: warning: initialized field overwritten
  src/peers.c:1000: warning: (near initialization for 'p.updt')
  src/peers.c:1001: warning: initialized field overwritten
  src/peers.c:1001: warning: (near initialization for 'p.updt')
  src/peers.c:1002: warning: initialized field overwritten
  src/peers.c:1002: warning: (near initialization for 'p.updt')
  src/peers.c:1003: warning: initialized field overwritten
  src/peers.c:1003: warning: (near initialization for 'p.updt')
  src/peers.c:1004: warning: initialized field overwritten
  src/peers.c:1004: warning: (near initialization for 'p.updt')

Fixing this is trivial, we just have to initialize one level at
a time.
2020-07-03 19:09:29 +02:00