Given the following example configuration:
frontend foo
bind *:8080
mode http
http-request set-var(txn.foo) str(bar)
Running a configuration check within valgrind reports:
==23665== Memcheck, a memory error detector
==23665== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==23665== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==23665== Command: ./haproxy -c -f ./crasher.cfg
==23665==
[WARNING] 165/002941 (23665) : config : missing timeouts for frontend 'foo'.
| While not properly invalid, you will certainly encounter various problems
| with such a configuration. To fix this, please ensure that all following
| timeouts are set to a non-zero value: 'client', 'connect', 'server'.
Warnings were found.
Configuration file is valid
==23665==
==23665== HEAP SUMMARY:
==23665== in use at exit: 314,008 bytes in 87 blocks
==23665== total heap usage: 160 allocs, 73 frees, 1,448,074 bytes allocated
==23665==
==23665== 132 (48 direct, 84 indirect) bytes in 1 blocks are definitely lost in loss record 15 of 28
==23665== at 0x4C2FB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==23665== by 0x4A2612: sample_parse_expr (sample.c:876)
==23665== by 0x54DF84: parse_store (vars.c:766)
==23665== by 0x528BDF: parse_http_req_cond (http_rules.c:95)
==23665== by 0x469F36: cfg_parse_listen (cfgparse-listen.c:1339)
==23665== by 0x459E33: readcfgfile (cfgparse.c:2167)
==23665== by 0x5074FD: init (haproxy.c:2021)
==23665== by 0x418262: main (haproxy.c:3126)
==23665==
==23665== LEAK SUMMARY:
==23665== definitely lost: 48 bytes in 1 blocks
==23665== indirectly lost: 84 bytes in 2 blocks
==23665== possibly lost: 0 bytes in 0 blocks
==23665== still reachable: 313,876 bytes in 84 blocks
==23665== suppressed: 0 bytes in 0 blocks
==23665== Reachable blocks (those to which a pointer was found) are not shown.
==23665== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==23665==
==23665== For counts of detected and suppressed errors, rerun with: -v
==23665== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
After this patch is applied the leak is gone as expected.
This is a very minor leak that can only be observed if deinit() is called,
shortly before the OS will free all memory of the process anyway. No
backport needed.
Particularly cleanly deinit() after a configuration check to clean up the
output of valgrind which reports "possible losses" without a deinit() and
does not with a deinit(), converting actual losses into proper hard losses
which makes the whole stuff easier to analyze.
As an example, given an example configuration of the following:
frontend foo
bind *:8080
mode http
Running `haproxy -c -f cfg` within valgrind will report 4 possible losses:
$ valgrind --leak-check=full ./haproxy -c -f ./example.cfg
==21219== Memcheck, a memory error detector
==21219== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==21219== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==21219== Command: ./haproxy -c -f ./example.cfg
==21219==
[WARNING] 165/001100 (21219) : config : missing timeouts for frontend 'foo'.
| While not properly invalid, you will certainly encounter various problems
| with such a configuration. To fix this, please ensure that all following
| timeouts are set to a non-zero value: 'client', 'connect', 'server'.
Warnings were found.
Configuration file is valid
==21219==
==21219== HEAP SUMMARY:
==21219== in use at exit: 1,436,631 bytes in 130 blocks
==21219== total heap usage: 153 allocs, 23 frees, 1,447,758 bytes allocated
==21219==
==21219== 7 bytes in 1 blocks are possibly lost in loss record 5 of 54
==21219== at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==21219== by 0x5726489: strdup (strdup.c:42)
==21219== by 0x468FD9: bind_conf_alloc (listener.h:158)
==21219== by 0x468FD9: cfg_parse_listen (cfgparse-listen.c:557)
==21219== by 0x459DF3: readcfgfile (cfgparse.c:2167)
==21219== by 0x5056CD: init (haproxy.c:2021)
==21219== by 0x418232: main (haproxy.c:3121)
==21219==
==21219== 14 bytes in 1 blocks are possibly lost in loss record 9 of 54
==21219== at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==21219== by 0x5726489: strdup (strdup.c:42)
==21219== by 0x468F9B: bind_conf_alloc (listener.h:154)
==21219== by 0x468F9B: cfg_parse_listen (cfgparse-listen.c:557)
==21219== by 0x459DF3: readcfgfile (cfgparse.c:2167)
==21219== by 0x5056CD: init (haproxy.c:2021)
==21219== by 0x418232: main (haproxy.c:3121)
==21219==
==21219== 128 bytes in 1 blocks are possibly lost in loss record 35 of 54
==21219== at 0x4C2FB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==21219== by 0x468F90: bind_conf_alloc (listener.h:152)
==21219== by 0x468F90: cfg_parse_listen (cfgparse-listen.c:557)
==21219== by 0x459DF3: readcfgfile (cfgparse.c:2167)
==21219== by 0x5056CD: init (haproxy.c:2021)
==21219== by 0x418232: main (haproxy.c:3121)
==21219==
==21219== 608 bytes in 1 blocks are possibly lost in loss record 46 of 54
==21219== at 0x4C2FB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==21219== by 0x4B953A: create_listeners (listener.c:576)
==21219== by 0x4578F6: str2listener (cfgparse.c:192)
==21219== by 0x469039: cfg_parse_listen (cfgparse-listen.c:568)
==21219== by 0x459DF3: readcfgfile (cfgparse.c:2167)
==21219== by 0x5056CD: init (haproxy.c:2021)
==21219== by 0x418232: main (haproxy.c:3121)
==21219==
==21219== LEAK SUMMARY:
==21219== definitely lost: 0 bytes in 0 blocks
==21219== indirectly lost: 0 bytes in 0 blocks
==21219== possibly lost: 757 bytes in 4 blocks
==21219== still reachable: 1,435,874 bytes in 126 blocks
==21219== suppressed: 0 bytes in 0 blocks
==21219== Reachable blocks (those to which a pointer was found) are not shown.
==21219== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==21219==
==21219== For counts of detected and suppressed errors, rerun with: -v
==21219== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 0 from 0)
Re-running the same command with the patch applied will not report any
losses any more:
$ valgrind --leak-check=full ./haproxy -c -f ./example.cfg
==22124== Memcheck, a memory error detector
==22124== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==22124== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==22124== Command: ./haproxy -c -f ./example.cfg
==22124==
[WARNING] 165/001503 (22124) : config : missing timeouts for frontend 'foo'.
| While not properly invalid, you will certainly encounter various problems
| with such a configuration. To fix this, please ensure that all following
| timeouts are set to a non-zero value: 'client', 'connect', 'server'.
Warnings were found.
Configuration file is valid
==22124==
==22124== HEAP SUMMARY:
==22124== in use at exit: 313,864 bytes in 82 blocks
==22124== total heap usage: 153 allocs, 71 frees, 1,447,758 bytes allocated
==22124==
==22124== LEAK SUMMARY:
==22124== definitely lost: 0 bytes in 0 blocks
==22124== indirectly lost: 0 bytes in 0 blocks
==22124== possibly lost: 0 bytes in 0 blocks
==22124== still reachable: 313,864 bytes in 82 blocks
==22124== suppressed: 0 bytes in 0 blocks
==22124== Reachable blocks (those to which a pointer was found) are not shown.
==22124== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==22124==
==22124== For counts of detected and suppressed errors, rerun with: -v
==22124== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
It might be worth investigating what exactly HAProxy does to lose pointers
to the start of those 4 memory areas and then to be able to still free them
during deinit(). If HAProxy is able to free them, they ideally should be
"still reachable" and not "possibly lost".
The allocation did not account for either the trailing null byte or the
space, leading to a buffer overwrite.
This bug was detected by an assertion failure in the allocator. But can
also be easily detected using valgrind:
==25827== Invalid write of size 1
==25827== at 0x6529759: __vsprintf_chk (vsprintf_chk.c:84)
==25827== by 0x65296AC: __sprintf_chk (sprintf_chk.c:31)
==25827== by 0x4D6AB7: sprintf (stdio2.h:33)
==25827== by 0x4D6AB7: proxy_parse_smtpchk_opt (check.c:1799)
==25827== by 0x4A7DDD: cfg_parse_listen (cfgparse-listen.c:2269)
==25827== by 0x494AD3: readcfgfile (cfgparse.c:2167)
==25827== by 0x542995: init (haproxy.c:2021)
==25827== by 0x421DD2: main (haproxy.c:3121)
==25827== Address 0x78712a8 is 0 bytes after a block of size 24 alloc'd
==25827== at 0x4C2FB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==25827== by 0x4D6A8C: proxy_parse_smtpchk_opt (check.c:1797)
==25827== by 0x4A7DDD: cfg_parse_listen (cfgparse-listen.c:2269)
==25827== by 0x494AD3: readcfgfile (cfgparse.c:2167)
==25827== by 0x542995: init (haproxy.c:2021)
==25827== by 0x421DD2: main (haproxy.c:3121)
This patch fixes issue #681.
This bug was introduced in commit fbcc77c6ba,
which first appeared in 2.2-dev7. No backport needed.
When building with gcc-8 -fsanitize=address, we get this warning once on
an strncpy() call in proto_uxst.c:
src/proto_uxst.c:262:3: warning: 'strncpy' output may be truncated copying 107 bytes from a string of length 4095 [-Wstringop-truncation]
strncpy(addr.sun_path, tempname, sizeof(addr.sun_path) - 1);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It happens despite the test on snprintf() at the top (since gcc's string
handling is totally empiric), and requires the strlen() test to be placed
"very close" to the strncpy() call (with "very close" yet to be determined).
There's no other way to shut this one except disabling it. Given there's
only one instance of this warning and the cost of dealing with it in the
code is not huge, let's decorate the code to make gcc happily believe it
is smart since it seems to have a mind of itself.
clang just failed on fd.c with this error:
src/fd.c:491:9: error: logical not is only applied to the left hand side of this comparison [-Werror,-Wlogical-not-parentheses]
while (HA_SPIN_TRYLOCK(OTHER_LOCK, &log_lock) != 0) {
^ ~~
That's because this expands to this:
while (!pl_try_s(&log_lock) != 0) {
Let's just add parenthesis in the TRYLOCK macros to avoid this.
This may need to be backported if commit df187875d ("BUG/MEDIUM: log:
don't hold the log lock during writev() on a file descriptor") is
backported as well as it seems to be the first one to trigger it.
In bug #676, it was reported that ssl-min-ver SSLv3 does not work in
Amazon environments with OpenSSL 1.0.2.
The reason for this is a patch of Amazon OpenSSL which sets
SSL_OP_NO_SSLv3 in SSL_CTX_new(). Which is kind of a problem with our
implementation of ssl-{min,max}-ver in old openSSL versions, because it
does not try to clear existing version flags.
This patch fixes the bug by cleaning versions flags known by HAProxy in
the SSL_CTX before applying the right ones.
Should be backported as far as 1.8.
Getting rid of this warning is cleaner solved using a 'fall through' comment,
because it clarifies intent to a human reader.
This patch adjust a few places that cause -Wimplicit-fallthrough to trigger:
- Fix typos in the comment.
- Remove redundant 'no break' that trips up gcc from comment.
- Move the comment out of the block when the 'case' is completely surrounded
by braces.
- Add comments where I could determine that the fall through was intentional.
Changes tested on
gcc (Debian 9.3.0-13) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
using
make -j4 all TARGET=linux-glibc USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_GETADDRINFO=1
Commit b5997f740 ("MAJOR: threads/map: Make acls/maps thread safe")
introduced a subtle bug in the pattern matching code. In order to cope
with the possibility that another thread might be modifying the pattern's
sample_data while it's being used, we return a thread-local static
sample_data which is a copy of the one found in the matched pattern. The
copy is performed depending on the sample_data's type. But the switch
statement misses some breaks and doesn't set the new sample_data pointer
at the right place, resulting in the original sample_data being restored
at the end before returning.
The net effect overall is that the correct sample_data is returned (hence
functionally speaking the matching works fine) but it's not thread-safe
so any del_map() or set_map() action could modify the pattern on one
thread while it's being used on another one. It doesn't seem likely to
cause a crash but could result in corrupted data appearing where the
value is consumed (e.g. when appended in a header or when logged) or an
ACL occasionally not matching after a map lookup.
This fix should be backported as far as 1.8.
Thanks to Tim for reporting it and to Emeric for the analysis.
Tested with
make -j4 all TARGET=linux-glibc USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_GETADDRINFO=1
against
gcc (Debian 9.3.0-13) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
In issue #648 a second problem was reported, indicating that some users
mistakenly send the log to an FD mapped on a file. This situation doesn't
even enable O_NONBLOCK and results in huge access times in the order of
milliseconds with the lock held and other threads waiting till the
watchdog fires to unblock the situation.
The problem with files is that O_NONBLOCK is ignored, and we still need
to lock otherwise we can end up with interleaved log messages.
What this patch does is different. Instead of locking all writers, it
uses a trylock so that there's always at most one logger and that other
candidates can simply give up and report a failure, just as would happen
if writev() returned -1 due to a pipe full condition. This solution is
elegant because it gives back the control to haproxy to decide to give
up when it takes too much time, while previously it was the kernel that
used to block the syscall.
However at high log rates (500000 req/s) there was up to 50% dropped logs
due to the contention on the lock. In order to address this, we try to
grab the lock up to 200 times and call ha_thread_relax() on failure. This
results in almost no failure (no more than previously with O_NONBLOCK). A
typical test with 6 competing threads writing to stdout chained to a pipe
to a single process shows around 1000 drops for 10 million logs at 480000
lines per second.
Please note that this doesn't mean that writing to a blocking FD is a good
idea, and it might only be temporarily done on testing environments for
debugging. A file or a terminal will continue to block the writing thread
while others spin a little bit and lose their logs, but the writing thread
will still experience performance-killing latencies.
This patch should be backported to 2.1 and 2.0. The code is in log.c in
2.0, but the principle is the same.
Apparently Cygwin requires sys/types.h before netinet/tcp.h but doesn't
include it by itself, as shown here:
https://github.com/haproxy/haproxy/actions/runs/131943890
This patch makes sure it's always present, which is in server.c and
the SPOA example.
Released version 2.2-dev9 with the following main changes :
- BUG/MINOR: http-htx: Don't forget to release the http reply in release function
- BUG/MINOR: http-htx: Fix a leak on error path during http reply parsing
- MINOR: checks: Remove dead code from process_chk_conn()
- REGTESTS: checks: Fix tls_health_checks when IPv6 addresses are used
- REGTESTS: Add missing OPENSSL to REQUIRE_OPTIONS for lua/txn_get_priv
- MINOR: lua: Use vars_unset_by_name_ifexist()
- CLEANUP: vars: Remove void vars_unset_by_name(const char*, size_t, struct sample*)
- MINOR: vars: Make vars_(un|)set_by_name(_ifexist|) return a success value
- MINOR: lua: Make `set_var()` and `unset_var()` return success
- MEDIUM: lua: Add `ifexist` parameter to `set_var`
- MEDIUM: ring: new section ring to declare custom ring buffers.
- REGTESTS: Add missing OPENSSL to REQUIRE_OPTIONS for compression/lua_validation
- REGTESTS: Require the version 2.2 to execute lua/set_var
- BUG/MEDIUM: checks: Refresh the conn-stream and the connection after a connect
- MINOR: checks: Remove useless tests on the connection and conn-stream
- BUG/MEDIUM: contrib/spoa: do not register python3.8 if --embed fail
- BUG/MEDIUM: connection: Ignore PP2 unique ID for stream-less connections
- BUG/MINOR: connection: Always get the stream when available to send PP2 line
- BUG/MEDIUM: backend: set the connection owner to the session when using alpn.
- MINOR: pools: compute an estimate of each pool's average needed objects
- MEDIUM: pools: directly free objects when pools are too much crowded
- REGTEST: Add connection/proxy_protocol_send_unique_id_alpn
- MINOR: http-ana: Make the function http_reply_to_htx() public
- MINOR: http-ana: Use proxy's error replies to emit 401/407 responses
- MINOR: http-rules: Use an action function to eval http-request auth rules
- CLEANUP: http: Remove unused HTTP message templates
- BUG/MEDIUM: checks: Don't blindly subscribe for receive if waiting for connect
- MINOR: checks: I/O callback function only rely on the data layer wake callback
- BUG/MINOR: lua: Add missing string length for lua sticktable lookup
- BUG/MEDIUM: logs: fix trailing zeros on log message.
- CI: cirrus-ci: skip reg-tests/connection/proxy_protocol_send_unique_id_alpn.vtc on CentOS 6
- BUG/MINOR: nameservers: fix error handling in parsing of resolv.conf
- BUG/MEDIUM: checks: Don't add a tcpcheck ruleset twice in the shared tree
- MEDIUM: ssl: use TLSv1.2 as the minimum default on bind lines
- CLEANUP: pools: use the regular lock for the flush operation on lockless pools
- SCRIPTS: publish-release: pass -n to gzip to remove timestamp
- MINOR: ring: re-work ring attach generic API.
- BUG/MINOR: error on unknown statement in ring section.
- MEDIUM: ring: add server statement to forward messages from a ring
- MEDIUM: ring: add new srv statement to support octet counting forward
- MINOR: ssl: set ssl-min-ver in ambiguous configurations
- CLEANUP: ssl: remove comment from dump_crtlist_sslconf()
- BUILD: sink: address build warning on 32-bit architectures
- BUG/MINOR: peers: fix internal/network key type mapping.
- CLEANUP: regex: remove outdated support for regex actions
- Revert "MINOR: ssl: rework add cert chain to CTX to be libssl independent"
- MINOR: mux-h1/proxy: Add a proxy option to disable clear h2 upgrade
- BUG/MEDIUM: lua: Reset analyse expiration timeout before executing a lua action
- DOC: add a line about comments in crt-list
- BUG/MEDIUM: hlua: Lock pattern references to perform set/add/del operations
- BUG/MINOR: checks: Fix test on http-check rulesets during config validity check
- BUG/MEDIUM: contrib/prometheus-exporter: Properly set flags to dump metrics
- BUG/MEDIUM: mworker: fix the copy of options in copy_argv()
- BUG/MINOR: init: -x can have a parameter starting with a dash
- BUG/MINOR: init: -S can have a parameter starting with a dash
- BUG/MEDIUM: mworker: fix the reload with an -- option
- BUG/MINOR: ssl: fix a trash buffer leak in some error cases
- BUG/MINOR: mworker: fix a memleak when execvp() failed
- MINOR: sample: Add secure_memcmp converter
- REORG: ebtree: move the C files from ebtree/ to src/
- REORG: ebtree: move the include files from ebtree to include/import/
- REORG: ebtree: clean up remains of the ebtree/ directory
- REORG: include: create new file haproxy/api-t.h
- REORG: include: create new file haproxy/api.h
- REORG: include: update all files to use haproxy/api.h or api-t.h if needed
- CLEANUP: include: remove common/config.h
- CLEANUP: include: remove unused template.h
- REORG: include: move MIN/MAX from tools.h to compat.h
- REORG: include: move SWAP/MID_RANGE/MAX_RANGE from tools.h to standard.h
- CLEANUP: include: remove unused common/tools.h
- REORG: include: move the base files from common/ to haproxy/
- REORG: include: move version.h to haproxy/
- REORG: include: move base64.h, errors.h and hash.h from common to to haproxy/
- REORG: include: move openssl-compat.h from common/ to haproxy/
- REORG: include: move ist.h from common/ to import/
- REORG: include: move the BUG_ON() code to haproxy/bug.h
- REORG: include: move debug.h from common/ to haproxy/
- CLEANUP: debug: drop unused function p_malloc()
- REORG: include: split buf.h into haproxy/buf-t.h and haproxy/buf.h
- REORG: include: move istbuf.h to haproxy/
- REORG: include: split mini-clist into haproxy/list and list-t.h
- REORG: threads: extract atomic ops from hathreads.h
- CLEANUP: threads: remove a few needless includes of hathreads.h
- REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h
- CLEANUP: thread: rename __decl_hathreads() to __decl_thread()
- REORG: include: move time.h from common/ to haproxy/
- REORG: include: move integer manipulation functions from standard.h to intops.h
- CLEANUP: include: remove excessive includes of common/standard.h
- REORG: include: move freq_ctr to haproxy/
- CLEANUP: pool: include freq_ctr.h and remove locally duplicated functions
- REORG: memory: move the pool type definitions to haproxy/pool-t.h
- REORG: memory: move the OS-level allocator to haproxy/pool-os.h
- MINOR: memory: don't let __pool_get_first() pick from the cache
- MEDIUM: memory: don't let pool_put_to_cache() free the objects itself
- MINOR: memory: move pool-specific path of the locked pool_free() to __pool_free()
- MEDIUM: memory: make local pools independent on lockless pools
- REORG: include: move common/memory.h to haproxy/pool.h
- REORG: include: move common/chunk.h to haproxy/chunk.h
- REORG: include: move activity to haproxy/
- REORG: include: move common/buffer.h to haproxy/dynbuf{,-t}.h
- REORG: include: move common/net_helper.h to haproxy/net_helper.h
- REORG: include: move common/namespace.h to haproxy/namespace{,-t}.h
- REORG: include: split common/regex.h into haproxy/regex{,-t}.h
- REORG: include: split common/xref.h into haproxy/xref{,-t}.h
- REORG: include: move common/ticks.h to haproxy/ticks.h
- REORG: include: split common/http.h into haproxy/http{,-t}.h
- REORG: include: split common/http-hdr.h into haproxy/http-hdr{,-t}.h
- REORG: include: move common/h1.h to haproxy/h1.h
- REORG: include: split common/htx.h into haproxy/htx{,-t}.h
- REORG: include: move hpack*.h to haproxy/ and split hpack-tbl
- REORG: include: move common/h2.h to haproxy/h2.h
- REORG: include: move common/fcgi.h to haproxy/
- REORG: include: move protocol.h to haproxy/protocol{,-t}.h
- REORG: tools: split common/standard.h into haproxy/tools{,-t}.h
- REORG: include: move dict.h to hparoxy/dict{,-t}.h
- REORG: include: move shctx to haproxy/shctx{,-t}.h
- REORG: include: move port_range.h to haproxy/port_range{,-t}.h
- REORG: include: move fd.h to haproxy/fd{,-t}.h
- REORG: include: move ring to haproxy/ring{,-t}.h
- REORG: include: move sink.h to haproxy/sink{,-t}.h
- REORG: include: move pipe.h to haproxy/pipe{,-t}.h
- CLEANUP: include: remove empty raw_sock.h
- REORG: include: move proto_udp.h to haproxy/proto_udp{,-t}.h
- REORG: include: move proto/proto_sockpair.h to haproxy/proto_sockpair.h
- REORG: include: move compression.h to haproxy/compression{,-t}.h
- REORG: include: move h1_htx.h to haproxy/h1_htx.h
- REORG: include: move http_htx.h to haproxy/http_htx{,-t}.h
- REORG: include: move hlua.h to haproxy/hlua{,-t}.h
- REORG: include: move hlua_fcn.h to haproxy/hlua_fcn.h
- REORG: include: move action.h to haproxy/action{,-t}.h
- REORG: include: move arg.h to haproxy/arg{,-t}.h
- REORG: include: move auth.h to haproxy/auth{,-t}.h
- REORG: include: move dns.h to haproxy/dns{,-t}.h
- REORG: include: move flt_http_comp.h to haproxy/
- REORG: include: move counters.h to haproxy/counters-t.h
- REORG: include: split mailers.h into haproxy/mailers{,-t}.h
- REORG: include: move capture.h to haproxy/capture{,-t}.h
- REORG: include: move frontend.h to haproxy/frontend.h
- REORG: include: move obj_type.h to haproxy/obj_type{,-t}.h
- REORG: include: move http_rules.h to haproxy/http_rules.h
- CLEANUP: include: remove unused mux_pt.h
- REORG: include: move mworker.h to haproxy/mworker{,-t}.h
- REORG: include: move ssl_utils.h to haproxy/ssl_utils.h
- REORG: include: move ssl_ckch.h to haproxy/ssl_ckch{,-t}.h
- REORG: move ssl_crtlist.h to haproxy/ssl_crtlist{,-t}.h
- REORG: include: move lb_chash.h to haproxy/lb_chash{,-t}.h
- REORG: include: move lb_fas.h to haproxy/lb_fas{,-t}.h
- REORG: include: move lb_fwlc.h to haproxy/lb_fwlc{,-t}.h
- REORG: include: move lb_fwrr.h to haproxy/lb_fwrr{,-t}.h
- REORG: include: move listener.h to haproxy/listener{,-t}.h
- REORG: include: move pattern.h to haproxy/pattern{,-t}.h
- REORG: include: move map to haproxy/map{,-t}.h
- REORG: include: move payload.h to haproxy/payload.h
- REORG: include: move sample.h to haproxy/sample{,-t}.h
- REORG: include: move protocol_buffers.h to haproxy/protobuf{,-t}.h
- REORG: include: move vars.h to haproxy/vars{,-t}.h
- REORG: include: split global.h into haproxy/global{,-t}.h
- REORG: include: move task.h to haproxy/task{,-t}.h
- REORG: include: move proto_tcp.h to haproxy/proto_tcp.h
- REORG: include: move signal.h to haproxy/signal{,-t}.h
- REORG: include: move tcp_rules.h to haproxy/tcp_rules.h
- REORG: include: move connection.h to haproxy/connection{,-t}.h
- REORG: include: move checks.h to haproxy/check{,-t}.h
- REORG: include: move http_fetch.h to haproxy/http_fetch.h
- REORG: include: move peers.h to haproxy/peers{,-t}.h
- REORG: include: move stick_table.h to haproxy/stick_table{,-t}.h
- REORG: include: move session.h to haproxy/session{,-t}.h
- REORG: include: move trace.h to haproxy/trace{,-t}.h
- REORG: include: move acl.h to haproxy/acl.h{,-t}.h
- REORG: include: split common/uri_auth.h into haproxy/uri_auth{,-t}.h
- REORG: move applet.h to haproxy/applet{,-t}.h
- REORG: include: move stats.h to haproxy/stats{,-t}.h
- REORG: include: move cli.h to haproxy/cli{,-t}.h
- REORG: include: move lb_map.h to haproxy/lb_map{,-t}.h
- REORG: include: move ssl_sock.h to haproxy/ssl_sock{,-t}.h
- REORG: include: move stream_interface.h to haproxy/stream_interface{,-t}.h
- REORG: include: move channel.h to haproxy/channel{,-t}.h
- REORG: include: move http_ana.h to haproxy/http_ana{,-t}.h
- REORG: include: move filters.h to haproxy/filters{,-t}.h
- REORG: include: move fcgi-app.h to haproxy/fcgi-app{,-t}.h
- REORG: include: move log.h to haproxy/log{,-t}.h
- REORG: include: move proxy.h to haproxy/proxy{,-t}.h
- REORG: include: move spoe.h to haproxy/spoe{,-t}.h
- REORG: include: move backend.h to haproxy/backend{,-t}.h
- REORG: include: move queue.h to haproxy/queue{,-t}.h
- REORG: include: move server.h to haproxy/server{,-t}.h
- REORG: include: move stream.h to haproxy/stream{,-t}.h
- REORG: include: move cfgparse.h to haproxy/cfgparse.h
- CLEANUP: hpack: export debug functions and move inlines to .h
- REORG: check: move the e-mail alerting code to mailers.c
- REORG: check: move tcpchecks away from check.c
- REORG: check: move email_alert* from proxy-t.h to mailers-t.h
- REORG: check: extract the external checks from check.{c,h}
- CLEANUP: include: don't include stddef.h directly
- CLEANUP: include: don't include proxy-t.h in global-t.h
- CLEANUP: include: move sample_data out of sample-t.h
- REORG: include: move the error reporting functions to from log.h to errors.h
- BUILD: reorder objects in the Makefile for faster builds
- CLEANUP: compiler: add a THREAD_ALIGNED macro and use it where appropriate
- CLEANUP: include: make atomic.h part of the base API
- REORG: include: move MAX_THREADS to defaults.h
- REORG: include: move THREAD_LOCAL and __decl_thread() to compiler.h
- CLEANUP: include: tree-wide alphabetical sort of include files
- REORG: include: make list-t.h part of the base API
- REORG: dgram: rename proto_udp to dgram
The set of files proto_udp.{c,h} were misleadingly named, as they do not
provide anything related to the UDP protocol but to datagram handling
instead, since currently all UDP processing is hard-coded where it's used
(dns, logs). They are to UDP what connection.{c,h} are to proto_tcp. This
was causing confusion about how to insert UDP socket management code,
so let's rename them right now to dgram.{c,h} which more accurately
matches what's inside since every function and type is already prefixed
with "dgram_".
There are list definitions everywhere in the code, let's drop the need
for including list-t.h to declare them. The rest of the list manipulation
is huge however and not needed everywhere so using the list walking macros
still requires to include list.h.
This patch fixes all the leftovers from the include cleanup campaign. There
were not that many (~400 entries in ~150 files) but it was definitely worth
doing it as it revealed a few duplicates.
Since these are used as type attributes or conditional clauses, they
are used about everywhere and should not require a dependency on
thread.h. Moving them to compiler.h along with other similar statements
like ALIGN() etc looks more logical; this way they become part of the
base API. This allowed to remove thread-t.h from ~12 files, one was
found to only require thread-t and not thread and dict.c was found to
require thread.h.
That's already where MAX_PROCS is set, and we already handle the case of
the default value so there is no reason for placing it in thread.h given
that most call places don't need the rest of the threads definitions. The
include was removed from global-t.h and activity.c.
Sometimes we need to align a struct member or a struct's size only when
threads are enabled. This is the case on fdtab for example. Instead of
using ugly ifdefs in the code itself, let's have a THREAD_ALIGNED() macro
performing the alignment only when threads are enabled. For now this was
only applied to fd-t.h as it was the only place found.
Splitting large files and changing includes has changed the per-file
build time. After a careful reordering based on build time, we're now
down to 5.8s at -O0 on the PC at -j8 and 2.4-2.6s on the farm at -j120.
Some room for at least one file name was left on each line to ease
future additions.
Most of the files dealing with error reports have to include log.h in order
to access ha_alert(), ha_warning() etc. But while these functions don't
depend on anything, log.h depends on a lot of stuff because it deals with
log-formats and samples. As a result it's impossible not to embark long
dependencies when using ha_warning() or qfprintf().
This patch moves these low-level functions to errors.h, which already
defines the error codes used at the same places. About half of the users
of log.h could be adjusted, sometimes revealing other issues such as
missing tools.h. Interestingly the total preprocessed size shrunk by
4%.
The struct sample_data is used by pattern, map and vars, and currently
requires to include sample-t which comes with many other dependencies.
Let's move sample_data into its own file to shorten the dependency tree.
This revealed a number of issues in adjacent files which were hidden by
the fact that sample-t.h brought everything that was missing.
Directly including stddef.h in many files results in it being processed
multiple times while it can be centralized in api-t.h and be guarded
against multiple inclusions. Doing so reduces the number of preprocessed
lines by 1200!
Checks.c remains one of the largest file of the project and it contains
too many things. The tcpchecks code represents half of this file, and
both parts are relatively isolated, so let's move it away into its own
file. We now have tcpcheck.c, tcpcheck{,-t}.h.
Doing so required to export quite a number of functions because check.c
has almost everything made static, which really doesn't help to split!
check.c is one of the largest file and contains too many things. The
e-mail alerting code is stored there while nothing is in mailers.c.
Let's move this code out. That's only 4% of the code but a good start.
In order to do so, a few tcp-check functions had to be exported.
When building contrib/hpack there is a warning about an unused static
function. Actually it makes no sense to make it static, instead it must
be regularly exported. Similarly there is hpack_dht_get_tail() which is
inlined in the C file and which would make more sense with all other ones
in the H file.
There's no point splitting the file in two since only cfgparse uses the
types defined there. A few call places were updated and cleaned up. All
of them were in C files which register keywords.
There is nothing left in common/ now so this directory must not be used
anymore.
This one was not easy because it was embarking many includes with it,
which other files would automatically find. At least global.h, arg.h
and tools.h were identified. 93 total locations were identified, 8
additional includes had to be added.
In the rare files where it was possible to finalize the sorting of
includes by adjusting only one or two extra lines, it was done. But
all files would need to be rechecked and cleaned up now.
It was the last set of files in types/ and proto/ and these directories
must not be reused anymore.
extern struct dict server_name_dict was moved from the type file to the
main file. A handful of inlined functions were moved at the bottom of
the file. Call places were updated to use server-t.h when relevant, or
to simply drop the entry when not needed.
The files remained mostly unchanged since they were OK. However, half of
the users didn't need to include them, and about as many actually needed
to have it and used to find functions like srv_currently_usable() through
a long chain that broke when moving the file.
This one is particularly difficult to split because it provides all the
functions used to manipulate a proxy state and to retrieve names or IDs
for error reporting, and as such, it was included in 73 files (down to
68 after cleanup). It would deserve a small cleanup though the cut points
are not obvious at the moment given the number of structs involved in
the struct proxy itself.
The current state of the logging is a real mess. The main problem is
that almost all files include log.h just in order to have access to
the alert/warning functions like ha_alert() etc, and don't care about
logs. But log.h also deals with real logging as well as log-format and
depends on stream.h and various other things. As such it forces a few
heavy files like stream.h to be loaded early and to hide missing
dependencies depending where it's loaded. Among the missing ones is
syslog.h which was often automatically included resulting in no less
than 3 users missing it.
Among 76 users, only 5 could be removed, and probably 70 don't need the
full set of dependencies.
A good approach would consist in splitting that file in 3 parts:
- one for error output ("errors" ?).
- one for log_format processing
- and one for actual logging.
It was moved without any change, however many callers didn't need it at
all. This was a consequence of the split of proto_http.c into several
parts that resulted in many locations to still reference it.
Almost no change except moving the cli_kw struct definition after the
defines. Almost all users had both types&proto included, which is not
surprizing since this code is old and it used to be the norm a decade
ago. These places were cleaned.
Just some minor reordering, and the usual cleanup of call places for
those which didn't need it. We don't include the whole tools.h into
stats-t anymore but just tools-t.h.
The type file was slightly tidied. The cli-specific APPCTX_CLI_ST1_* flag
definitions were moved to cli.h. The type file was adjusted to include
buf-t.h and not the huge buf.h. A few call places were fixed because they
did not need this include.
Initially it looked like this could have been placed into auth.h or
stats.h but it's not the case as it's what makes the link between them
and the HTTP layer. However the file needed to be split in two. Quite
a number of call places were dropped because these were mostly leftovers
from the early days where the stats and cli were packed together.
