Commit Graph

622 Commits

Author SHA1 Message Date
Aurelien DARRAGON
ab9efc25f0 BUG/MINOR: log: fix parse_log_message rfc5424 size check
In parse_log_message(), if log is rfc5424 compliant, p pointer
is incremented and size is not. However size is still used in further
checks as if p pointer was not incremented.

This could lead to logic error or buffer overflow if input buf is not
null-terminated.

Fixing this by making sure size is up to date where it is needed.

It could be backported up to 2.4.
2022-11-22 16:27:52 +01:00
Willy Tarreau
80f9a63184 BUILD: logs: use __fallthrough in build_log_header()
This avoids 4 build warnings when preprocessing happens before compiling
with gcc >= 7.
2022-11-14 11:14:02 +01:00
Aurelien DARRAGON
7faffdc6ab BUG/MINOR: log: fixing bug in tcp syslog_io_handler Octet-Counting
syslog_io_handler does specific treatment to handle syslog tcp octet
counting:

Logic was good, but a sneaky mistake prevented
rfc-6587 octet counting from working properly.

trash.area was used as an input buffer.
It does not make sense here since it is uninitialized.
Compilation was unaffected because trash is a thread
local "global" variable.

buf->area should definitely be used instead.

This should be backported as far as 2.4.
2022-10-27 11:28:53 +02:00
Christopher Faulet
cc640e851a BUG/MINOR: log: Preserve message facility when the log target is a ring buffer
When a ring is used as log target, the original facility, if any, must be
preserved. The default facility must only be used if there no facility was
found in the incoming log message.

This patch should fix the issue #1901. It must be backported as far as 2.4.
2022-10-20 09:03:19 +02:00
Aurelien DARRAGON
c5bff8e550 BUG/MINOR: log: improper behavior when escaping log data
Patrick Hemmer reported an improper log behavior when using
log-format to escape log data (+E option):
Some bytes were truncated from the output:

- escape_string() function now takes an extra parameter that
  allow the caller to specify input string stop pointer in
  case the input string is not guaranteed to be zero-terminated.
- Minors checks were added into lf_text_len() to make sure dst
  string will not overflow.
- lf_text_len() now makes proper use of escape_string() function.

This should be backported as far as 1.8.
2022-09-20 16:25:30 +02:00
Emeric Brun
a8942cd9c4 BUG/MAJOR: log-forward: Fix ssl layer not initialized on bind even if configured
Since commit 2071a99df ("MINOR: listener/ssl: set the SSL xprt layer only
once the whole config is known") the xprt is initialized for ssl directly
from a generic funtion used to parse bind args.

But the 'bind' lines from 'log-forward' sections were forgotten in commit
55f0f7bb5 ("MINOR: config: use the new bind_parse_args_list() to parse a
"bind" line").

This patch re-works 'log-forward' section parsing to use the generic
function to parse bind args and fix the issue.

Since the generic way to parse was introduced in 2.6, this patch
should be backported as far as this version.
2022-08-19 16:09:06 +02:00
Christopher Faulet
a892b7f15f BUG/MINOR: log: Properly test connection retries to fix dontlog-normal option
The commit 731c8e6cf ("MINOR: stream: Simplify retries counter calculation")
introduced a regression. It broke the dontlog-normal option because the test
on the connection retries counter was not updated accordingly.

This patch should fix the issue #1754. It must be backported to 2.6.
2022-06-17 14:53:21 +02:00
Willy Tarreau
c12b321661 CLEANUP: applet: rename appctx_cs() to appctx_sc()
It returns a stream connector, not a conn_stream anymore, so let's
fix its name.
2022-05-27 19:33:35 +02:00
Willy Tarreau
270a4574a4 CLEANUP: log-forward: rename all occurrences of stconn "cs" to "sc"
In the log-forwarding applet, function arguments and local variables
called "cs" were renamed to "sc" to avoid future confusion.
2022-05-27 19:33:35 +02:00
Willy Tarreau
bde14ad499 CLEANUP: check: rename all occurrences of stconn "cs" to "sc"
The check struct had a "cs" field renamed to "sc", which also required
a tiny update to a few functions using it to distinguish a check from
a stream (log.c, payload.c, ssl_sample.c, tcp_sample.c, tcpcheck.c,
connection.c).

Function arguments and local variables called "cs" were renamed to "sc".
The presence of one "cs=" in the debugging traces was also turned to
"sc=" for consistency.
2022-05-27 19:33:35 +02:00
Willy Tarreau
cb086c6de1 REORG: stconn: rename conn_stream.{c,h} to stconn.{c,h}
There's no more reason for keepin the code and definitions in conn_stream,
let's move all that to stconn. The alphabetical ordering of include files
was adjusted.
2022-05-27 19:33:35 +02:00
Willy Tarreau
5edca2f0e1 REORG: rename cs_utils.h to sc_strm.h
This file contains all the stream-connector functions that are specific
to application layers of type stream. So let's name it accordingly so
that it's easier to figure what's located there.

The alphabetical ordering of include files was preserved.
2022-05-27 19:33:35 +02:00
Willy Tarreau
f61dd19284 CLEANUP: stconn: rename cs_{shut,chk}* to sc_*
This applies the following renaming:

cs_shutr() -> sc_shutr()
cs_shutw() -> sc_shutw()
cs_chk_rcv() -> sc_chk_rcv()
cs_chk_snd() -> sc_chk_snd()
cs_must_kill_conn() -> sc_must_kill_conn()
2022-05-27 19:33:35 +02:00
Willy Tarreau
d68ff018c5 CLEANUP: stconn: rename cs{,_get}_{src,dst} to sc_*
The following functions were renamed:

cs_src() -> sc_src()
cs_dst() -> sc_dst()
cs_get_src() -> sc_get_src()
cs_get_dst() -> sc_get_dst()
2022-05-27 19:33:35 +02:00
Willy Tarreau
fd9417ba3f CLEANUP: stconn: rename cs_conn() to sc_conn()
It's mostly used from upper layers. Both the checked and unchecked
functions were updated, or ~150 entries.
2022-05-27 19:33:34 +02:00
Willy Tarreau
ea27f48c5a CLEANUP: stconn: rename cs_{check,strm,strm_task} to sc_strm_*
These functions return the app-layer associated with an stconn, which
is a check, a stream or a stream's task. They're used a lot to access
channels, flags and for waking up tasks. Let's just name them
appropriately for the stream connector.
2022-05-27 19:33:34 +02:00
Willy Tarreau
40a9c32e3a CLEANUP: stconn: rename cs_{i,o}{b,c} to sc_{i,o}{b,c}
We're starting to propagate the stream connector's new name through the
API. Most call places of these functions that retrieve the channel or its
buffer are in applets. The local variable names are not changed in order
to keep the changes small and reviewable. There were ~92 uses of cs_ic(),
~96 of cs_oc() (due to co_get*() being less factorizable than ci_put*),
and ~5 accesses to the buffer itself.
2022-05-27 19:33:34 +02:00
Willy Tarreau
7cb9e6c6ba CLEANUP: stream: rename "csf" and "csb" to "scf" and "scb"
These are the stream connectors, let's give them consistent names. The
patch is large (405 locations) but totally trivial.
2022-05-27 19:33:34 +02:00
Willy Tarreau
4596fe20d9 CLEANUP: conn_stream: tree-wide rename to stconn (stream connector)
This renames the "struct conn_stream" to "struct stconn" and updates
the descriptions in all comments (and the rare help descriptions) to
"stream connector" or "connector". This touches a lot of files but
the change is minimal. The local variables were not even renamed, so
there's still a lot of "cs" everywhere.
2022-05-27 19:33:34 +02:00
Willy Tarreau
91b47263f7 MINOR: protocol: replace ctrl_type with xprt_type and clarify it
There's been some great confusion between proto_type, ctrl_type and
sock_type. It turns out that ctrl_type was improperly chosen because
it's not the control layer that is of this or that type, but the
transport layer, and it turns out that the transport layer doesn't
(normally) denaturate the underlying control layer, except for QUIC
which turns dgrams to streams. The fact that the SOCK_{DGRAM|STREAM}
set of values was used added to the confusion.

Let's replace it with xprt_type which reuses the later introduced
PROTO_TYPE_* values, and update the comments to explain which one
works at what level.
2022-05-20 18:39:43 +02:00
Willy Tarreau
0698c80a58 CLEANUP: applet: remove the unneeded appctx->owner
This one is the pointer to the conn_stream which is always in the
endpoint that is always present in the appctx, thus it's not needed.
This patch removes it and replaces it with appctx_cs() instead. A
few occurences that were using __cs_strm(appctx->owner) were moved
directly to appctx_strm() which does the equivalent.
2022-05-13 14:28:48 +02:00
Willy Tarreau
382474348c CLEANUP: tree-wide: use fd_set_nonblock() and fd_set_cloexec()
This gets rid of most open-coded fcntl() calls, some of which were passed
through DISGUISE() to avoid a useless test. The FD_CLOEXEC was most often
set without preserving previous flags, which could become a problem once
new flags are created. Now this will not happen anymore.
2022-04-26 10:59:48 +02:00
Willy Tarreau
acef5e27b0 MINOR: tree-wide: always consider EWOULDBLOCK in addition to EAGAIN
Some older systems may routinely return EWOULDBLOCK for some syscalls
while we tend to check only for EAGAIN nowadays. Modern systems define
EWOULDBLOCK as EAGAIN so that solves it, but on a few older ones (AIX,
VMS etc) both are different, and for portability we'd need to test for
both or we never know if we risk to confuse some status codes with
plain errors.

There were few entries, the most annoying ones are the switch/case
because they require to only add the entry when it differs, but the
other ones are really trivial.
2022-04-25 20:32:15 +02:00
Christopher Faulet
6b0a0fb2f9 CLEANUP: tree-wide: Remove any ref to stream-interfaces
Stream-interfaces are gone. Corresponding files can be safely be removed. In
addition, comments are updated accordingly.
2022-04-13 15:10:16 +02:00
Christopher Faulet
da098e6c17 MINOR: stream-int/conn-stream: Move si_shut* and si_chk* in conn-stream scope
si_shutr(), si_shutw(), si_chk_rcv() and si_chk_snd() are moved in the
conn-stream scope and renamed, respectively, cs_shutr(), cs_shutw(),
cs_chk_rcv(), cs_chk_snd() and manipulate a conn-stream instead of a
stream-interface.
2022-04-13 15:10:15 +02:00
Christopher Faulet
8da67aae3e MEDIUM: stream-int/conn-stream: Move src/dst addresses in the conn-stream
The source and destination addresses at the applicative layer are moved from
the stream-interface to the conn-stream. This simplifies a bit the code and
it is a logicial step to remove the stream-interface.
2022-04-13 15:10:14 +02:00
Christopher Faulet
731c8e6cf9 MINOR: stream: Simplify retries counter calculation
The conn_retries counter was set to the max value and decremented at each
connection retry. Thus the counter reflected the number of retries left and
not the real number of retries. All calculations of redispatch or reporting
of number of retries experienced were made using subtracts from the
configured retries, which was complicated and didn't bring any benefit.

Now, this counter is set to 0 and incremented at each retry. We know we've
reached the maximum allowed connection retries by comparing it to the
configured value. In all other cases, we directly use the counter.

This patch should address the feature request #1608.
2022-04-13 15:10:14 +02:00
Christopher Faulet
909f318259 MINOR: stream-int/stream: Move conn_retries counter in the stream
The conn_retries counter may be moved into the stream structure. It only
concerns the connection establishment. The frontend stream-interface does not
use it. So it is a logical change.
2022-04-13 15:10:14 +02:00
Christopher Faulet
908628c4c0 MEDIUM: tree-wide: Use CS util functions instead of SI ones
At many places, we now use the new CS functions to get a stream or a channel
from a conn-stream instead of using the stream-interface API. It is the
first step to reduce the scope of the stream-interfaces. The main change
here is about the applet I/O callback functions. Before the refactoring, the
stream-interface was the appctx owner. Thus, it was heavily used. Now, as
far as possible,the conn-stream is used. Of course, it remains many calls to
the stream-interface API.
2022-04-13 15:10:14 +02:00
Willy Tarreau
807a3a53bb MINOR: log: add '~' to frontend when the transport layer provides SSL
We used to check if the transport layer was ssl_sock to decide to log
"~" after a frontend's name. Now that QUIC is present, this doesn't work
anymore. Better rely on the transport layer's get_ssl_sock_ctx() method.
2022-04-12 08:08:33 +02:00
Christopher Faulet
b4f96eda56 BUG/MINOR: log: Initialize the list element when allocating a new log server
211ea252d ("BUG/MINOR: logs: fix logsrv leaks on clean exit") introduced a
regression because the list element of a new log server is not intialized. Thus
HAProxy crashes on error path when an invalid log server is released.

This patch shoud fix the issue #1636. It must be backported if the above commit
is backported. For now, it is 2.6-specific and no backport is needed.
2022-03-29 14:17:10 +02:00
Tim Duesterhus
7750850594 CLEANUP: Reapply ist.cocci with --include-headers-for-types --recursive-includes
Previous uses of `ist.cocci` did not add `--include-headers-for-types` and
`--recursive-includes` preventing Coccinelle seeing `struct ist` members of
other structs.

Reapply the patch with proper flags to further clean up the use of the ist API.

The command used was:

    spatch -sp_file dev/coccinelle/ist.cocci -in_place --include-headers --include-headers-for-types --recursive-includes --dir src/
2022-03-21 08:30:47 +01:00
Willy Tarreau
211ea252d9 BUG/MINOR: logs: fix logsrv leaks on clean exit
Log servers are a real mess because:
  - entries are duplicated using memcpy() without their strings being
    reallocated, which results in these ones not being freeable every
    time.

  - a new field, ring_name, was added in 2.2 by commit 99c453df9
    ("MEDIUM: ring: new section ring to declare custom ring buffers.")
    but it's never initialized during copies, causing the same issue

  - no attempt is made at freeing all that.

Of course, running "haproxy -c" under ASAN quickly notices that and
dumps a core.

This patch adds the missing strdup() and initialization where required,
adds a new free_logsrv() function to cleanly free() such a structure,
calls it from the proxy when iterating over logsrvs instead of silently
leaking their file names and ring names, and adds the same logsrv loop
to the proxy_free_defaults() function so that we don't leak defaults
sections on exit.

It looks a bit entangled, but it comes as a whole because all this stuff
is inter-dependent and was missing.

It's probably preferable not to backport this in the foreseable future
as it may reveal other jokes if some obscure parts continue to memcpy()
the logsrv struct.
2022-03-17 19:53:46 +01:00
Christopher Faulet
02fc86e8f6 MINOR: log: Always access the stream-int via the conn-stream
To be able to move the stream-interface from the stream to the conn-stream,
all access to the SI is done via the conn-stream. This patch is limited to
the log part.
2022-02-24 11:00:02 +01:00
Christopher Faulet
95a61e8a0e MINOR: stream: Add pointer to front/back conn-streams into stream struct
frontend and backend conn-streams are now directly accesible from the
stream. This way, and with some other changes, it will be possible to remove
the stream-interfaces from the stream structure.
2022-02-24 11:00:02 +01:00
Christopher Faulet
86e1c3381b MEDIUM: applet: Set the conn-stream as appctx owner instead of the stream-int
Because appctx is now an endpoint of the conn-stream, there is no reason to
still have the stream-interface as appctx owner. Thus, the conn-stream is
now the appctx owner.
2022-02-24 11:00:02 +01:00
Christopher Faulet
13a35e5752 MAJOR: conn_stream/stream-int: move the appctx to the conn-stream
Thanks to previous changes, it is now possible to set an appctx as endpoint
for a conn-stream. This means the appctx is no longer linked to the
stream-interface but to the conn-stream. Thus, a pointer to the conn-stream
is explicitly stored in the stream-interface. The endpoint (connection or
appctx) can be retrieved via the conn-stream.
2022-02-24 11:00:02 +01:00
Emeric Brun
2ad2b1c94c BUG/MAJOR: segfault using multiple log forward sections.
For each new log forward section, the proxy was added to the log forward
proxy list but the ref on the previous log forward section's proxy was
scratched using "init_new_proxy" which performs a memset. After configuration
parsing this list contains only the last section's proxy.

The post processing walk through this list to resolve "ring" names.
Since some section's proxies are missing in this list, the resolving
is not done for those ones and the pointer on the ring is kept to null
causing a segfault at runtime trying to write a log message
into the ring.

This patch shift the "init_new_proxy" before adding the ref on the
previous log forward section's proxy on currently parsed one.

This patch shoud fix github issue #1464

This patch should be backported to 2.3
2021-12-01 15:21:56 +01:00
Christopher Faulet
1ccbe12f4a DOC: log: Add comments to specify when session's listener is defined or not
When a log message is emitted, The session's listener is always defined when
the session's owner is an inbound connection while it is undefined for a
health-check. It is not obvious. So, comments have been added to make it
clear.

This patch is related to the issue #1434.
2021-11-15 11:31:09 +01:00
Tim Duesterhus
2471f5c2b2 CLEANUP: Apply ist.cocci
Make use of the new rules to use `isttrim()`.
2021-11-08 12:08:26 +01:00
Willy Tarreau
68574dd492 MEDIUM: log: add the client's SNI to the default HTTPS log format
During a troublehooting it came obvious that the SNI always ought to
be logged on httpslog, as it explains errors caused by selection of
the default certificate (or failure to do so in case of strict-sni).

This expectation was also confirmed on the mailing list.

Since the field may be empty it appeared important not to leave an
empty string in the current format, so it was decided to place the
field before a '/' preceding the SSL version and ciphers, so that
in the worst case a missing field leads to a field looking like
"/TLSv1.2/AES...", though usually a missing element still results
in a "-" in logs.

This will change the log format for users who already deployed the
2.5-dev versions (hence the medium level) but no released version
was using this format yet so there's no harm for stable deployments.
The reg-test was updated to check for "-" there since we don't send
SNI in reg-tests.

Link: https://www.mail-archive.com/haproxy@formilux.org/msg41410.html
Cc: William Lallemand <wlallemand@haproxy.org>
2021-11-06 09:20:07 +01:00
Willy Tarreau
6f7497616e MEDIUM: connection: rename fc_conn_err and bc_conn_err to fc_err and bc_err
Commit 3d2093af9 ("MINOR: connection: Add a connection error code sample
fetch") added these convenient sample-fetch functions but it appears that
due to a misunderstanding the redundant "conn" part was kept in their
name, causing confusion, since "fc" already stands for "front connection".

Let's simply call them "fc_err" and "bc_err" to match all other related
ones before they appear in a final release. The VTC they appeared in were
also updated, and the alpha sort in the keywords table updated.

Cc: William Lallemand <wlallemand@haproxy.org>
2021-11-06 09:20:07 +01:00
Christopher Faulet
52b28d2f30 BUILD: log: Fix compilation without SSL support
When compiled without SSL support, a variable is reported as not used by
GCC.

src/log.c: In function ‘sess_build_logline’:
src/log.c:2056:36: error: unused variable ‘conn’ [-Werror=unused-variable]
 2056 |                 struct connection *conn;
      |                                    ^~~~

This does not need to be backported.
2021-10-27 12:00:15 +02:00
Christopher Faulet
f9c4d8d5be MINOR: log: Rely on client addresses at the appropriate level to log messages
When a log message is emitted, if the stream exits, we use the frontend
stream-interface to retrieve the client source and destination
addresses. Otherwise, the session is used. For now, stream-interface or
session addresses are never set. So, thanks to the fallback mechanism, no
changes are expected with this patch. But its purpose is to rely on
addresses at the appropriate level when set instead of those at the
connection level.
2021-10-27 11:34:21 +02:00
Christopher Faulet
6ff7de5d64 MINOR: tcpcheck: Support 2-steps args resolution in defaults sections
With the commit eaba25dd9 ("BUG/MINOR: tcpcheck: Don't use arg list for
default proxies during parsing"), we restricted the use of sample fetch in
tcpcheck rules defined in a defaults section to those depending on explicit
arguments only. This means a tcpcheck rules defined in a defaults section
cannot rely on argument unresolved during the configuration parsing.

Thanks to recent changes, it is now possible again.

This patch is mandatory to support TCP/HTTP rules in defaults sections.
2021-10-15 14:12:19 +02:00
William Lallemand
1d58b01316 MINOR: ssl: add ssl_fc_is_resumed to "option httpslog"
In order to trace which session were TLS resumed, add the
ssl_fc_is_resumed in the httpslog option.
2021-10-14 14:27:48 +02:00
Willy Tarreau
5554264f31 REORG: time: move time-keeping code and variables to clock.c
There is currently a problem related to time keeping. We're mixing
the functions to perform calculations with the os-dependent code
needed to retrieve and adjust the local time.

This patch extracts from time.{c,h} the parts that are solely dedicated
to time keeping. These are the "now" or "before_poll" variables for
example, as well as the various now_*() functions that make use of
gettimeofday() and clock_gettime() to retrieve the current time.

The "tv_*" functions moved there were also more appropriately renamed
to "clock_*".

Other parts used to compute stolen time are in other files, they will
have to be picked next.
2021-10-08 17:22:26 +02:00
Willy Tarreau
b7fc4c4e9f BUILD: tree-wide: add missing http_ana.h from many places
At least 6 files make use of s->txn without including http_ana which
defines it. They used to get it from other includes.
2021-10-07 01:36:51 +02:00
Christopher Faulet
eaba25dd97 BUG/MINOR: tcpcheck: Don't use arg list for default proxies during parsing
During tcp/http check rules parsing, when a sample fetch or a log-format
string is parsed, the proxy's argument list used to track unresolved
argument is no longer passed for default proxies. It means it is no longer
possible to rely on sample fetches depending on the execution context (for
instance 'nbsrv').

It is important to avoid HAProxy crashes because these arguments are
resolved during the configuration validity check. But, default proxies are
not evaluated during this stage. Thus, these arguments remain unresolved.

It will probably be possible to relax this rule. But to ease backports, it
is forbidden for now.

This patch must be backported as far as 2.2. It depends on the commit
"MINOR: arg: Be able to forbid unresolved args when building an argument
list".  It must be adapted for the 2.3 because PR_CAP_DEF capability was
introduced in the 2.4. A solution may be to test The proxy's id agains NULL.
2021-09-30 16:37:05 +02:00
Remi Tricot-Le Breton
1fe0fad88b MINOR: ssl: Rename ssl_bc_hsk_err to ssl_bc_err
The ssl_bc_hsk_err sample fetch will need to raise more errors than only
handshake related ones hence its renaming to a more generic ssl_bc_err.
This patch is required because some handshake failures that should have
been caught by this fetch (verify error on the server side for instance)
were missed. This is caused by a change in TLS1.3 in which the
'Finished' state on the client is reached before its certificate is sent
(and verified) on the server side (see the "Protocol Overview" part of
RFC 8446).
This means that the SSL_do_handshake call is finished long before the
server can verify and potentially reject the client certificate.

The ssl_bc_hsk_err will then need to be expanded to catch other types of
errors.

This change is also applied to the frontend fetches (ssl_fc_hsk_err
becomes ssl_fc_err) and to their string counterparts.
2021-09-30 11:04:35 +02:00
Christopher Faulet
a015b3ec8b MINOR: log: Try to get the status code when MUX_EXIT_STATUS is retrieved
The mux .ctl callback can provide some information about the mux to the
caller if the third parameter is provided. Thus, when MUX_EXIT_STATUS is
retrieved, a pointer on the status is now passed. The mux may fill it. It
will be pretty handy to provide custom error code from h1 mux instead of
default ones (400/408/500/501).
2021-09-28 13:52:25 +02:00
Tim Duesterhus
16554245e2 CLEANUP: Apply bug_on.cocci
The changes look safe to me, even if `DEBUG_STRICT` is not enabled.
2021-09-17 17:22:05 +02:00
Willy Tarreau
20b7a0f9ed MINOR: log: make log-format expressions completely usable outside of req/resp
The log-format strings are usable at plenty of places, but the expressions
using %[] were restricted to request or response context and nothing else.
This prevents from using them from the config context or the CLI, let's
relax this.
2021-09-03 11:01:48 +02:00
Remi Tricot-Le Breton
fe21fe76bd MINOR: log: Add new "error-log-format" option
This option can be used to define a specific log format that will be
used in case of error, timeout, connection failure on a frontend... It
will be used for any log line concerned by the log-separate-errors
option. It will also replace the format of specific error messages
decribed in section 8.2.6.
If no "error-log-format" is defined, the legacy error messages are still
emitted and the other error logs keep using the regular log-format.
2021-08-31 12:13:08 +02:00
Remi Tricot-Le Breton
98b930d043 MINOR: ssl: Define a default https log format
This patch adds a new httpslog option and a new HTTP over SSL log-format
that expands the default HTTP format and adds SSL specific information.
2021-07-29 15:40:45 +02:00
Amaury Denoyelle
c453f9547e MINOR: http: use http uri parser for path
Replace http_get_path by the http_uri_parser API. The new functions is
renamed http_parse_path. Replace duplicated code for scheme and
authority parsing by invocations to http_parse_scheme/authority.

If no scheme is found for an URI detected as an absolute-uri/authority,
consider it to be an authority format : no path will be found. For an
absolute-uri or absolute-path, use the remaining of the string as the
path. A new http_uri_parser state is declared to mark the path parsing
as done.
2021-07-08 17:11:17 +02:00
Willy Tarreau
2a651e2d0d BUILD: log: remove unused fmt_directive()
fmt_directive() became unused after the removal of the deprecated
tags, and it emits a warning on some compilers. Let's drop it.
2021-06-11 17:32:03 +02:00
Willy Tarreau
fd6ab66041 MINOR: log: remove the long-deprecated early log-format tags
The following 10 log-format tags were implemented during log-format
development and changed before the release. They were marked as deprecated
in 2012 by commit 2beef5888 ("MEDIUM: log: change a few log tokens to make
them easier to remember") and were not documented. They've been emitting a
warning since then, with a suggestion of the one to use instead. Let's get
rid of them now.

      Bi => bi, Bp => bp, Ci => ci, Cp => cp, Fi => fi
      Fp => fp, Si => si, Sp => sp, cc => CC, cs => CS
2021-06-11 16:57:34 +02:00
Amaury Denoyelle
ce986e1ce8 REORG: errors: split errors reporting function from log.c
Move functions related to errors output on stderr from log.c to a newly
created errors.c file. It targets print_message and
ha_alert/warning/notice/diag functions and related startup_logs feature.
2021-06-07 16:58:15 +02:00
Willy Tarreau
b205bfdab7 CLEANUP: cli/tree-wide: properly re-align the CLI commands' help messages
There were 102 CLI commands whose help were zig-zagging all along the dump
making them unreadable. This patch realigns all these messages so that the
command now uses up to 40 characters before the delimiting colon. About a
third of the commands did not correctly list their arguments which were
added after the first version, so they were all updated. Some abuses of
the term "id" were fixed to use a more explanatory term. The
"set ssl ocsp-response" command was not listed because it lacked a help
message, this was fixed as well. The deprecated enable/disable commands
for agent/health/server were prominently written as deprecated. Whenever
possible, clearer explanations were provided.
2021-05-07 11:51:26 +02:00
Willy Tarreau
9f903af510 MEDIUM: log: slightly refine the output format of alerts/warnings/etc
For about 20 years we've been emitting cryptic messages on warnings and
alerts, that nobody knows how to parse:

  [NOTICE] 126/080118 (3115) : haproxy version is 2.4-dev18-0b7c78-49
  [NOTICE] 126/080118 (3115) : path to executable is ./haproxy
  [WARNING] 126/080119 (3115) : Server default/srv1 is DOWN via static/srv1. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  [ALERT] 126/080119 (3115) : backend 'default' has no server available!

Hint: the first 3-digit number is the day of year, and the 6 digits
after it represent the time of day in format HHMMSS, then the pid in
parenthesis. These are not quite user-friendly and such cryptic into
are not useful at all.

This patch slightly adjusts the output by performing these minimal changes:
  - removing the date/time, as they were added very early when haproxy
    was meant to be used in foreground as a debugging tool, and they're
    provided in more details in logs nowadays ;

  - better aligning the fields by padding the severity tag to 10 chars.
    The diag output was renamed to "DIAG" only.

Now the output provides this:

  [NOTICE]   (4563) : haproxy version is 2.4-dev18-75a428-51
  [NOTICE]   (4563) : path to executable is ./haproxy
  [WARNING]  (4563) : Server default/srv1 is DOWN via static/srv1. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  [ALERT]    (4563) : backend 'default' has no server available!

The useless space before the colon was kept so as not to confuse any
possible output parser.

The few entries in the doc referring to this format were adjusted to
reflect the new one.

The change was tagged "MEDIUM" as it may have visible consequences on
home-grown monitoring tools, though it is extremely unlikely due to the
limited extent of these changes.
2021-05-07 08:55:11 +02:00
Willy Tarreau
2b71810cb3 CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion
The current "ADD" vs "ADDQ" is confusing because when thinking in terms
of appending at the end of a list, "ADD" naturally comes to mind, but
here it does the opposite, it inserts. Several times already it's been
incorrectly used where ADDQ was expected, the latest of which was a
fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain
why we queue the stream at the head of the server list").

Let's use more explicit (but slightly longer) names now:

   LIST_ADD        ->       LIST_INSERT
   LIST_ADDQ       ->       LIST_APPEND
   LIST_ADDED      ->       LIST_INLIST
   LIST_DEL        ->       LIST_DELETE

The same is true for MT_LISTs, including their "TRY" variant.
LIST_DEL_INIT keeps its short name to encourage to use it instead of the
lazier LIST_DELETE which is often less safe.

The change is large (~674 non-comment entries) but is mechanical enough
to remain safe. No permutation was performed, so any out-of-tree code
can easily map older names to new ones.

The list doc was updated.
2021-04-21 09:20:17 +02:00
Christopher Faulet
1d26f22e05 BUG/MINOR: logs: Report the true number of retries if there was no connection
When the session is aborted before any connection attempt to any server, the
number of connection retries reported in the logs is wrong. It happens
because when the retries counter is not strictly positive, we consider the
max number of retries was reached and the backend retries value is used. It
is obviously wrong when no connectioh was performed.

In fact, at this stage, the retries counter is initialized to 0. But the
backend stream-interface is in the INI state. Once it is set to SI_ST_REQ,
the counter is set to the backend value. And it is the only possible state
transition from INI state. Thus it is safe to rely on it to fix the bug.

This patch must be backported to all stable versions.
2021-04-19 08:52:17 +02:00
Christopher Faulet
fd81848c22 MINOR: logs: Add support of checks as session origin to format lf strings
When a log-format string is built from an health-check, the session origin
is the health-check itself and not a connection. In addition, there is no
stream. It means for now some formats are not supported: %s, %sc, %b, %bi,
%bp, %si and %sp.

Thanks to this patch, the session origin is converted to a check. So it is
possible to retrieve the backend and the backend connection. Note this
session have no listener, thus %ft format must be guarded.

This patch is light and standalone, thus it may be backported as far as 2.2
if required. However, because the error is human, it is probably better to
wait a bit to be sure everything is properly protected.
2021-04-19 08:22:15 +02:00
Christopher Faulet
0c6d1dcf7d BUG/MINOR: listener: Handle allocation error when allocating a new bind_conf
Allocation error are now handled in bind_conf_alloc() functions. Thus
callers, when not already done, are also updated to catch NULL return value.

This patch may be backported (at least partially) to all stable
versions. However, it only fix errors durung configuration parsing. Thus it
is not mandatory.
2021-04-12 21:33:43 +02:00
Willy Tarreau
4781b1521a CLEANUP: atomic/tree-wide: replace single increments/decrements with inc/dec
This patch replaces roughly all occurrences of an HA_ATOMIC_ADD(&foo, 1)
or HA_ATOMIC_SUB(&foo, 1) with the equivalent HA_ATOMIC_INC(&foo) and
HA_ATOMIC_DEC(&foo) respectively. These are 507 changes over 45 files.
2021-04-07 18:18:37 +02:00
Willy Tarreau
185157201c CLEANUP: atomic: add a fetch-and-xxx variant for common operations
The fetch_and_xxx variant is often missing for add/sub/and/or. In fact
it was only provided for ADD under the name XADD which corresponds to
the x86 instruction name. But for destructive operations like AND and
OR it's missing even more as it's not possible to know the value before
modifying it.

This patch explicitly adds HA_ATOMIC_FETCH_{OR,AND,ADD,SUB} which
cover these standard operations, and renames XADD to FETCH_ADD (there
were only 6 call places).

In the future, backport of fixes involving such operations could simply
remap FETCH_ADD(x) to XADD(x), FETCH_SUB(x) to XADD(-x), and for the
OR/AND if needed, these could possibly be done using BTS/BTR.

It's worth noting that xchg could have been renamed to fetch_and_store()
but xchg already has well understood semantics and it wasn't needed to
go further.
2021-04-07 18:18:37 +02:00
Willy Tarreau
f509065191 MEDIUM: fd: merge fdtab[].ev and state for FD_EV_* and FD_POLL_* into state
For a long time we've had fdtab[].ev and fdtab[].state which contain two
arbitrary sets of information, one is mostly the configuration plus some
shutdown reports and the other one is the latest polling status report
which also contains some sticky error and shutdown reports.

These ones used to be stored into distinct chars, complicating certain
operations and not even allowing to clearly see concurrent accesses (e.g.
fd_delete_orphan() would set the state to zero while fd_insert() would
only set the event to zero).

This patch creates a single uint with the two sets in it, still delimited
at the byte level for better readability. The original FD_EV_* values
remained at the lowest bit levels as they are also known by their bit
value. The next step will consist in merging the remaining bits into it.

The whole bits are now cleared both in fd_insert() and _fd_delete_orphan()
because after a complete check, it is certain that in both cases these
functions are the only ones touching these areas. Indeed, for
_fd_delete_orphan(), the thread_mask has already been zeroed before a
poller can call fd_update_event() which would touch the state, so it
is certain that _fd_delete_orphan() is alone. Regarding fd_insert(),
only one thread will get an FD at any moment, and it as this FD has
already been released by _fd_delete_orphan() by definition it is certain
that previous users have definitely stopped touching it.

Strictly speaking there's no need for clearing the state again in
fd_insert() but it's cheap and will remove some doubts during some
troubleshooting sessions.
2021-04-07 18:04:39 +02:00
Emeric Brun
26754901e9 BUG/MEDIUM: log: fix config parse error logging on stdout/stderr or any raw fd
The regression was introduced by commit previous commit 94aab06:
MEDIUM: log: support tcp or stream addresses on log lines.

This previous patch tries to retrieve the used protocol parsing
the address using the str2sa_range function but forgets that
the raw file descriptor adresses don't specify a protocol
and str2sa_range probes an error.

This patch re-work the str2sa_range function to stop
probing error if an authorized RAW_FD address is parsed
whereas the caller request also a protocol.

It also modify the code of parse_logsrv to switch on stream
logservers only if a protocol was detected.
2021-04-07 15:01:00 +02:00
Emeric Brun
94aab06e24 MEDIUM: log: support tcp or stream addresses on log lines.
An explicit stream address prefix such as "tcp6@" "tcp4@"
"stream+ipv6@" "stream+ipv4@" or "stream+unix@" will
allocate an implicit ring buffer with a forward server
targeting the given address.

This is usefull to simply send logs to a log server in tcp
and It doesn't need to declare a ring section in configuration.
2021-04-07 09:18:34 +02:00
Emeric Brun
9533a70381 MINOR: log: register config file and line number on log servers.
This patch registers the parsed file and the line where a log server
is declared to make those information available in configuration
post check.

Those new informations were added on error messages probed resolving
ring names on post configuration check.
2021-04-07 09:18:34 +02:00
Amaury Denoyelle
7b01a8dbdd MINOR: global: define diagnostic mode of execution
Define MODE_DIAG which is used to run haproxy in diagnostic mode. This
mode is used to output extra warnings about possible configuration
blunder or sub-optimal usage. It can be activated with argument '-dD'.

A new output function ha_diag_warning is implemented reserved for
diagnostic output. It serves to standardize the format of diagnostic
messages.

A macro HA_DIAG_WARN_COND is also available to automatically check if
diagnostic mode is on before executing the diagnostic check.
2021-04-01 18:03:37 +02:00
Christopher Faulet
581db2b829 MINOR: payload/config: Warn if a L6 sample fetch is used from an HTTP proxy
L6 sample fetches are now ignored when called from an HTTP proxy. Thus, a
warning is emitted during the startup if such usage is detected. It is true
for most ACLs and for log-format strings. Unfortunately, it is a bit painful
to do so for sample expressions.

This patch relies on the commit "MINOR: action: Use a generic function to
check validity of an action rule list".
2021-04-01 15:34:22 +02:00
Willy Tarreau
433b05fa64 MINOR: cfgparse/bind: suggest correct spelling for unknown bind keywords
Just like with the server keywords, now's the turn of "bind" keywords.
The difference is that 100% of the bind keywords are registered, thus
we do not need the list of extra keywords.

There are multiple bind line parsers today, all were updated:
  - peers
  - log
  - dgram-bind
  - cli

$ printf "listen f\nbind :8000 tcut\n" | ./haproxy -c -f /dev/stdin
[NOTICE] 070/101358 (25146) : haproxy version is 2.4-dev11-7b8787-26
[NOTICE] 070/101358 (25146) : path to executable is ./haproxy
[ALERT] 070/101358 (25146) : parsing [/dev/stdin:2] : 'bind :8000' unknown keyword 'tcut'; did you mean 'tcp-ut' maybe ?
[ALERT] 070/101358 (25146) : Error(s) found in configuration file : /dev/stdin
[ALERT] 070/101358 (25146) : Fatal errors found in configuration.
2021-03-12 14:13:21 +01:00
Tim Duesterhus
7b5777d9b4 CLEANUP: Use isttest(const struct ist) whenever possible
Refactoring performed with the following Coccinelle patch:

    @@
    struct ist i;
    @@

    - i.ptr != NULL
    + isttest(i)
2021-03-03 05:07:10 +01:00
Tim Duesterhus
92c696e663 CLEANUP: Use ist2(const void*, size_t) whenever possible
Refactoring performed with the following Coccinelle patch:

    @@
    struct ist i;
    expression p, l;
    @@

    - i.ptr = p;
    - i.len = l;
    + i = ist2(p, l);
2021-03-01 15:44:20 +01:00
Willy Tarreau
66161326fd MINOR: listener: refine the default MAX_ACCEPT from 64 to 4
The maximum number of connections accepted at once by a thread for a single
listener used to default to 64 divided by the number of processes but the
tasklet-based model is much more scalable and benefits from smaller values.
Experimentation has shown that 4 gives the highest accept rate for all
thread values, and that 3 and 5 come very close, as shown below (HTTP/1
connections forwarded per second at multi-accept 4 and 64):

 ac\thr|    1     2    4     8     16
 ------+------------------------------
      4|   80k  106k  168k  270k  336k
     64|   63k   89k  145k  230k  274k

Some tests were also conducted on SSL and absolutely no change was observed.

The value was placed into a define because it used to be spread all over the
code.

It might be useful at some point to backport this to 2.3 and 2.2 to help
those who observed some performance regressions from 1.6.
2021-02-19 16:02:04 +01:00
Christopher Faulet
142dd33912 MINOR: muxes: Add exit status for errors about not implemented features
The MUX_ES_NOTIMPL_ERR exit status is added to allow the multiplexers to
report errors about not implemented features. This will be used by the H1
mux to return 501-not-implemented errors.
2021-01-21 15:21:12 +01:00
Tim Duesterhus
e5ff14100a CLEANUP: Compare the return value of XXXcmp() functions with zero
According to coding-style.txt it is recommended to use:

`strcmp(a, b) == 0` instead of `!strcmp(a, b)`

So let's do this.

The change was performed by running the following (very long) coccinelle patch
on src/:

    @@
    statement S;
    expression E;
    expression F;
    @@

      if (
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) != 0
      )
    (
      S
    |
      { ... }
    )

    @@
    statement S;
    expression E;
    expression F;
    @@

      if (
    - !
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) == 0
      )
    (
      S
    |
      { ... }
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    G &&
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) != 0
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    G ||
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) != 0
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) != 0
    && G
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) != 0
    || G
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    G &&
    - !
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) == 0
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    G ||
    - !
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) == 0
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    - !
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) == 0
    && G
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    - !
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) == 0
    || G
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    - !
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) == 0
    )
2021-01-04 10:09:02 +01:00
Ilya Shipitsin
f38a01884a CLEANUP: assorted typo fixes in the code and comments
This is 13n iteration of typo fixes
2020-12-21 11:24:48 +01:00
Miroslav Zagorac
7f8314c8d1 MINOR: opentracing: add ARGC_OT enum
Due to the addition of the OpenTracing filter it is necessary to define
ARGC_OT enum.  This value is used in the functions fmt_directive() and
smp_resolve_args().
2020-12-16 15:49:53 +01:00
Christopher Faulet
ce5e6bcb04 MINOR: logs: Get the multiplexer exist status when no stream is provided
When a log message is emitted from the session level, by a multiplexer,
there is no stream. Thus for HTTP session, there no status code and the
termination flags are not correctly set.

Thanks to previous patch, the HTTP status code is deduced from the mux exist
status, using the MUX_EXIT_STATE ctl param. This is only done for HTTP
frontends. If it is defined ( != 0), it is used to deduce the termination
flags.
2020-12-04 14:41:49 +01:00
Christopher Faulet
dd78921c66 MINOR: logs: Use session idle duration when no stream is provided
When a log message is emitted from the session, using sess_log() function,
there is no stream available. In this case, instead of deducing the idle
duration from the accept date, we use the one provided by the session. 0 is
used if it is undefined (i.e set to -1).
2020-12-04 14:41:48 +01:00
Maciej Zdeb
fcdfd857b3 MINOR: log: Logging HTTP path only with %HPO
This patch adds a new logging variable '%HPO' for logging HTTP path only
(without query string) from relative or absolute URI.

For example:
log-format "hpo=%HPO hp=%HP hu=%HU hq=%HQ"

GET /r/1 HTTP/1.1
=>
hpo=/r/1 hp=/r/1 hu=/r/1 hq=

GET /r/2?q=2 HTTP/1.1
=>
hpo=/r/2 hp=/r/2 hu=/r/2?q=2 hq=?q=2

GET http://host/r/3 HTTP/1.1
=>
hpo=/r/3 hp=http://host/r/3 hu=http://host/r/3 hq=

GET http://host/r/4?q=4 HTTP/1.1
=>
hpo=/r/4 hp=http://host/r/4 hu=http://host/r/4?q=4 hq=?q=4
2020-12-01 09:32:44 +01:00
Emeric Brun
0237c4e3f5 BUG/MEDIUM: local log format regression.
Since 2.3 default local log format always adds hostame field.
This behavior change was due to log/sink re-work, because according
to rfc3164 the hostname field is mandatory.

This patch re-introduce a legacy "local" format which is analog
to rfc3164 but with hostname stripped. This is the new
default if logs are generated by haproxy.

To stay compliant with previous configurations, the option
"log-send-hostname" acts as if the default format is switched
to rfc3164.

This patch addresses the github issue #963

This patch should be backported in branches >= 2.3.
2020-12-01 06:58:42 +01:00
Willy Tarreau
a0133fcf35 BUG/MINOR: log: fix risk of null deref on error path
Previous commit ae32ac74db ("BUG/MINOR: log: fix memory leak on logsrv
parse error") addressed one issue and introduced another one, the logsrv
pointer may also be null at the end of the function so we must test it
before deciding to dereference it.

This should be backported along with the patch above to 2.2.
2020-10-27 10:35:32 +01:00
Willy Tarreau
ae32ac74db BUG/MINOR: log: fix memory leak on logsrv parse error
In case of parsing error on logsrv, we can leave parse_logsrv() without
releasing logsrv->ring_name or smp_rgs. Let's free them on the error path.
This should fix issue #926 detected by Coverity.

The impact is only a tiny leak just before reporting a fatal error, so it
will essentially annoy valgrind.

This can be backported to 2.0 (just drop the ring part).
2020-10-27 09:55:00 +01:00
Willy Tarreau
e140a6921f MINOR: log: set the UDP receiver's I/O handler in the receiver
The I/O handler is syslog_fd_handler(), let's set it when creating
the receivers.
2020-10-15 21:47:56 +02:00
Emeric Brun
b0c331f71f BUG/MINOR: proxy/log: frontend/backend and log forward names must differ
This patch disallow to use same name for a log forward section
and a frontend/backend section.
2020-10-08 08:53:26 +02:00
Emeric Brun
cbb7bf7dd1 MEDIUM: log: syslog TCP support on log forward section.
This patch re-introduce the "bind" statement on log forward
sections to handle syslog TCP listeners as defined in
rfc-6587.

As complement it introduce "maxconn", "backlog" and "timeout
client" statements to parameter those listeners.
2020-10-07 17:17:27 +02:00
Emeric Brun
a39ecbdac1 BUG/MINOR: proxy: inc req counter on new syslog messages.
Increase req counter instead of conn counter on
new syslog messages.

This should be backported on branches including the
syslog forward feature.
2020-10-07 17:17:27 +02:00
Willy Tarreau
5fc9328aa2 MINOR: tools: make str2sa_range() directly return the protocol
We'll need this so that it can return pointers to stacked protocol in
the future (for QUIC). In addition this removes a lot of tests for
protocol validity in the callers.

Some of them were checked further apart, or after a call to
str2listener() and they were simplified as well.

There's still a trick, we can fail to return a protocol in case the caller
accepts an fqdn for use later. This is what servers do and in this case it
is valid to return no protocol. A typical example is:

   server foo localhost:1111
2020-09-16 22:08:08 +02:00
Willy Tarreau
26ff5dabc0 MINOR: log-forward: use str2receiver() to parse the dgram-bind address
Thanks to this we don't need to specify "udp@" as it's implicitly a
datagram type listener that is expected, so any AF_INET/AF_INET4 address
will work.
2020-09-16 22:08:08 +02:00
Willy Tarreau
3baec249b1 MEDIUM: tools: make str2sa_range() only report AF_CUST_UDP on listeners
For now only listeners can make use of AF_CUST_UDP and it requires hacks
in the DNS and logsrv code to remap it to AF_INET. Make str2sa_range()
smarter by detecting that it's called for a listener and only set these
protocol families for listeners. This way we can get rid of the hacks.
2020-09-16 22:08:08 +02:00
Willy Tarreau
895992619d MINOR: log: detect LOG_TARGET_FD from the fd and not from the syntax
Now that we have the FD value reported we don't need to cheat and detect
"fd@" in the address, we can safely rely on the FD value.
2020-09-16 22:08:08 +02:00
Willy Tarreau
a93e5c7fae MINOR: tools: make str2sa_range() optionally return the fd
If a file descriptor was passed, we can optionally return it. This will
be useful for listening sockets which are both a pre-bound FD and a ready
socket.
2020-09-16 22:08:08 +02:00
Willy Tarreau
328199348b MINOR: tools: add several PA_O_* flags in str2sa_range() callers
These flags indicate whether the call is made to fill a bind or a server
line, or even just send/recv calls (like logs or dns). Some special cases
are made for outgoing FDs (e.g. pipes for logs) or socket FDs (e.g external
listeners), and there's a distinction between stream or dgram usage that's
expected to significantly help str2sa_range() proceed appropriately with
the input information. For now they are not used yet.
2020-09-16 22:08:08 +02:00
Willy Tarreau
8b0fa8f0ab MEDIUM: config: remove all checks for missing/invalid ports/ranges
Now that str2sa_range() checks for appropriate port specification, we
don't need to implement adhoc test cases in every call place, if the
result is valid, the conditions are met otherwise the error message is
appropriately filled.
2020-09-16 22:08:08 +02:00
Willy Tarreau
809587635e MINOR: tools: add several PA_O_PORT_* flags in str2sa_range() callers
These flags indicate what is expected regarding port specifications. Some
callers accept none, some need fixed ports, some have it mandatory, some
support ranges, and some take an offset. Each possibilty is reflected by
an option. For now they are not exploited, but the goal is to instrument
str2sa_range() to properly parse that.
2020-09-16 22:08:07 +02:00
Willy Tarreau
cd3a5591f6 MINOR: tools: make str2sa_range() take more options than just resolve
We currently have an argument to require that the address is resolved
but we'll soon add more, so let's turn it into a bit field. The old
"resolve" boolean is now PA_O_RESOLVE.
2020-09-16 22:08:07 +02:00
Willy Tarreau
a5b325f92c MINOR: protocol: add a real family for existing FDs
At some places (log fd@XXX, bind fd@XXX) we support using an explicit
file descriptor number, that is placed into the sockaddr for later use.
The problem is that till now it was done with an AF_UNSPEC family, which
is also used for other situations like missing info or rings (for logs).

Let's create an "official" family AF_CUST_EXISTING_FD for this case so
that we are certain the FD can be found in the address when it is set.
2020-09-16 22:08:07 +02:00
Willy Tarreau
b743661f04 REORG: listener: move the listener's proto to the receiver
The receiver is the one which depends on the protocol while the listener
relies on the receiver. Let's move the protocol there. Since there's also
a list element to get back to the listener from the proto list, this list
element (proto_list) was moved as well. For now when scanning protos, we
still see listeners which are linked by their rx.proto_list part.
2020-09-16 22:08:05 +02:00
Willy Tarreau
e1c4c80441 BUG/MINOR: log: gracefully handle the "udp@" address format for log servers
Commit 3835c0dcb ("MEDIUM: udp: adds minimal proto udp support for
message listeners.") introduced a problematic side effect in log server
address parser: if "udp@", "udp4@" or "udp6@" prefixes a log server's
address, the adress is passed as-is to the log server with a non-existing
family and fails like this when trying to send:

  [ALERT] 259/195708 (3474) : socket() failed in logger #1: Address family not supported by protocol (errno=97)

The problem is that till now there was no UDP family, so logs expect an
AF_INET family to be passed for UDP there.

This patch manually remaps AF_CUST_UDP4 and AF_CUST_UDP6 to their "tcp"
equivalent that the log server parser expects. No backport is needed.
2020-09-16 20:11:52 +02:00
Willy Tarreau
3b139e540a BUG/MEDIUM: log-forward: always quit on parsing errors
The err_code and goto were misplaced, causing a fatal parse error to be
ignored when parsing a UDP listener's address. No backport is needed.
2020-09-16 16:25:29 +02:00
Willy Tarreau
76aaa7f5b7 MEDIUM: log-forward: use "dgram-bind" instead of "bind" for the listener
The use of "bind" wasn't that wise but was temporary. The problem is that
it will not allow to coexist with tcp. Let's explicitly call it "dgram-bind"
so that datagram listeners are expected here, leaving some room for stream
listeners later. This is the only change.
2020-09-16 15:07:22 +02:00
Willy Tarreau
f9feec2813 BUG/MINOR: log-forward: fail on unknown keywords
The log-forward section silently ignores junk and unknown keywords, make
it fail! No backport is needed.
2020-09-16 15:04:33 +02:00
Ilya Shipitsin
6b79f38a7a CLEANUP: assorted typo fixes in the code and comments
This is 12th iteration of typo fixes
2020-07-31 11:18:07 +02:00
Emeric Brun
45c457a629 MINOR: log: adds counters on received syslog messages.
This patch adds a global counter of received syslog messages
and this one is exported on CLI "show info" as "CumRecvLogs".

This patch also updates internal conn counter and freq
of the listener and the proxy for each received log message to
prepare a further export on the "show stats".
2020-07-15 17:50:12 +02:00
Emeric Brun
12941c82d0 MEDIUM: log: adds log forwarding section.
Log forwarding:

It is possible to declare one or multiple log forwarding section,
haproxy will forward all received log messages to a log servers list.

log-forward <name>
  Creates a new log forwarder proxy identified as <name>.

bind <addr> [param*]
  Used to configure a log udp listener to receive messages to forward.
  Only udp listeners are allowed, address must be prefixed using
  'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters
  found in 5.1 paragraph but most of them are irrelevant for udp/syslog case.

log global
log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>]
    <facility> [<level> [<minlevel>]]
  Used to configure target log servers. See more details on proxies
  documentation.
  If no format specified, haproxy tries to keep the incoming log format.
  Configured facility is ignored, except if incoming message does not
  present a facility but one is mandatory on the outgoing format.
  If there is no timestamp available in the input format, but the field
  exists in output format, haproxy will use the local date.

  Example:
    global
       log stderr format iso local7

    ring myring
        description "My local buffer"
        format rfc5424
        maxlen 1200
        size 32764
        timeout connect 5s
        timeout server 10s
        # syslog tcp server
        server mysyslogsrv 127.0.0.1:514 log-proto octet-count

    log-forward sylog-loadb
        bind udp4@127.0.0.1:1514
        # all messages on stderr
        log global
        # all messages on local tcp syslog server
        log ring@myring local0
        # load balance messages on 4 udp syslog servers
        log 127.0.0.1:10001 sample 1:4 local0
        log 127.0.0.1:10002 sample 2:4 local0
        log 127.0.0.1:10003 sample 3:4 local0
        log 127.0.0.1:10004 sample 4:4 local0
2020-07-15 17:50:12 +02:00
Emeric Brun
54932b4408 MINOR: log: adds syslog udp message handler and parsing.
This patch introduce a new fd handler used to parse syslog
message on udp.

The parsing function returns level, facility and metadata that
can be immediatly reused to forward message to a log server.

This handler is enabled on udp listeners if proxy is internally set
to mode PR_MODE_SYSLOG
2020-07-15 17:50:12 +02:00
Emeric Brun
546488559a MEDIUM: log/sink: re-work and merge of build message API.
This patch merges build message code between sink and log
and introduce a new API based on struct ist array to
prepare message header with zero copy, targeting the
log forwarding feature.

Log format 'iso' and 'timed' are now avalaible on logs line.
A new log format 'priority' is also added.
2020-07-15 17:50:12 +02:00
Emeric Brun
2f4cc28e0f BUG/MEDIUM: log: issue mixing sampled to not sampled log servers.
A boolean was mistakenly declared 'static THREAD_LOCAL' causing
the probe of a log to a 'not sampled' log server conditionned by
the last evaluated 'sampled log' server test on the same thread.

This results to unpredictable drops of logs on 'not sampled'
log servers as soon a 'sampled' log server is declared.

This patch removes the static THREAD_LOCAL attribute from this
boolean, fixing the issue and allowing to mix 'sampled' and
'not sampled' servers.

This fix should be backported in any branches which includes
the log sampling feature.
2020-07-15 17:50:12 +02:00
Christopher Faulet
b4174d5a79 MINOR: log: Remove unused case statement during the log-format string parsing
Since the commit cd0d2ed6e ("MEDIUM: log-format: make the LF parser aware of
sample expressions' end"), the LF_STEXPR label in the last switch-case statement
at the end of the for loop in the parse_logformat_string() function cannot be
reached anymore.

This patch should fix the issue #723.
2020-07-07 14:28:43 +02:00
Emeric Brun
9f9b22c4f1 MINOR: log: add time second fraction field to rfc5424 log timestamp.
This patch adds the time second fraction in microseconds
as supported by the rfc.
2020-07-02 17:56:06 +02:00
Dragan Dosen
1e3b16f74f MINOR: log-format: allow to preserve spacing in log format strings
Now it's possible to preserve spacing everywhere except in "log-format",
"log-format-sd" and "unique-id-format" directives, where spaces are
delimiters and are merged. That may be useful when the response payload
is specified as a log format string by "lf-file" or "lf-string", or even
for headers or anything else.

In order to merge spaces, a new option LOG_OPT_MERGE_SPACES is applied
exclusively on options passed to function parse_logformat_string().

This patch fixes an issue #701 ("http-request return log-format file
evaluation altering spacing of ASCII output/art").
2020-07-02 10:11:44 +02:00
Dragan Dosen
2866acfb23 BUG/MEDIUM: log-format: fix possible endless loop in parse_logformat_string()
This patch adds a missing break to end the loop in case when '%[' is not
properly closed with ']'.

The issue has been introduced with commit cd0d2ed ("MEDIUM: log-format:
make the LF parser aware of sample expressions' end").
2020-07-01 06:30:50 +02:00
Willy Tarreau
b2551057af CLEANUP: include: tree-wide alphabetical sort of include files
This patch fixes all the leftovers from the include cleanup campaign. There
were not that many (~400 entries in ~150 files) but it was definitely worth
doing it as it revealed a few duplicates.
2020-06-11 10:18:59 +02:00
Willy Tarreau
dfd3de8826 REORG: include: move stream.h to haproxy/stream{,-t}.h
This one was not easy because it was embarking many includes with it,
which other files would automatically find. At least global.h, arg.h
and tools.h were identified. 93 total locations were identified, 8
additional includes had to be added.

In the rare files where it was possible to finalize the sorting of
includes by adjusting only one or two extra lines, it was done. But
all files would need to be rechecked and cleaned up now.

It was the last set of files in types/ and proto/ and these directories
must not be reused anymore.
2020-06-11 10:18:58 +02:00
Willy Tarreau
aeed4a85d6 REORG: include: move log.h to haproxy/log{,-t}.h
The current state of the logging is a real mess. The main problem is
that almost all files include log.h just in order to have access to
the alert/warning functions like ha_alert() etc, and don't care about
logs. But log.h also deals with real logging as well as log-format and
depends on stream.h and various other things. As such it forces a few
heavy files like stream.h to be loaded early and to hide missing
dependencies depending where it's loaded. Among the missing ones is
syslog.h which was often automatically included resulting in no less
than 3 users missing it.

Among 76 users, only 5 could be removed, and probably 70 don't need the
full set of dependencies.

A good approach would consist in splitting that file in 3 parts:
  - one for error output ("errors" ?).
  - one for log_format processing
  - and one for actual logging.
2020-06-11 10:18:58 +02:00
Willy Tarreau
5e539c9b8d REORG: include: move stream_interface.h to haproxy/stream_interface{,-t}.h
Almost no changes, removed stdlib and added buf-t and connection-t to
the types to avoid a warning.
2020-06-11 10:18:58 +02:00
Willy Tarreau
209108dbbd REORG: include: move ssl_sock.h to haproxy/ssl_sock{,-t}.h
Almost nothing changed, just moved a static inline at the end and moved
an export from the types to the main file.
2020-06-11 10:18:58 +02:00
Willy Tarreau
83487a833c REORG: include: move cli.h to haproxy/cli{,-t}.h
Almost no change except moving the cli_kw struct definition after the
defines. Almost all users had both types&proto included, which is not
surprizing since this code is old and it used to be the norm a decade
ago. These places were cleaned.
2020-06-11 10:18:58 +02:00
Willy Tarreau
3f0f82e7a9 REORG: move applet.h to haproxy/applet{,-t}.h
The type file was slightly tidied. The cli-specific APPCTX_CLI_ST1_* flag
definitions were moved to cli.h. The type file was adjusted to include
buf-t.h and not the huge buf.h. A few call places were fixed because they
did not need this include.
2020-06-11 10:18:58 +02:00
Willy Tarreau
f268ee8795 REORG: include: split global.h into haproxy/global{,-t}.h
global.h was one of the messiest files, it has accumulated tons of
implicit dependencies and declares many globals that make almost all
other file include it. It managed to silence a dependency loop between
server.h and proxy.h by being well placed to pre-define the required
structs, forcing struct proxy and struct server to be forward-declared
in a significant number of files.

It was split in to, one which is the global struct definition and the
few macros and flags, and the rest containing the functions prototypes.

The UNIX_MAX_PATH definition was moved to compat.h.
2020-06-11 10:18:58 +02:00
Willy Tarreau
e6ce10be85 REORG: include: move sample.h to haproxy/sample{,-t}.h
This one is particularly tricky to move because everyone uses it
and it depends on a lot of other types. For example it cannot include
arg-t.h and must absolutely only rely on forward declarations to avoid
dependency loops between vars -> sample_data -> arg. In order to address
this one, it would be nice to split the sample_data part out of sample.h.
2020-06-11 10:18:58 +02:00
Willy Tarreau
762d7a5117 REORG: include: move frontend.h to haproxy/frontend.h
There was no type file for this one, it only contains frontend_accept().
2020-06-11 10:18:57 +02:00
Willy Tarreau
ba2f73d40e REORG: include: move sink.h to haproxy/sink{,-t}.h
The sink files could be moved with almost no change at since they
didn't rely on anything fancy. ssize_t required sys/types.h and
thread.h was needed for the locks.
2020-06-11 10:18:57 +02:00
Willy Tarreau
d2ad57c352 REORG: include: move ring to haproxy/ring{,-t}.h
Some includes were wrong in the type definition but beyond this no
change was needed.
2020-06-11 10:18:57 +02:00
Willy Tarreau
0f6ffd652e REORG: include: move fd.h to haproxy/fd{,-t}.h
A few includes were missing in each file. A definition of
struct polled_mask was moved to fd-t.h. The MAX_POLLERS macro was
moved to defaults.h

Stdio used to be silently inherited from whatever path but it's needed
for list_pollers() which takes a FILE* and which can thus not be
forward-declared.
2020-06-11 10:18:57 +02:00
Willy Tarreau
48fbcae07c REORG: tools: split common/standard.h into haproxy/tools{,-t}.h
And also rename standard.c to tools.c. The original split between
tools.h and standard.h dates from version 1.3-dev and was mostly an
accident. This patch moves the files back to what they were expected
to be, and takes care of not changing anything else. However this
time tools.h was split between functions and types, because it contains
a small number of commonly used macros and structures (e.g. name_desc)
which in turn cause the massive list of includes of tools.h to conflict
with the callers.

They remain the ugliest files of the whole project and definitely need
to be cleaned and split apart. A few types are defined there only for
functions provided there, and some parts are even OS-specific and should
move somewhere else, such as the symbol resolution code.
2020-06-11 10:18:57 +02:00
Willy Tarreau
cd72d8c981 REORG: include: split common/http.h into haproxy/http{,-t}.h
So the enums and structs were placed into http-t.h and the functions
into http.h. This revealed that several files were dependeng on http.h
but not including it, as it was silently inherited via other files.
2020-06-11 10:18:57 +02:00
Willy Tarreau
92b4f1372e REORG: include: move time.h from common/ to haproxy/
This one is included almost everywhere and used to rely on a few other
.h that are not needed (unistd, stdlib, standard.h). It could possibly
make sense to split it into multiple parts to distinguish operations
performed on timers and the internal time accounting, but at this point
it does not appear much important.
2020-06-11 10:18:56 +02:00
Willy Tarreau
d678805783 REORG: include: move version.h to haproxy/
Few files were affected. The release scripts was updated.
2020-06-11 10:18:56 +02:00
Willy Tarreau
4c7e4b7738 REORG: include: update all files to use haproxy/api.h or api-t.h if needed
All files that were including one of the following include files have
been updated to only include haproxy/api.h or haproxy/api-t.h once instead:

  - common/config.h
  - common/compat.h
  - common/compiler.h
  - common/defaults.h
  - common/initcall.h
  - common/tools.h

The choice is simple: if the file only requires type definitions, it includes
api-t.h, otherwise it includes the full api.h.

In addition, in these files, explicit includes for inttypes.h and limits.h
were dropped since these are now covered by api.h and api-t.h.

No other change was performed, given that this patch is large and
affects 201 files. At least one (tools.h) was already freestanding and
didn't get the new one added.
2020-06-11 10:18:42 +02:00
Emeric Brun
fa9d780119 BUG/MEDIUM: logs: fix trailing zeros on log message.
This patch removes all trailing LFs and Zeros from
log messages. Previously only the last LF was removed.

It's a regression from e8ea0ae6f6 "BUG/MINOR: logs:
prevent double line returns in some events."

This should fix github issue #654
2020-05-28 15:30:51 +02:00
Emeric Brun
99c453df9d MEDIUM: ring: new section ring to declare custom ring buffers.
It is possible to globally declare ring-buffers, to be used as target for log
servers or traces.

ring <ringname>
  Creates a new ring-buffer with name <ringname>.

description <text>
  The descritpition is an optional description string of the ring. It will
  appear on CLI. By default, <name> is reused to fill this field.

format <format>
  Format used to store events into the ring buffer.

  Arguments:
    <format> is the log format used when generating syslog messages. It may be
             one of the following :

      iso     A message containing only the ISO date, followed by the text.
              The PID, process name and system name are omitted. This is
              designed to be used with a local log server.

      raw     A message containing only the text. The level, PID, date, time,
              process name and system name are omitted. This is designed to be
              used in containers or during development, where the severity
              only depends on the file descriptor used (stdout/stderr). This
              is the default.

      rfc3164 The RFC3164 syslog message format. This is the default.
              (https://tools.ietf.org/html/rfc3164)

      rfc5424 The RFC5424 syslog message format.
              (https://tools.ietf.org/html/rfc5424)

      short   A message containing only a level between angle brackets such as
              '<3>', followed by the text. The PID, date, time, process name
              and system name are omitted. This is designed to be used with a
              local log server. This format is compatible with what the systemd
              logger consumes.

      timed   A message containing only a level between angle brackets such as
              '<3>', followed by ISO date and by the text. The PID, process
              name and system name are omitted. This is designed to be
              used with a local log server.

maxlen <length>
  The maximum length of an event message stored into the ring,
  including formatted header. If an event message is longer than
  <length>, it will be truncated to this length.

size <size>
  This is the optional size in bytes for the ring-buffer. Default value is
  set to BUFSIZE.

  Example:
    global
        log ring@myring local7

    ring myring
        description "My local buffer"
        format rfc3164
        maxlen 1200

Note: ring names are resolved during post configuration processing.
2020-05-26 08:03:15 +02:00
Christopher Faulet
3b967c1210 MINOR: http-htx/proxy: Add http-error directive using http return syntax
The http-error directive can now be used instead of errorfile to define an error
message in a proxy section (including default sections). This directive uses the
same syntax that http return rules. The only real difference is the limitation
on status code that may be specified. Only status codes supported by errorfile
directives are supported for this new directive. Parsing of errorfile directive
remains independent from http-error parsing. But functionally, it may be
expressed in terms of http-errors :

  errorfile <status> <file> ==> http-errror status <status> errorfile <file>
2020-05-20 18:27:14 +02:00
Emeric Brun
e709e1e777 MEDIUM: logs: buffer targets now rely on new sink_write
Before this path, they rely directly on ring_write bypassing
a part of the sink API.

Now the maxlen parameter of the log will apply only on the text
message part (and not the header, for this you woud prefer
to use the maxlen parameter on the sink/ring).

sink_write prototype was also reviewed to return the number of Bytes
written to be compliant with the other write functions.
2020-05-19 11:04:11 +02:00
Emeric Brun
bd163817ed MEDIUM: sink: build header in sink_write for log formats
This patch extends the sink_write prototype and code to
handle the rfc5424 and rfc3164 header.

It uses header building tools from log.c. Doing this some
functions/vars have been externalized.

facility and minlevel have been removed from the struct sink
and passed to args at sink_write because they depends of the log
and not of the sink (they remained unused by rest of the code
until now).
2020-05-19 11:04:11 +02:00
Emeric Brun
9e8ea0ae6f BUG/MINOR: logs: prevent double line returns in some events.
Historically some messages used to already contain the trailing LF but
not all, and __do_send_log adds a new one in needed cases. It also does
trim a trailing LF in certain cases while computing the max message
length, as a result of subtracting 1 to the available room in the
destination buffer. But the way it's done is wrong since some messages
still contain it.

So the code was fixed to always trim the trailing LF from messages if
present, and then only subtract 1 from the destination buffer room
instead of the size..

Note: new sink API is not designed to receive a trailing LF on
event messages

This could be backported to relevant stable versions with particular
care since the logic of the code changed a bit since 1.6 and there
may be other locations that need to be adjusted.
2020-05-19 10:59:53 +02:00
Damien Claisse
57c8eb939d MINOR: log: Add "Tu" timer
It can be sometimes useful to measure total time of a request as seen
from an end user, including TCP/TLS negotiation, server response time
and transfer time. "Tt" currently provides something close to that, but
it also takes client idle time into account, which is problematic for
keep-alive requests as idle time can be very long. "Ta" is also not
sufficient as it hides TCP/TLS negotiationtime. To improve that, introduce
a "Tu" timer, without idle time and everything else. It roughly estimates
time spent time spent from user point of view (without DNS resolution
time), assuming network latency is the same in both directions.
2020-04-28 16:30:13 +02:00
Christopher Faulet
d2236cdcc4 MINOR: log: Don't systematically set LW_REQ when a sample expr is added
When a log-format string is parsed, if a sample fetch is found, the flag LW_REQ
is systematically added on the proxy. Unfortunately, this produce a warning
during HAProxy start-up when a log-format string is used for a tcp-check send
rule. Now this flag is only added if the parsed sample fetch depends on HTTP
information.
2020-04-27 09:39:37 +02:00
Christopher Faulet
5f940703b3 MINOR: log: Don't depends on a stream to process samples in log-format string
When a log-format string is evaluated, there is no reason to process sample
fetches only when a stream is defined. Several sample fetches are available
outside the stream scope. All others should handle calls without stream. This
patch is mandatory to support log-format string in tcp-check rules.
2020-04-27 09:39:37 +02:00
Ilya Shipitsin
ae40dbc93c CLEANUP: log: fix comment of parse_logformat_string()
"fmt" is passed to parse_logformat_string, adjust comment
accordingly
2020-04-21 10:52:25 +02:00
Ilya Shipitsin
856aabcda5 CLEANUP: assorted typo fixes in the code and comments
This is 8th iteration of typo fixes
2020-04-17 09:37:36 +02:00
Willy Tarreau
bb86986253 MINOR: init: report the haproxy version and executable path once on errors
If haproxy fails to start and emits an alert, then it can be useful
to have it also emit the version and the path used to load it. Some
users may be mistakenly launching the wrong binary due to a misconfigured
PATH variable and this will save them some troubleshooting time when it
reports that some keywords are not understood.

What we do here is that we *try* to extract the binary name from the
AUX vector on glibc, and we report this as a NOTICE tag before the
very first alert is emitted.
2020-04-16 10:52:41 +02:00
Willy Tarreau
bebd212064 MINOR: init: report in "haproxy -c" whether there were warnings or not
This helps quickly checking if the config produces any warning. For
this we reuse the "warned" bit field to add a new WARN_ANY bit that is
set by ha_warning(). The rest of the bit field was also cleaned from
unused bits.
2020-04-15 16:42:00 +02:00
Tim Duesterhus
cf6e0c8a83 MEDIUM: proxy_protocol: Support sending unique IDs using PPv2
This patch adds the `unique-id` option to `proxy-v2-options`. If this
option is set a unique ID will be generated based on the `unique-id-format`
while sending the proxy protocol v2 header and stored as the unique id for
the first stream of the connection.

This feature is meant to be used in `tcp` mode. It works on HTTP mode, but
might result in inconsistent unique IDs for the first request on a keep-alive
connection, because the unique ID for the first stream is generated earlier
than the others.

Now that we can send unique IDs in `tcp` mode the `%ID` log variable is made
available in TCP mode.
2020-03-13 17:26:43 +01:00
Tim Duesterhus
a17e66289c MEDIUM: stream: Make the unique_id member of struct stream a struct ist
The `unique_id` member of `struct stream` now is a `struct ist`.
2020-03-05 20:21:58 +01:00
Tim Duesterhus
2825b4b0ca MINOR: stream: Use stream_generate_unique_id
This patch replaces the ad-hoc generation of stream's `unique_id` values
by calls to `stream_generate_unique_id`.
2020-03-05 07:23:00 +01:00
Willy Tarreau
908071171b BUILD: general: always pass unsigned chars to is* functions
The isalnum(), isalpha(), isdigit() etc functions from ctype.h are
supposed to take an int in argument which must either reflect an
unsigned char or EOF. In practice on some platforms they're implemented
as macros referencing an array, and when passed a char, they either cause
a warning "array subscript has type 'char'" when lucky, or cause random
segfaults when unlucky. It's quite unconvenient by the way since none of
them may return true for negative values. The recent introduction of
cygwin to the list of regularly tested build platforms revealed a lot
of breakage there due to the same issues again.

So this patch addresses the problem all over the code at once. It adds
unsigned char casts to every valid use case, and also drops the unneeded
double cast to int that was sometimes added on top of it.

It may be backported by dropping irrelevant changes if that helps better
support uncommon platforms. It's unlikely to fix bugs on platforms which
would already not emit any warning though.
2020-02-25 08:16:33 +01:00
Willy Tarreau
cd0d2ed6ee MEDIUM: log-format: make the LF parser aware of sample expressions' end
For a very long time it used to be impossible to pass a closing square
bracket as a valid character in argument to a sample fetch function or
to a converter because the LF parser used to stop on the first such
character found and to pass what was between the first '[' and the first
']' to sample_parse_expr().

This patch addresses this by passing the whole string to sample_parse_expr()
which is the only one authoritative to indicate the first character that
does not belong to the expression. The LF parser then verifies it matches
a ']' or fails. As a result it is finally possible to write rules such as
the following, which is totally valid an unambigous :

    http-request redirect location %[url,regsub([.:/?-],!,g)]
                                                |-----| | |
                                                  arg1  | `---> arg3
                                                        `-----> arg2
                                         |-----------------|
                                              converter
                                     |---------------------|
                                        sample expression
                                   |------------------------|
                                         log-format tag
2020-02-14 19:02:06 +01:00
Willy Tarreau
e3b57bf92f MINOR: sample: make sample_parse_expr() able to return an end pointer
When an end pointer is passed, instead of complaining that a comma is
missing after a keyword, sample_parse_expr() will silently return the
pointer to the current location into this return pointer so that the
caller can continue its parsing. This will be used by more complex
expressions which embed sample expressions, and may even permit to
embed sample expressions into arguments of other expressions.
2020-02-14 19:02:06 +01:00
Willy Tarreau
51013e82d4 BUG/MINOR: log: fix minor resource leaks on logformat error path
As reported by Ilya in issue #392, Coverity found that we're leaking
allocated strings on error paths in parse_logformat(). Let's use a
proper exit label for failures instead of seeding return 0 everywhere.

This should be backported to all supported versions.
2019-12-11 12:05:39 +01:00
Willy Tarreau
869efd5eeb BUG/MINOR: log: make "show startup-log" use a ring buffer instead
The copy of the startup logs used to rely on a re-allocated memory area
on the fly, that would attempt to be delivered at once over the CLI. But
if it's too large (too many warnings) it will take time to start up, and
may not even show up on the CLI as it doesn't fit in a buffer.

The ring buffer infrastructure solves all this with no more code, let's
switch to this instead. It simply requires a parsing function to attach
the ring via ring_attach_cli() and all the rest is automatically handled.

Initially this was imagined as a code cleanup, until a test with a config
involving 100k backends and just one occurrence of
"load-server-state-from-file global" in the defaults section took approx
20 minutes to parse due to the O(N^2) cost of concatenating the warnings
resulting in ~1 TB of data to be copied, while it took only 0.57s with
the ring.

Ideally this patch should be backported to 2.0 and 1.9, though it relies
on the ring infrastructure which will then also need to be backported.
Configs able to trigger the bug are uncommon, so another workaround for
older versions without backporting the rings would consist in simply
limiting the size of the error message in print_message() to something
always printable, which will only return the first errors.
2019-11-15 15:50:16 +01:00
Christopher Faulet
5c6fefc8eb MINOR: log: Provide a function to emit a log for an application
Application is a generic term here. It is a modules which handle its own log
server list, with no dependency on a proxy. Such applications can now call the
function app_log() to log messages, passing a log server list and a tag as
parameters. Internally, the function __send_log() has been adapted accordingly.
2019-09-17 10:18:54 +02:00
Willy Tarreau
c046d167e4 MEDIUM: log: add support for logging to a ring buffer
Now by prefixing a log server with "ring@<name>" it's possible to send
the logs to a ring buffer. One nice thing is that it allows multiple
sessions to consult the logs in real time in parallel over the CLI, and
without requiring file system access. At the moment, ring0 is created as
a default sink for tracing purposes and is available. No option is
provided to create new rings though this is trivial to add to the global
section.
2019-08-30 15:24:59 +02:00
Willy Tarreau
f3dc30f6de MINOR: log: add a target type instead of hacking the address family
Instead of detecting an AF_UNSPEC address family for a log server and
to deduce a file descriptor, let's create a target type field and
explicitly mention that the socket is of type FD.
2019-08-30 15:07:25 +02:00
Willy Tarreau
d52a7f8c8d MEDIUM: log: use the new generic fd_write_frag_line() function
When logging to a file descriptor, we'd rather use the unified
fd_write_frag_line() which uses the FD's lock than perform the
writev() ourselves and use a per-server lock, because if several
loggers point to the same output (e.g. stdout) they are still
not locked and their logs may interleave. The function above
instead relies on the fd's lock so this is safer and will even
protect against concurrent accesses from other areas (e.g traces).
The function also deals with the FD's non-blocking mode so we do
not have to keep specific code for this anymore in the logs.
2019-08-30 15:07:25 +02:00
Willy Tarreau
7e9776ad7b MINOR: fd/log/sink: make the non-blocking initialization depend on the initialized bit
Logs and sinks were resorting to dirty hacks to initialize an FD to
non-blocking mode. Now we have a bit for this in the fd tab so we can
do it on the fly on first use of the file descriptor. Previously it was
set per log server by writing value 1 to the port, or during a sink
initialization regardless of the usage of the fd.
2019-08-30 15:07:25 +02:00
Willy Tarreau
9fbcb7e2e9 BUG/MINOR: log: make sure writev() is not interrupted on a file output
Since 1.9 we support sending logs to various non-blocking outputs like
stdou/stderr or flies, by using writev() which guarantees that it only
returns after having written everything or nothing. However the syscall
may be interrupted while doing so, and this is visible when writing to
a tty during debug sessions, as some logs occasionally appear interleaved
if an xterm or SSH connection is not very fast. Performance here is not a
critical concern, log correctness is. Let's simply take the logger's lock
around the writev() call to prevent multiple senders from stepping onto
each other's toes.

This may be backported to 2.0 and 1.9.
2019-07-26 15:46:18 +02:00
Willy Tarreau
6c6365f455 MINOR: log: use conn->{src,dst} instead of conn->addr.{from,to}
This is used to retrieve the addresses to be logged (client, frontend,
backend, server). In all places the validity check was already performed.
2019-07-19 13:50:09 +02:00
Willy Tarreau
8fa9984a17 MINOR: log: use conn_get_{dst,src}() to retrieve the cli/frt/bck/srv/ addresses
This also allows us to check that the operation succeeded without
logging whatever remained in the memory area in case of failure.
2019-07-19 13:50:09 +02:00
Christopher Faulet
711ed6ae4a MAJOR: http: Remove the HTTP legacy code
First of all, all legacy HTTP analyzers and all functions exclusively used by
them were removed. So the most of the functions in proto_http.{c,h} were
removed. Only functions to deal with the HTTP transaction have been kept. Then,
http_msg and hdr_idx modules were entirely removed. And finally the structure
http_msg was lightened of all its useless information about the legacy HTTP. The
structure hdr_ctx was also removed because unused now, just like unused states
in the enum h1_state. Note that the memory pool "hdr_idx" was removed and
"http_txn" is now smaller.
2019-07-19 09:24:12 +02:00
Tim Duesterhus
2164800c1b BUG/MINOR: log: Detect missing sampling ranges in config
Consider a config like:

    global
    	log 127.0.0.1:10001 sample :10 local0

No sampling ranges are given here, leading to NULL being passed
as the first argument to qsort.

This configuration does not make sense anyway, a log without ranges
would never log. Thus output an error if no ranges are given.

This bug was introduced in d95ea2897e.
This fix must be backported to HAProxy 2.0.
2019-06-26 11:15:49 +02:00
Willy Tarreau
1bfd6020ce MINOR: logs: use the new bitmap functions instead of fd_sets for encoding maps
The fd_sets we've been using in the log encoding functions are not portable
and were shown to break at least under Cygwin. This patch gets rid of them
in favor of the new bitmap functions. It was verified with the config below
that the log output was exactly the same before and after the change :

    defaults
        mode http
        option httplog
        log stdout local0
        timeout client 1s
        timeout server 1s
        timeout connect 1s

    frontend foo
        bind :8001
        capture request header chars len 255

    backend bar
        option httpchk "GET" "/" "HTTP/1.0\r\nchars: \x01\x02\x03\x04\x05\x06\x07\x08\x09\x0b\x0c\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f\x20\x21\x22\x23\x24\x25\x26\x27\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f\x30\x31\x32\x33\x34\x35\x36\x37\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f\x40\x41\x42\x43\x44\x45\x46\x47\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f\x50\x51\x52\x53\x54\x55\x56\x57\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f\x60\x61\x62\x63\x64\x65\x66\x67\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f\x70\x71\x72\x73\x74\x75\x76\x77\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff"
        server foo 127.0.0.1:8001 check
2019-06-07 11:13:24 +02:00
Willy Tarreau
082b62828d BUG/MEDIUM: init/threads: provide per-thread alloc/free function callbacks
We currently have the ability to register functions to be called early
on thread creation and at thread deinitialization. It turns out this is
not sufficient because certain such functions may use resources that are
being allocated by the other ones, thus creating a race condition depending
only on the linking order. For example the mworker needs to register a
file descriptor while the pollers will reallocate the fd_updt[] array.
Similarly logs and trashes may be used by some init functions while it's
unclear whether they have been deduplicated.

The same issue happens on deinit, if the fd_updt[] or trash is released
before some functions finish to use them, we'll get into trouble.

This patch creates a couple of early and late callbacks for per-thread
allocation/freeing of resources. A few init functions were moved there,
and the fd init code was split between the two (since it used to both
allocate and initialize at once). This way the init/deinit sequence is
expected to be safe now.

This patch should be backported to 1.9 as at least the trash/log issue
seems to be present. The run_thread_poll_loop() code is a bit different
there as the mworker is not a callback, but it will have no effect and
it's enough to drop the mworker changes.

This bug was reported by Ilya Shipitsin in github issue #104.
2019-05-22 14:59:08 +02:00
Frdric Lcaille
90a10aeb65 BUG/MINOR: log: Wrong log format initialization.
This patch fixes an issue introduced by 0bad840b commit
"MINOR: log: Extract some code to send syslog messages" which leaded
to wrong log format variable initializations at least for "short" and "raw" format.
This commit skipped the cases where even if passed to __do_send_log(), the
syslog tag and syslog pid string must not be used to format the log message
with "short" and "raw". This is done iniatilizing "tag_max" and "pid_max"
variables (the lengths of the tag and pid strings) to 0, then updating to them to
the length of the tag and pid strings passed as variables to __do_send_log()
depending on the log format and in every cases using this length for the iovec
variable used to send() the log.

This bug is specific to 2.0.
2019-05-14 11:12:00 +02:00
Willy Tarreau
c125cef6da CLEANUP: ssl: make inclusion of openssl headers safe
It's always a pain to have to stuff lots of #ifdef USE_OPENSSL around
ssl headers, it even results in some of them appearing in a random order
and multiple times just to benefit form an existing ifdef block. Let's
make these headers safe for inclusion when USE_OPENSSL is not defined,
they now perform the test themselves and do nothing if USE_OPENSSL is
not defined. This allows to remove no less than 8 such ifdef blocks
and make include blocks more readable.
2019-05-10 09:58:43 +02:00
Willy Tarreau
55e2f5ad14 BUG/MINOR: logs/threads: properly split the log area upon startup
If logs were emitted before creating the threads, then the dataptr pointer
keeps a copy of the end of the log header. Then after the threads are
created, the headers are reallocated for each thread. However the end
pointer was not reset until the end of the first second, which may result
in logs emitted by multiple threads during the first second to be mangled,
or possibly in some cases to use a memory area that was reused for something
else. The fix simply consists in reinitializing the end pointers immediately
when the threads are created.

This fix must be backported to 1.9 and 1.8.
2019-05-05 10:16:13 +02:00
Dragan Dosen
61302da0e7 BUG/MINOR: log: properly free memory on logformat parse error and deinit()
This patch may be backported to all supported versions.
2019-05-02 10:05:07 +02:00
Frdric Lcaille
d803e475e5 MINOR: log: Enable the log sampling and load-balancing feature.
This patch implements the sampling and load-balancing of log servers configured
with "sample" new keyword implemented by this commit:
    'MINOR: log: Add "sample" new keyword to "log" lines'.
As the list of ranges used to sample the log to balance is ordered, we only
have to maintain ->curr_idx member of smp_info struct which is the index of
the sample and check if it belongs or not to the current range to decide if we
must send it to the log server or not.
2019-04-30 09:25:09 +02:00
Frdric Lcaille
d95ea2897e MINOR: log: Add "sample" new keyword to "log" lines.
This patch implements the parsing of "sample" new optional keyword for "log" lines
to be able to sample and balance the load of log messages between serveral log
destinations declared by "log" lines. This keyword must be followed by a list of
comma seperated ranges of indexes numbered from 1 to define the samples to be used
to balance the load of logs to send. This "sample" keyword must be used on "log" lines
obviously before the remaining optional ones without keyword. The list of ranges
must be followed by a colon character to separate it from the log sampling size.

With such following configuration declarations:

   log stderr local0
   log 127.0.0.1:10001 sample 2-3,8-11:11 local0
   log 127.0.0.2:10002 sample 5:5 local0

in addition to being sent to stderr, about the second "log" line, every 11 logs
the logs #2 up to #3 would be sent to 127.0.0.1:10001, then #8 up tp #11 four
logs would be sent to the same log server and so on periodically. Logs would be
sent to 127.0.0.2:100002 every 5 logs.

It is also possible to define the size of the sample with a value different of
the maximum of the high limits of the ranges, for instance as follows:

   log 127.0.0.1:10001 sample 2-3,8-11:15 local0

as before the two logs #2 and #3 would be sent to 127.0.0.1:10001, then #8
up tp #11 logs, but in this case here, this would be done periodically every 15
messages.

Also note that the ranges must not overlap each others. This is to ease the
way the logs are periodically sent.
2019-04-30 09:25:09 +02:00
Frdric Lcaille
0bad840b4d MINOR: log: Extract some code to send syslog messages.
This patch extracts the code of __send_log() responsible of sending a syslog
message to a syslog destination represented as a logsrv struct to define
__do_send_log() function. __send_log() calls __do_send_log() for each syslog
destination of a proxy after having prepared some of its parameters.
2019-04-23 14:16:51 +02:00
Robin H. Johnson
543d4507ca MINOR: skip get_gmtime where tm is unused
For LOG_FMT_TS (%Ts), the tm variable is not used, so save some cycles
on the call to get_gmtime.

Backport: 1.9 1.8
Signed-off-by: Robin H. Johnson <rjohnson@digitalocean.com>
2019-04-11 14:58:32 +02:00
Radek Zajic
594c456d14 BUG/MINOR: log: properly format IPv6 address when LOG_OPT_HEXA modifier is used.
In lf_ip(), when LOG_OPT_HEXA modifier is used, there is a code to format the
IP address as a hexadecimal string. This code does not properly handle cases
when the IP address is IPv6. In such case, the code only prints `00000000`.

This patch adds support for IPv6. For legacy IPv4, the format remains
unchanged. If IPv6 socket is used to accept IPv6 connection, the full IPv6
address is returned. For example, IPv6 localhost, ::1, is printed as
00000000000000000000000000000001.

If IPv6 socket accepts IPv4 connection, the IPv4 address is mapped by the
kernel into the IPv4-mapped-IPv6 address space (RFC4291, section 2.5.5.2)
and is formatted as such. For example, 127.0.0.1 becomes ::ffff:127.0.0.1,
which is printed as 00000000000000000000FFFF7F000001.

This should be backported to 1.9.
2019-03-22 17:31:18 +01:00
Olivier Houchard
d2ee3e7227 MEDIUM: logs: Use the new _HA_ATOMIC_* macros.
Use the new _HA_ATOMIC_* macros and add barriers where needed.
2019-03-11 17:02:38 +01:00
Olivier Houchard
7c49711d60 BUG/MEDIUM: logs: Only attempt to free startup_logs once.
deinit_log_buffers() can be called once per thread, however startup_logs
is common to all threads. So only attempt to free it once.

This should be backported to 1.9 and 1.8.
2019-03-07 14:59:34 +01:00
Willy Tarreau
b1d7b700bb BUG/MEDIUM: log: don't mark log FDs as non-blocking on terminals
With the new ability to log to a terminal, it's convenient to be able
to use "log stdout" in a config file, except that it now results in
setting the terminal to non-blocking mode, breaking every utility
relying on stdin afterwards. Since the only reason for logging to a
terminal is to debug, do not set the FD to non-blocking mode when it's
a terminal.

This fix must be backported to 1.9.
2019-01-02 20:12:02 +01:00
Willy Tarreau
a648399c90 BUG/MINOR: logs: leave startup-logs global and not per-thread
Commit f8188c6 ("MEDIUM: threads/logs: Make logs thread-safe") made logs
thread-local but it also made the copy of the startup-logs thread-local,
meaning that when threads are configured, upon startup the list of startup
logs appears to be empty. Let's just remove the THEAD_LOCAL directive
there, as the check for the startup period is already present.

This fix should be backported to 1.8.
2018-12-15 16:55:36 +01:00
Willy Tarreau
204e3f1fab BUG/MINOR: log: fix logging to both FD and IP
PiBa-NL reported an issue affecting logs when stdout is enabled at the
same time as an IP address. It does not affect FD and UNIX, but does
still affect multiple FDs. What happens is that the condition to detect
that the initialization was not made relies on the FD being -1, and in
this case the FD points to the *unique* FD used for AF_INET sockets, so
the configured socket used for outgoing logs over UDP gets overwritten
by the last configured FD. This is not appropriate, so instead we rely
on the sin_port part of the IPv4-mapped address to store the
initialization state for each FD.

This part deserves being significantly revamped, as IPv6 is still not
possible due to the way the FDs are managed, and inherited FDs are a
bit hackish.

Note that this patch relies on "MINOR: tools: preset the port of
fd-based "sockets" to zero" in order to operate properly.

No backport is needed.
2018-12-15 15:54:13 +01:00
Olivier Houchard
f90db445cd BUG/MEDIUM: log: Don't call sample_fetch_as_type if we don't have a stream.
In sess_build_logline(), don't attempt to call sample_fetch_as_type()
if we don't have a stream.

It used never to happen in the past before commit 09bb27c ("MEDIUM: log:
make sess_build_logline() support being called with no stream"). But now
it can happen when sess_log() is called from the lower layers (i.e. the
H2 mux got garbage when it was expecting a preface frame), and it reveals
that some sample fetch functions and some converter fnuctions which rely
on the stream don't test for its existence. For the sample fetch functions,
a durable solution is easy and would consist in adapting sample_process()
to verify the SMP_USE_* bits when the stream is not set. But for the
converters we don't have this option as they don't declare whether or not
they use a stream (which possibly is the real issue).

Thus for now let's disable sample_fetch_as_type() if a stream does not
exist, until it can be more accurately refined later.
2018-12-15 14:34:31 +01:00
Willy Tarreau
30925659ef CLEANUP: h1: remove some occurrences of unneeded h1.h inclusions
Several places where h1.h was included didn't need it at all since
they in fact relied on the legacy HTTP definitions.
2018-12-11 17:15:13 +01:00
Willy Tarreau
b6b3df3ed3 MEDIUM: initcall: use initcalls for a few initialization functions
signal_init(), init_log(), init_stream(), and init_task() all used to
only preset some values and lists. This needs to be done very early to
provide a reliable interface to all other users. The calls used to be
explicit in haproxy.c:init(). Now they're placed in initcalls at the
STG_PREPARE stage. The functions are not exported anymore.
2018-11-26 19:50:32 +01:00
Willy Tarreau
172f5ce948 MINOR: initcall: use initcalls for most post_{check,deinit} and per_thread*
Most calls to hap_register_post_check(), hap_register_post_deinit(),
hap_register_per_thread_init(), hap_register_per_thread_deinit() can
be done using initcalls and will not require a constructor anymore.
Let's create a set of simplified macros for this, called respectively
REGISTER_POST_CHECK, REGISTER_POST_DEINIT, REGISTER_PER_THREAD_INIT,
and REGISTER_PER_THREAD_DEINIT.

Some files were not modified because they wouldn't benefit from this
or because they conditionally register (e.g. the pollers).
2018-11-26 19:50:32 +01:00
Willy Tarreau
0108d90c6c MEDIUM: init: convert all trivial registration calls to initcalls
This switches explicit calls to various trivial registration methods for
keywords, muxes or protocols from constructors to INITCALL1 at stage
STG_REGISTER. All these calls have in common to consume a single pointer
and return void. Doing this removes 26 constructors. The following calls
were addressed :

- acl_register_keywords
- bind_register_keywords
- cfg_register_keywords
- cli_register_kw
- flt_register_keywords
- http_req_keywords_register
- http_res_keywords_register
- protocol_register
- register_mux_proto
- sample_register_convs
- sample_register_fetches
- srv_register_keywords
- tcp_req_conn_keywords_register
- tcp_req_cont_keywords_register
- tcp_req_sess_keywords_register
- tcp_res_cont_keywords_register
- flt_register_keywords
2018-11-26 19:50:32 +01:00
William Lallemand
9c56a22b20 MINOR: log: introduce ha_notice()
It's like ha_warning() or ha_alert() but with a NOTICE prefix.
2018-11-21 19:02:23 +01:00
Joseph Herlant
85b4059b82 CLEANUP: Fix typos in the log subsystem
Fix some misspells in the code comments of the log subsystem.
2018-11-18 22:26:42 +01:00
Willy Tarreau
db398435aa MINOR: stream-int: replace si_cant_put() with si_rx_room_{blk,rdy}()
Remaining calls to si_cant_put() were all for lack of room and were
turned to si_rx_room_blk(). A few places where SI_FL_RXBLK_ROOM was
cleared by hand were converted to si_rx_room_rdy().

The now unused si_cant_put() function was removed.
2018-11-18 21:41:50 +01:00
William Lallemand
a8b2671cf6 BUG/MEDIUM: log: don't CLOEXEC the inherited FDs
When configuring the logs with a FD and using the master worker, the FD
was closed upon a reload because it was configured with CLOEXEC. It
leads to using the wrong FD for the logs and to close them. Which is
unfortunate since the master rely on the FD left opened during a reload.

The fix is to stop doing a CLOEXEC when the FD is inherited.
No backport needed.
2018-11-13 19:32:45 +01:00
Willy Tarreau
c1b0645dac MEDIUM: log: add a new "raw" format
This format is pretty similar to the previous "short" format except
that it also removes the severity level. Thus only the raw message is
sent. This is suitable for use in containers, where only the raw
information is expected and where the severity is supposed to come
from the file descriptor used.
2018-11-12 18:37:55 +01:00
Willy Tarreau
e8746a08b2 MEDIUM: log: support a new "short" format
This format is meant to be used with local file descriptors. It emits
messages only prefixed with a level, removing all the process name,
system name, date and so on. It is similar to the printk() format used
on Linux. It's suitable to be sent to a local logger compatible with
systemd's output format.

Note that the facility is still required but not used, hence it is
suggested to use "daemon" to remind that it's a local logger.
Example :

    log stdout format short daemon          # send everything to stdout
    log stderr format short daemon notice   # send important events to stderr
2018-11-12 18:37:55 +01:00
Willy Tarreau
5a32ecc6cf MEDIUM: log: add support for logging to existing file descriptors
In certain situations it would be desirable to log to an existing file
descriptor, the most common case being a pipe between containers or
processes. The main issue with pipes is that using write() on them will
randomly truncate messages. But there is a trick. By using writev(), we
can atomically deliver or drop a message, which perfectly fits the
purpose. The only caveat is that large messages (4096 bytes on modern
operating systems) may be interleaved with messages from other processes
if using nbproc for example. In practice such messages are rare and most
of the time when users need such type of logging, the load is low enough
for a single process to be running so this is not really a problem.

This logging method thus uses unbuffered writev() calls and is uses more
CPU than if it used its own buffer with large writes at once, though this
is not a problem for moderate loads.

Logging to a file descriptor attached to a file also works with the side
effect that the process is significantly slowed down during disk accesses
and that it's not possible to rotate the file without restarting the
process. For this reason this option is not offered as a configuration
option, since it would confuse most users, but one could decide to
redirect haproxy's output to a file during debugging sessions. Two aliases
"stdout" and "stderr" are provided, but keep in mind that these are closed
by default in daemon mode.

When logging to a pipe or socket at a high enough rate, some logs will be
dropped and the number of dropped messages is reported in "show info".
2018-11-12 18:37:55 +01:00
Willy Tarreau
13ef773722 MINOR: log: report the number of dropped logs in the stats
It's easy to detect when logs on some paths are lost as sendmsg() will
return EAGAIN. This is particularly true when sending to /dev/log, which
often doesn't support a big logging capacity. Let's keep track of these
and report the total number of dropped messages in "show info".
2018-11-12 18:37:55 +01:00
Willy Tarreau
251fe34ca2 MINOR: log: slightly improve error message syntax on log failure
The error messages used to say something along "socket logger 2 failed"
or "sendmsg logger 2 failed" which are confusing. Let's rephrase this
"sendmsg() failed for logger 2".
2018-11-12 18:37:55 +01:00
Willy Tarreau
0cd3bd628a MINOR: stream-int: rename si_applet_{want|stop|cant}_{get|put}
It doesn't make sense to limit this code to applets, as any stream
interface can use it. Let's rename it by simply dropping the "applet_"
part of the name. No other change was made except updating the comments.
2018-11-11 10:18:37 +01:00
Willy Tarreau
9fa267dada MINOR: log: make sess_log() support sess=NULL
At many places in muxes we'll have to add tests to check if the
connection is front or back before deciding to log. Instead let's
centralize this test in sess_log() to simply do nothing when sess=NULL.
2018-10-12 16:58:01 +02:00
Willy Tarreau
1aaf324227 BUILD: log: silent build warnings due to unchecked __objt_{server,applet}
These ones are safe to use there since the same check is performed in
the switch/case they're used it. Let's use the unguarded versions
instead.
2018-09-20 11:42:15 +02:00
Willy Tarreau
e10cd48a83 REORG: http: move the log encoding tables to log.c
There are 3 tables in proto_http which are used exclusively by logs :
hdr_encode_map[], url_encode_map[] and http_encode_map[]. They indicate
what characters are safe to be emitted in logs depending on the part of
the message where they are placed. Let's move this to log.c, as well as
its initialization. It's worth noting that the rfc5424 map was already
initialized there.
2018-09-11 10:30:25 +02:00
Willy Tarreau
35b51c6e5b REORG: http: move the HTTP semantics definitions to http.h/http.c
It's a bit painful to have to deal with HTTP semantics for each protocol
version (H1 and H2), and working on the version-agnostic code further
emphasizes the problem.

This patch creates http.h and http.c which are agnostic to the version
in use, and which borrow a few parts from proto_http and from h1. For
example the once thought h1-specific h1_char_classes array is in fact
dictated by RFC7231 and is used to parse HTTP headers. A few changes
were made to a few files which were including proto_http.h while they
only needed http.h.

Certain string definitions pre-dated the introduction of indirect
strings (ist) so some were used to simplify the definition of the known
HTTP methods. The current lookup code saves 2 kB of a heavily used table
and is faster than the previous table based lookup (typ. 14 ns vs 16
before).
2018-09-11 10:30:25 +02:00
Willy Tarreau
5383935856 MINOR: log: provide a function to emit a log for a session
The new function sess_log() only needs a session to emit a log. It will
ignore the parts that depend on the stream. It is usable to emit a log
to report early errors in muxes. These ones will typically mention
"<BADREQ>" for the request and 0 for the HTTP status code.
2018-09-06 09:43:41 +02:00
Willy Tarreau
09bb27cdea MEDIUM: log: make sess_build_logline() support being called with no stream
Till now it was impossible to emit logs from the lower layers only because
a stream was mandatory. From now on it will at least be possible to emit a
log to report a bad request or some timings for example. When the stream
is null, sess_build_logline() will use default values and will extract the
timing information from the session just like stream_new() does, so the
resulting log line is perfectly valid.

The termination state will indicate a proxy error during the request phase
since it is the only realistic use for such a call with no stream.
2018-09-06 09:43:06 +02:00
Willy Tarreau
5cacab63e1 MINOR: log: use zero as the request counter if there is no stream
When s==NULL we don't have any assigned request counter. Ideally we
should proceed exactly like when a stream is initialized and assign
a unique value. For now we only place it into a local variable.
2018-09-05 20:01:23 +02:00
Willy Tarreau
b8bc52522c MINOR: log: keep a copy of s->flags early to avoid a dereference
By placing s->flags into a local variable we'll be able to force it new
values when s is NULL.
2018-09-05 20:01:23 +02:00
Willy Tarreau
02fdf4f77b MINOR: log: use NULL for the unique_id if there is no stream
Now s->unique_id is used as NULL (not set) if s==NULL.
2018-09-05 20:01:23 +02:00
Willy Tarreau
abd71a5c2e MINOR: log: don't check the stream-int's conn_retries if the stream is NULL
Let's simply forget the conn_retries when there is no stream since we
haven't tried to connect yet.
2018-09-05 20:01:23 +02:00
Willy Tarreau
e1809dfdaf MINOR: log: be sure not to dereference a null stream for a target
The supported targets are either a server or an applet, so both are
NULL if the stream is NULL.
2018-09-05 20:01:23 +02:00
Willy Tarreau
d4f9166f4e MINOR: log: do not dereference a null stream to access captures
If the stream is null, let's simply not check captures. That's already
done if there is no capture.
2018-09-05 20:01:23 +02:00
Willy Tarreau
2393c5b6a9 MINOR: log: keep a copy of the backend connection early in sess_build_logline()
This way we can avoid dereferencing a possibly inexisting stream.
2018-09-05 20:01:23 +02:00
Willy Tarreau
26ffa8544d CLEANUP: log: make the low_level lf_{ip,port,text,text_len} functions take consts
These ones were abusively relying on variables making it hard to integrate
with const arguments.
2018-09-05 20:01:23 +02:00
Willy Tarreau
372ac5abff MINOR: log: don't unconditionally pick log info from s->logs
We'll soon support s==NULL so let's use an intermediary variable for the
logs structure. For now it only points to s->logs but will support a local
variable as an alternative later.
2018-09-05 20:01:23 +02:00
Willy Tarreau
56a91dddc6 MINOR: log: make sess_build_logline() not dereference a NULL stream for txn
If the stream is NULL, the txn is NULL as well. This condition is already
handled everywhere else.
2018-09-05 20:01:23 +02:00
Willy Tarreau
a21c0e60d2 MINOR: log: make the backend fall back to the frontend when there's no stream
This is already what happens before the backend is assigned, except that
now we don't need to dereference a NULL stream to figure this.
2018-09-05 20:01:23 +02:00
Willy Tarreau
43c538eab6 MINOR: log: move the log code to sess_build_logline() to add extra arguments
The current build_logline() can only be used with valid streams, which
means it is not suitable for use from muxes. We start by moving it into
another more generic function which takes the session as an argument,
to avoid complexifying all the internal API for jsut a few use cases.
This new function is not supposed to be called directly from outside so
we'll be able to instrument it to support several calling conventions.

For now the behaviour and conditions remain unchanged.
2018-09-05 20:01:23 +02:00
Patrick Hemmer
ffe5e8c638 MINOR: stream: rename {srv,prx}_queue_size to *_queue_pos
The current name is misleading as it implies a queue size, but the value
instead indicates a position in the queue.
The value is only the queue size at the exact moment the element is enqueued.
Soon we will gain the ability to insert anywhere into the queue, upon which
clarity of the name is more important.
2018-08-10 15:04:14 +02:00
Willy Tarreau
83061a820e MAJOR: chunks: replace struct chunk with struct buffer
Now all the code used to manipulate chunks uses a struct buffer instead.
The functions are still called "chunk*", and some of them will progressively
move to the generic buffer handling code as they are cleaned up.
2018-07-19 16:23:43 +02:00
Willy Tarreau
843b7cbe9d MEDIUM: chunks: make the chunk struct's fields match the buffer struct
Chunks are only a subset of a buffer (a non-wrapping version with no head
offset). Despite this we still carry a lot of duplicated code between
buffers and chunks. Replacing chunks with buffers would significantly
reduce the maintenance efforts. This first patch renames the chunk's
fields to match the name and types used by struct buffers, with the goal
of isolating the code changes from the declaration changes.

Most of the changes were made with spatch using this coccinelle script :

  @rule_d1@
  typedef chunk;
  struct chunk chunk;
  @@
  - chunk.str
  + chunk.area

  @rule_d2@
  typedef chunk;
  struct chunk chunk;
  @@
  - chunk.len
  + chunk.data

  @rule_i1@
  typedef chunk;
  struct chunk *chunk;
  @@
  - chunk->str
  + chunk->area

  @rule_i2@
  typedef chunk;
  struct chunk *chunk;
  @@
  - chunk->len
  + chunk->data

Some minor updates to 3 http functions had to be performed to take size_t
ints instead of ints in order to match the unsigned length here.
2018-07-19 16:23:43 +02:00
Christopher Faulet
28ac099907 MINOR: log: Keep the ref when a log server is copied to avoid duplicate entries
With "log global" line, the global list of loggers are copied into the proxy's
struct. The list coming from the default section is also copied when a frontend
or a backend section is parsed. So it is possible to have duplicate entries in
the proxy's list. For instance, with this following config, all messages will be
logged twice:

    global
        log 127.0.0.1 local0 debug
        daemon

    defaults
        mode   http
        log    global
        option httplog

    frontend front-http
        log global
        bind *:8888
        default_backend back-http

    backend back-http
        server www 127.0.0.1:8000
2018-04-05 15:13:54 +02:00
Christopher Faulet
4b0b79dd56 MINOR: log: move 'log' keyword parsing in dedicated function
Now, the function parse_logsrv should be used to parse a "log" line. This
function will update the list of loggers passed in argument. It can release all
log servers when "no log" line was parsed (by the caller) or it can parse "log
global" or "log <address> ... " lines. It takes care of checking the caller
context (global or not) to prohibit "log global" usage in the global section.
2018-04-05 15:13:54 +02:00
Willy Tarreau
c98aebcdb8 MINOR: log: stop emitting alerts when it's not possible to write on the socket
This is a recurring pain when using certain unix domain sockets or when
sending to temporarily unroutable addresses, if the process remains in
the foreground, the console is full of error which it's impossible to
do anything about. It's even worse when the process is remote, or when
run from a serial console which will slow the whole process down. Let's
send them only once now to warn about a possible config issue, and not
pollute the system nor slow everything down.
2018-03-20 16:44:25 +01:00
Christopher Faulet
789691778f BUG/MEDIUM: mworker: Set FD_CLOEXEC flag on log fd
A log socket (UDP or UNIX) is opened by the master during its startup, when the
first log message is sent. So, to prevent FD leaks, we must ensure we correctly
close it during a reload. By setting FD_CLOEXEC bit on it, we are sure it will
be automatically closed it during a reload.

This patch must be backported in 1.8.
2017-12-19 14:03:30 +01:00
Willy Tarreau
bafbe01028 CLEANUP: pools: rename all pool functions and pointers to remove this "2"
During the migration to the second version of the pools, the new
functions and pool pointers were all called "pool_something2()" and
"pool2_something". Now there's no more pool v1 code and it's a real
pain to still have to deal with this. Let's clean this up now by
removing the "2" everywhere, and by renaming the pool heads
"pool_head_something".
2017-11-24 17:49:53 +01:00
Christopher Faulet
767a84bcc0 CLEANUP: log: Rename Alert/Warning in ha_alert/ha_warning 2017-11-24 17:19:12 +01:00
Olivier Houchard
9aaf778129 MAJOR: connection : Split struct connection into struct connection and struct conn_stream.
All the references to connections in the data path from streams and
stream_interfaces were changed to use conn_streams. Most functions named
"something_conn" were renamed to "something_cs" for this. Sometimes the
connection still is what matters (eg during a connection establishment)
and were not always renamed. The change is significant and minimal at the
same time, and was quite thoroughly tested now. As of this patch, all
accesses to the connection from upper layers go through the pass-through
mux.
2017-10-31 18:03:23 +01:00
Christopher Faulet
cd7879adc2 BUG/MEDIUM: threads: Run the poll loop on the main thread too
There was a flaw in the way the threads was created. the main one was just used
to create all the others and just wait to exit. Now, it is used to run a poll
loop. So we only create nbthread-1 threads.

This also fixes a bug about the compression filter when there is only 1 thread
(nbthread == 1 or no threads support). The bug was in the way thread-local
resources was initialized. per-thread init/deinit callbacks were never called
for the main process. So, with nthread set to 1, some buffers remained
uninitialized.
2017-10-31 13:58:33 +01:00
Christopher Faulet
ff8abcd31d MEDIUM: threads/proxy: Add a lock per proxy and atomically update proxy vars
Now, each proxy contains a lock that must be used when necessary to protect
it. Moreover, all proxy's counters are now updated using atomic operations.
2017-10-31 13:58:30 +01:00
Christopher Faulet
f8188c69fa MEDIUM: threads/logs: Make logs thread-safe
log buffers and static variables used in log functions are now thread-local. So
there is no need to lock anything to log messages. Moreover, per-thread
init/deinit functions are now used to initialize these buffers.
2017-10-31 13:58:30 +01:00
Christopher Faulet
c1b730a41a MINOR: cli: Add "show startup-logs" command
This command will dump all startup_logs buffer containing all alerts and
warnings emitted during HAProxy startup.
2017-10-31 11:36:13 +01:00
Christopher Faulet
d46963865e MINOR: log: Save alerts and warnings emitted during HAProxy startup
Because we can't always display the standard error messages when HAProxy is
started, all alerts and warnings emitted during the startup will now be saved in
a buffer. It can also be handy to store these messages just in case you
missed something during the startup

To implement this feature, Alert and Warning functions now relies on
display_message. The difference is just on conditions to call this function and
it remains unchanged. In display_message, if MODE_STARTING flag is set, we save
the message.
2017-10-31 11:36:13 +01:00
Emmanuel Hocdet
01da571e21 MINOR: merge ssl_sock_get calls for log and ppv2
Merge ssl_sock_get_version and ssl_sock_get_proto_version.
Change ssl_sock_get_cipher to be used in ppv2.
2017-10-27 19:32:36 +02:00
David Carlier
93e8b88f06 BUG/MINOR: log: fixing small memory leak in error code path.
since we do not log the sample fetch when it is invalid, we can
free the log data.
2017-09-21 17:44:31 +02:00
Christopher Faulet
0132d06f68 MINOR: logs: Use dedicated function to init/deinit log buffers
Now, we use init_log_buffers and deinit_log_buffers to, respectively, initialize
and deinitialize log buffers used for syslog messages.

These functions have been introduced to be used by threads, to deal with
thread-local log buffers.
2017-09-05 10:29:31 +02:00
Willy Tarreau
d02286d6c8 BUG/MINOR: log: pin the front connection when front ip/ports are logged
Mathias Weiersmueller reported an interesting issue with logs which Lukas
diagnosed as dating back from commit 9b061e332 (1.5-dev9). When front
connection information (ip, port) are logged in TCP mode and the log is
emitted at the end of the connection (eg: because %B or any log tag
requiring LW_BYTES is set), the log is emitted after the connection is
closed, so the address and ports cannot be retrieved anymore.

It could be argued that we'd make a special case of these to immediatly
retrieve the source and destination addresses from the connection, but it
seems cleaner to simply pin the front connection, marking it "tracked" by
adding the LW_XPRT flag to mention that we'll need some of these elements
at the last moment. Only LW_FRTIP and LW_CLIP are affected. Note that after
this change, LW_FRTIP could simply be removed as it's not used anywhere.

Note that the problem doesn't happen when using %[src] or %[dst] since
all sample expressions set LW_XPRT.

This must be backported to 1.7, 1.6 and 1.5.
2017-06-23 11:34:57 +02:00
Jim Freeman
a2278c8bbb CLEANUP: logs: typo: simgle => single
Typo in error message. Backport to 1.7.
2017-04-18 14:52:07 +02:00
Willy Tarreau
a261e9b094 CLEANUP: connection: remove all direct references to raw_sock and ssl_sock
Now we exclusively use xprt_get(XPRT_RAW) instead of &raw_sock or
xprt_get(XPRT_SSL) for &ssl_sock. This removes a bunch of #ifdef and
include spread over a number of location including backend, cfgparse,
checks, cli, hlua, log, server and session.
2016-12-22 23:26:38 +01:00
Willy Tarreau
71a8c7c49e MINOR: listener: move the transport layer pointer to the bind_conf
A mistake was made when the socket layer was cut into proto and
transport, the transport was attached to the listener while all
listeners in a single "bind" line always have exactly the same
transport. It doesn't seem obvious but this is the reason why there
are so many #ifdefs USE_OPENSSL in cfgparse : a lot of operations
have to be open-coded because cfgparse only manipulates bind_conf
and we don't have the information of the transport layer here.

Very little code makes use of the transport layer, mainly session
setup and log. These places can afford an extra pointer indirection
(the listener points to the bind_conf). This change is thus very small,
it saves a little bit of memory (8B per listener) and makes the code
more flexible.
2016-12-22 23:26:37 +01:00
Thierry FOURNIER / OZON.IO
8a4e4420fb MEDIUM: log-format: Use standard HAProxy log system to report errors
The function log format emit its own error message using Alert(). This
patch replaces this behavior and uses the standard HAProxy error system
(with memprintf).

The benefits are:
 - cleaning the log system

 - the logformat can ignore the caller (actually the caller must set
   a flag designing the caller function).

 - Make the usage of the logformat function easy for future components.
2016-11-25 07:32:58 +01:00
Thierry FOURNIER / OZON.IO
4ed1c9585d MINOR: http/conf: store the use_backend configuration file and line for logs
The error log of the directive use_backend doesn't provide the
file and line containing the declaration. This patch stores
theses informations.
2016-11-25 07:15:09 +01:00
Thierry FOURNIER / OZON.IO
a2c38d7904 MEDIUM: log-format: strict parsing and enable fail
Until now, the function parse_logformat_string() never fails. It
send warnings when it parses bad format, and returns expression in
best effort.

This patch replaces warnings by alert and returns a fail code.

Maybe the warning mode is designed for a compatibility with old
configuration versions. If it is the case, now this compatibility
is broken.

[wt: no, the reason is that an alert must cause a startup failure,
 but this will be OK with next patch]
2016-11-24 18:54:26 +01:00
Thierry FOURNIER / OZON.IO
6fe0e1b977 CLEANUP: log-format: remove unused arguments
The log-format function parse_logformat_string() takes file and line
for building parsing logs. These two parameters are embedded in the
struct proxy curproxy, which is the current parsing context.

This patch removes these two unused arguments.
2016-11-24 18:54:26 +01:00
Thierry FOURNIER / OZON.IO
bca46f0d9d CLEANUP: log-format: fix return code of function parse_logformat_var_args()
This patch replace the successful return code from 0 to 1. The
error code is replaced from 1 to 0.

The return code of this function is actually unused, so this
patch cannot modify the behaviour.
2016-11-24 18:54:26 +01:00
Thierry FOURNIER / OZON.IO
eca4d95317 CLEANUP: log-format: fix return code of the function parse_logformat_var()
This patch replaces the successful return code from 0 to 1. The
error code is replaced from -1 to 0.

The return code of this function is actually unused, so this
patch cannot modify the behaviour.
2016-11-24 18:54:25 +01:00
Thierry FOURNIER / OZON.IO
9cbfef2455 BUG/MINOR: log-format: uncatched memory allocation functions
Some return code of memory allocation functions are not tested.
This patch fix theses checks.
2016-11-24 18:54:25 +01:00
Christopher Faulet
f7e4e7e096 MAJOR: spoe: Add an experimental Stream Processing Offload Engine
SPOE makes possible the communication with external components to retrieve some
info using an in-house binary protocol, the Stream Processing Offload Protocol
(SPOP). In the long term, its aim is to allow any kind of offloading on the
streams. This first version, besides being experimental, won't do lot of
things. The most important today is to validate the protocol design and lay the
foundations of what will, one day, be a full offload engine for the stream
processing.

So, for now, the SPOE can offload the stream processing before "tcp-request
content", "tcp-response content", "http-request" and "http-response" rules. And
it only supports variables creation/suppression. But, in spite of these limited
features, we can easily imagine to implement a SSO solution, an ip reputation
service or an ip geolocation service.

Internally, the SPOE is implemented as a filter. So, to use it, you must use
following line in a proxy proxy section:

  frontend my-front
      ...
      filter spoe [engine <name>] config <file>
      ...

It uses its own configuration file to keep the HAProxy configuration clean. It
is also a easy way to disable it by commenting out the filter line.

See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-11-09 22:57:01 +01:00
Thierry FOURNIER / OZON.IO
4cac359a39 MEDIUM: log: Decompose %Tq in %Th %Ti %TR
Tq is the time between the instant the connection is accepted and a
complete valid request is received. This time includes the handshake
(SSL / Proxy-Protocol), the idle when the browser does preconnect and
the request reception.

This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR
which returns respectively the handshake time, the idle time and the
duration of valid request reception. It also adds %Ta which reports
the request's active time, which is the total time without %Th nor %Ti.
It replaces %Tt as the total time, reporting accurate measurements for
HTTP persistent connections.

%Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only
avalaible for HTTP connections.

In addition to this, we have new timestamps %tr, %trg and %trl, which
log the date of start of receipt of the request, respectively in the
default format, in GMT time and in local time (by analogy with %t, %T
and %Tl). All of them are obviously only available for HTTP. These values
are more relevant as they more accurately represent the request date
without being skewed by a browser's preconnect nor a keep-alive idle
time.

The HTTP log format and the CLF log format have been modified to
use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This
way the default log formats now produce the expected output for users
who don't want to manually fiddle with the log-format directive.

Example with the following log-format :

   log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"

The request was sent by hand using "openssl s_client -connect" :

   Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1"

=> 6 ms of SSL handshake, 2375 waiting before sending the first char (in
fact the time to type the first line), 261 ms before the end of the request,
no time spent in queue, 1 ms spend connecting to the server, immediate
response, total active time for this request = 262ms. Total time from accept
to close : 2643 ms.

The timing now decomposes like this :

                 first request               2nd request
      |<-------------------------------->|<-------------- ...
      t         tr                       t    tr ...
   ---|----|----|----|----|----|----|----|----|--
      : Th   Ti   TR   Tw   Tc   Tr   Td : Ti   ...
      :<---- Tq ---->:                   :
      :<-------------- Tt -------------->:
                :<--------- Ta --------->:
2016-08-23 15:18:08 +02:00
Willy Tarreau
077edcba2e BUILD: log: iovec requires to include sys/uio.h on OpenBSD
The following commit merged into 1.6-dev6 broke the build on OpenBSD :

  609ac2a ("MEDIUM: log: replace sendto() with sendmsg() in __send_log()")

Including sys/uio.h is enough to fix this. This fix needs to be backported
to 1.6.
2016-08-10 19:32:06 +02:00
Dragan Dosen
db1b6f9ecb BUG/MEDIUM: log: use function "escape_string" instead of "escape_chunk"
In function lf_text_len(), we used escape_chunk() to escape special
characters. There could be a problem if len is greater than the real src
string length (zero-terminated), eg. when calling lf_text_len() from
lf_text().
2016-07-26 15:25:32 +02:00
Willy Tarreau
27b639d37f MINOR: log: add the %Td log-format specifier
As suggested by Pavlos, it's too bad that we didn't have a %Td log
format tag given that there are a few mentions of Td corresponding
to the data transmission time already in the doc, so this is now done.
Just like the other specifiers, we report -1 if the connection failed
before reaching the data transmission state.
2016-05-17 18:04:30 +02:00
Nenad Merdanovic
54e439f0b4 BUG/MINOR: log: fix a typo that would cause %HP to log <BADREQ>
Typo was introduced in 57bc891 ("BUG/MEDIUM: log: fix risk of
segfault when logging HTTP fields in TCP mode") which inverted the
condition in the test and caused <BADREQ> to be logged when using
%HP.

Signed-off-by: Nenad Merdanovic <nmerdan@anine.io>
2016-04-29 07:28:44 +02:00
Willy Tarreau
57bc8917c3 BUG/MEDIUM: log: fix risk of segfault when logging HTTP fields in TCP mode
David Torgerson faced an issue when using HTTP fields in log-format in TCP
sections. The txn is dereferenced while it's null, resulting in a crash of
the process. Such configurations are invalid and a warning is emitted, but
nevertheless the process must not crash. As found by Lukas Tribus, this is
a side effect of the split between the stream and the HTTP transaction that
happened in 1.6, making it possible to have txn==NULL there.

The fix consists in checking that txn is valid before using it. Fortunately
it's easy since almost all places already used to check for the existence
of a field (eg: txn->uri).

This patch should be backported to 1.6.
2016-04-25 17:15:58 +02:00