Commit Graph

1378 Commits

Author SHA1 Message Date
Jackie Tapia
749f74c622 DOC: Use gender neutral language
This patch updates the documentation files and code comments to avoid
the use of gender specific phrasing in favor of "they" or "it".
2020-07-26 22:35:43 +02:00
Lukas Tribus
a267b5df4a DOC: ssl: req_ssl_sni needs implicit TLS
req_ssl_sni is not compatible with protocols negotiating TLS
explicitly, like SMTP on port 25 or 587 and IMAP on port 143.

Fix an example referring to 587 (SMTPS port with implicit TLS
is 465) and amend the req_ssl_sni documentation.

This doc fix should be backported to supported versions.
2020-07-19 06:36:12 +02:00
Willy Tarreau
e732cbd8a9 [RELEASE] Released version 2.3-dev1
Released version 2.3-dev1 with the following main changes :
    - MINOR: config: make strict limits enabled by default
    - BUG/MINOR: acl: Fix freeing of expr->smp in prune_acl_expr
    - BUG/MINOR: sample: Fix freeing of conv_exprs in release_sample_expr
    - BUG/MINOR: haproxy: Free proxy->format_unique_id during deinit
    - BUG/MINOR: haproxy: Add missing free of server->(hostname|resolvers_id)
    - BUG/MINOR: haproxy: Free proxy->unique_id_header during deinit
    - BUG/MINOR: haproxy: Free srule->file during deinit
    - BUG/MINOR: haproxy: Free srule->expr during deinit
    - BUG/MINOR: sample: Free str.area in smp_check_const_bool
    - BUG/MINOR: sample: Free str.area in smp_check_const_meth
    - CLEANUP: haproxy: Free proxy_deinit_list in deinit()
    - CLEANUP: haproxy: Free post_deinit_list in deinit()
    - CLEANUP: haproxy: Free server_deinit_list in deinit()
    - CLEANUP: haproxy: Free post_server_check_list in deinit()
    - CLEANUP: Add static void vars_deinit()
    - CLEANUP: Add static void hlua_deinit()
    - CLEANUP: contrib/prometheus-exporter: typo fixes for ssl reuse metric
    - BUG/MEDIUM: lists: add missing store barrier on MT_LIST_BEHEAD()
    - BUG/MEDIUM: lists: add missing store barrier in MT_LIST_ADD/MT_LIST_ADDQ
    - MINOR: tcp: Support TCP keepalive parameters customization
    - BUILD: tcp: condition TCP keepalive settings to platforms providing them
    - MINOR: lists: rename some MT_LIST operations to clarify them
    - MINOR: buffer: use MT_LIST_ADDQ() for buffer_wait lists additions
    - MINOR: connection: use MT_LIST_ADDQ() to add connections to idle lists
    - MINOR: tasks: use MT_LIST_ADDQ() when killing tasks.
    - CONTRIB: da: fix memory leak in dummy function da_atlas_open()
    - CI: travis-ci: speed up osx build by running brew scripted, switch to latest osx image
    - BUG/MEDIUM: mux-h2: Don't add private connections in available connection list
    - BUG/MEDIUM: mux-fcgi: Don't add private connections in available connection list
    - MINOR: connection: Set the SNI on server connections before installing the mux
    - MINOR: connection: Set new connection as private on reuse never
    - MINOR: connection: Add a wrapper to mark a connection as private
    - MEDIUM: connection: Add private connections synchronously in session server list
    - MINOR: connection: Use a dedicated function to look for a session's connection
    - MINOR: connection: Set the conncetion target during its initialisation
    - MINOR: session: Take care to decrement idle_conns counter in session_unown_conn
    - MINOR: server: Factorize code to deal with reuse of server idle connections
    - MINOR: server: Factorize code to deal with connections removed from an idle list
    - CLEANUP: connection: remove unused field idle_time from the connection struct
    - BUG/MEDIUM: mux-h1: Continue to process request when switching in tunnel mode
    - MINOR: raw_sock: Report the number of bytes emitted using the splicing
    - MINOR: contrib/prometheus-exporter: Add missing global and per-server metrics
    - MINOR: backend: Add sample fetches to get the server's weight
    - BUG/MINOR: mux-fcgi: Handle empty STDERR record
    - BUG/MINOR: mux-fcgi: Set conn state to RECORD_P when skipping the record padding
    - BUG/MINOR: mux-fcgi: Set flags on the right stream field for empty FCGI_STDOUT
    - BUG/MINOR: backend: fix potential null deref on srv_conn
    - BUG/MEDIUM: log: issue mixing sampled to not sampled log servers.
    - MEDIUM: udp: adds minimal proto udp support for message listeners.
    - MEDIUM: log/sink: re-work and merge of build message API.
    - MINOR: log: adds syslog udp message handler and parsing.
    - MEDIUM: log: adds log forwarding section.
    - MINOR: log: adds counters on received syslog messages.
    - BUG/MEDIUM: fcgi-app: fix memory leak in fcgi_flt_http_headers
    - BUG/MEDIUM: server: resolve state file handle leak on reload
    - BUG/MEDIUM: server: fix possibly uninitialized state file on close
    - BUG/MEDIUM: channel: Be aware of SHUTW_NOW flag when output data are peeked
    - BUILD: config: address build warning on raspbian+rpi4
    - BUG/MAJOR: tasks: make sure to always lock the shared wait queue if needed
    - BUILD: config: fix again bugs gcc warnings on calloc
2020-07-17 15:13:19 +02:00
Emeric Brun
12941c82d0 MEDIUM: log: adds log forwarding section.
Log forwarding:

It is possible to declare one or multiple log forwarding section,
haproxy will forward all received log messages to a log servers list.

log-forward <name>
  Creates a new log forwarder proxy identified as <name>.

bind <addr> [param*]
  Used to configure a log udp listener to receive messages to forward.
  Only udp listeners are allowed, address must be prefixed using
  'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters
  found in 5.1 paragraph but most of them are irrelevant for udp/syslog case.

log global
log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>]
    <facility> [<level> [<minlevel>]]
  Used to configure target log servers. See more details on proxies
  documentation.
  If no format specified, haproxy tries to keep the incoming log format.
  Configured facility is ignored, except if incoming message does not
  present a facility but one is mandatory on the outgoing format.
  If there is no timestamp available in the input format, but the field
  exists in output format, haproxy will use the local date.

  Example:
    global
       log stderr format iso local7

    ring myring
        description "My local buffer"
        format rfc5424
        maxlen 1200
        size 32764
        timeout connect 5s
        timeout server 10s
        # syslog tcp server
        server mysyslogsrv 127.0.0.1:514 log-proto octet-count

    log-forward sylog-loadb
        bind udp4@127.0.0.1:1514
        # all messages on stderr
        log global
        # all messages on local tcp syslog server
        log ring@myring local0
        # load balance messages on 4 udp syslog servers
        log 127.0.0.1:10001 sample 1:4 local0
        log 127.0.0.1:10002 sample 2:4 local0
        log 127.0.0.1:10003 sample 3:4 local0
        log 127.0.0.1:10004 sample 4:4 local0
2020-07-15 17:50:12 +02:00
Emeric Brun
546488559a MEDIUM: log/sink: re-work and merge of build message API.
This patch merges build message code between sink and log
and introduce a new API based on struct ist array to
prepare message header with zero copy, targeting the
log forwarding feature.

Log format 'iso' and 'timed' are now avalaible on logs line.
A new log format 'priority' is also added.
2020-07-15 17:50:12 +02:00
Emeric Brun
3835c0dcb5 MEDIUM: udp: adds minimal proto udp support for message listeners.
This patch introduce proto_udp.c targeting a further support of
log forwarding feature.

This code was originally produced by Frederic Lecaille working on
QUIC support and only minimal requirements for syslog support
have been merged.
2020-07-15 17:50:12 +02:00
Christopher Faulet
1bea865811 MINOR: backend: Add sample fetches to get the server's weight
The following sample fetches have been added :

 * srv_iweight : returns the initial server's weight
 * srv_uweight : returns the user-visible server's weight
 * srv_weight  : returns the current (or effetctive) server's weight

The requested server must be passed as argument, evnetually preceded by the
backend name. For instance :

  srv_weight(back-http/www1)
2020-07-15 14:08:14 +02:00
Willy Tarreau
5254321d14 BUILD: tcp: condition TCP keepalive settings to platforms providing them
Previous commit b24bc0d ("MINOR: tcp: Support TCP keepalive parameters
customization") broke non-Linux builds as TCP_KEEP{CNT,IDLE,INTVL} are
not necessarily defined elsewhere.

This patch adds the required #ifdefs to condition the visibility of the
keywords, and adds a mention in the doc about their dependency on Linux.
2020-07-09 05:58:51 +02:00
MIZUTA Takeshi
b24bc0dfb6 MINOR: tcp: Support TCP keepalive parameters customization
It is now possible to customize TCP keepalive parameters.
These correspond to the socket options TCP_KEEPCNT, TCP_KEEPIDLE, TCP_KEEPINTVL
and are valid for the defaults, listen, frontend and backend sections.

This patch fixes GitHub issue #670.
2020-07-09 05:22:16 +02:00
William Dauchy
a5194607ab MINOR: config: make strict limits enabled by default
as agreed a few months ago, enable strict-limits for v2.3
update configuration manual accordingly

Signed-off-by: William Dauchy <w.dauchy@criteo.com>
2020-07-07 16:52:35 +02:00
Willy Tarreau
33205c23a7 [RELEASE] Released version 2.3-dev0
Released version 2.3-dev0 with the following main changes :
    - exact copy of 2.2.0
2020-07-07 16:35:28 +02:00
Willy Tarreau
3a00c915fd [RELEASE] Released version 2.2.0
Released version 2.2.0 with the following main changes :
    - BUILD: mux-h2: fix typo breaking build when using DEBUG_LOCK
    - CLEANUP: makefile: update the outdated list of DEBUG_xxx options
    - BUILD: tools: make resolve_sym_name() return a const
    - CLEANUP: auth: fix useless self-include of auth-t.h
    - BUILD: tree-wide: cast arguments to tolower/toupper to unsigned char
    - CLEANUP: assorted typo fixes in the code and comments
    - WIP/MINOR: ssl: add sample fetches for keylog in frontend
    - DOC: fix tune.ssl.keylog sample fetches array
    - BUG/MINOR: ssl: check conn in keylog sample fetch
    - DOC: configuration: various typo fixes
    - MINOR: log: Remove unused case statement during the log-format string parsing
    - BUG/MINOR: mux-h1: Fix the splicing in TUNNEL mode
    - BUG/MINOR: mux-h1: Don't read data from a pipe if the mux is unable to receive
    - BUG/MINOR: mux-h1: Disable splicing only if input data was processed
    - BUG/MEDIUM: mux-h1: Disable splicing for the conn-stream if read0 is received
    - MINOR: mux-h1: Improve traces about the splicing
    - BUG/MINOR: backend: Remove CO_FL_SESS_IDLE if a client remains on the last server
    - BUG/MEDIUM: connection: Don't consider new private connections as available
    - BUG/MINOR: connection: See new connection as available only on reuse always
    - DOC: configuration: remove obsolete mentions of H2 being converted to HTTP/1.x
    - CLEANUP: ssl: remove unrelevant comment in smp_fetch_ssl_x_keylog()
    - DOC: update INSTALL with new compiler versions
    - DOC: minor update to coding style file
    - MINOR: version: mention that it's an LTS release now
2020-07-07 16:33:14 +02:00
Willy Tarreau
253c2519c2 DOC: configuration: remove obsolete mentions of H2 being converted to HTTP/1.x
The first H2 implementation in version 1.8 used to turn HTTP/2 requests
to HTTP/1.1, causing many limitations. This is not true anymore and we
don't suffer from the lack of server-side H2 nor are we forced to close
mode anymore, so let's remove such obsolete mentions.

This could be backported to 2.0.
2020-07-07 15:55:23 +02:00
Daniel Corbett
67a8271cc7 DOC: configuration: various typo fixes
Quick round of typo corrections for configuration.txt
2020-07-07 11:55:16 +02:00
William Lallemand
d742b6caf7 DOC: fix tune.ssl.keylog sample fetches array
The labels EXPORTER_SECRET and EARLY_EXPORTER_SECRET were swapped in the
array.
2020-07-07 10:25:15 +02:00
William Lallemand
7d42ef5b22 WIP/MINOR: ssl: add sample fetches for keylog in frontend
OpenSSL 1.1.1 provides a callback registering function
SSL_CTX_set_keylog_callback, which allows one to receive a string
containing the keys to deciphers TLSv1.3.

Unfortunately it is not possible to store this data in binary form and
we can only get this information using the callback. Which means that we
need to store it until the connection is closed.

This patches add 2 pools, the first one, pool_head_ssl_keylog is used to
store a struct ssl_keylog which will be inserted as a ex_data in a SSL *.
The second one is pool_head_ssl_keylog_str which will be used to store
the hexadecimal strings.

To enable the capture of the keys, you need to set "tune.ssl.keylog on"
in your configuration.

The following fetches were implemented:

ssl_fc_client_early_traffic_secret,
ssl_fc_client_handshake_traffic_secret,
ssl_fc_server_handshake_traffic_secret,
ssl_fc_client_traffic_secret_0,
ssl_fc_server_traffic_secret_0,
ssl_fc_exporter_secret,
ssl_fc_early_exporter_secret
2020-07-06 19:08:03 +02:00
Willy Tarreau
62f11a589f [RELEASE] Released version 2.2-dev12
Released version 2.2-dev12 with the following main changes :
    - BUG/MINOR: mux_h2: don't lose the leaving trace in h2_io_cb()
    - MINOR: cli: make "show sess" stop at the last known session
    - CLEANUP: buffers: remove unused buffer_wq_lock lock
    - BUG/MEDIUM: buffers: always allocate from the local cache first
    - MINOR: connection: align toremove_{lock,connections} and cleanup into idle_conns
    - CONTRIB: debug: add missing flags SI_FL_L7_RETRY & SI_FL_D_L7_RETRY
    - BUG/MEDIUM: connections: Don't increase curr_used_conns for shared connections.
    - BUG/MEDIUM: checks: Increment the server's curr_used_conns
    - REORG: buffer: rename buffer.c to dynbuf.c
    - REORG: includes: create tinfo.h for the thread_info struct
    - CLEANUP: pool: only include the type files from types
    - MINOR: pools: move the LRU cache heads to thread_info
    - BUG/MINOR: debug: fix "show fd" null-deref when built with DEBUG_FD
    - MINOR: stats: add 3 new output values for the per-server idle conn state
    - MINOR: activity: add per-thread statistics on FD takeover
    - BUG/MINOR: server: start cleaning idle connections from various points
    - MEDIUM: server: improve estimate of the need for idle connections
    - MINOR: stats: add the estimated need of concurrent connections per server
    - BUG/MINOR: threads: Don't forget to init each thread toremove_lock.
    - BUG/MEDIUM: lists: Lock the element while we check if it is in a list.
    - Revert "BUG/MEDIUM: lists: Lock the element while we check if it is in a list."
    - BUG/MINOR: haproxy: don't wake already stopping threads on exit
    - BUG/MINOR: server: always count one idle slot for current thread
    - MEDIUM: server: use the two thresholds for the connection release algorithm
    - BUG/MINOR: http-rules: Fix ACLs parsing for http deny rules
    - BUG/MINOR: sched: properly cover for a rare MT_LIST_ADDQ() race
    - MINOR: mux-h1: avoid taking the toremove_lock in on dying tasks
    - MINOR: mux-h2: avoid taking the toremove_lock in on dying tasks
    - MINOR: mux-fcgi: avoid taking the toremove_lock in on dying tasks
    - MINOR: pools: increase MAX_BASE_POOLS to 64
    - DOC: ssl: add "allow-0rtt" and "ciphersuites" in crt-list
    - BUG/MEDIUM: pattern: Add a trailing \0 to match strings only if possible
    - BUG/MEDIUM: log-format: fix possible endless loop in parse_logformat_string()
    - BUG/MINOR: proxy: fix dump_server_state()'s misuse of the trash
    - BUG/MINOR: proxy: always initialize the trash in show servers state
    - MINOR: cli/proxy: add a new "show servers conn" command
    - MINOR: server: skip servers with no idle conns earlier
    - BUG/MINOR: server: fix the connection release logic regarding nearly full conditions
    - MEDIUM: server: add a new pool-low-conn server setting
    - BUG/MEDIUM: backend: always search in the safe list after failing on the idle one
    - MINOR: backend: don't always takeover from the same threads
    - MINOR: sched: make sched->task_list_size atomic
    - MEDIUM: sched: create a new TASK_KILLED task flag
    - MEDIUM: sched: implement task_kill() to kill a task
    - MEDIUM: mux-h1: use task_kill() during h1_takeover() instead of task_wakeup()
    - MEDIUM: mux-h2: use task_kill() during h2_takeover() instead of task_wakeup()
    - MEDIUM: mux-fcgi: use task_kill() during fcgi_takeover() instead of task_wakeup()
    - MINOR: list: Add MT_LIST_DEL_SAFE_NOINIT() and MT_LIST_ADDQ_NOCHECK()
    - CLEANUP: connections: rename the toremove_lock to takeover_lock
    - MEDIUM: connections: Don't use a lock when moving connections to remove.
    - DOC: configuration: add missing index entries for tune.pool-{low,high}-fd-ratio
    - DOC: configuration: fix alphabetical ordering for tune.pool-{high,low}-fd-ratio
    - MINOR: config: add a new tune.idle-pool.shared global setting.
    - MINOR: 51d: silence a warning about null pointer dereference
    - MINOR: debug: add a new "debug dev memstats" command
    - MINOR: log-format: allow to preserve spacing in log format strings
    - BUILD: debug: avoid build warnings with DEBUG_MEM_STATS
    - BUG/MAJOR: sched: make sure task_kill() always queues the task
    - BUG/MEDIUM: muxes: Make sure nobody stole the connection before using it.
    - BUG/MEDIUM: cli/proxy: don't try to dump idle connection state if there's none
    - BUILD: haproxy: fix build error when RLIMIT_AS is not set
    - BUG/MAJOR: sched: make it work also when not building with DEBUG_STRICT
    - MINOR: log: add time second fraction field to rfc5424 log timestamp.
    - BUG/MINOR: log: missing timezone on iso dates.
    - BUG/MEDIUM: server: don't kill all idle conns when there are not enough
    - MINOR: sched: split tasklet_wakeup() into tasklet_wakeup_on()
    - BUG/MEDIUM: connections: Set the tid for the old tasklet on takeover.
    - BUG/MEDIUM: connections: Let the xprt layer know a takeover happened.
    - BUG/MINOR: http_act: don't check capture id in backend (2)
    - BUILD: makefile: disable threads by default on OpenBSD
    - BUILD: peers: fix build warning with gcc 4.2.1
    - CI: cirrus-ci: exclude slow reg-tests
2020-07-04 07:10:24 +02:00
Dragan Dosen
1e3b16f74f MINOR: log-format: allow to preserve spacing in log format strings
Now it's possible to preserve spacing everywhere except in "log-format",
"log-format-sd" and "unique-id-format" directives, where spaces are
delimiters and are merged. That may be useful when the response payload
is specified as a log format string by "lf-file" or "lf-string", or even
for headers or anything else.

In order to merge spaces, a new option LOG_OPT_MERGE_SPACES is applied
exclusively on options passed to function parse_logformat_string().

This patch fixes an issue #701 ("http-request return log-format file
evaluation altering spacing of ASCII output/art").
2020-07-02 10:11:44 +02:00
Willy Tarreau
76cc699017 MINOR: config: add a new tune.idle-pool.shared global setting.
Enables ('on') or disables ('off') sharing of idle connection pools between
threads for a same server. The default is to share them between threads in
order to minimize the number of persistent connections to a server, and to
optimize the connection reuse rate. But to help with debugging or when
suspecting a bug in HAProxy around connection reuse, it can be convenient to
forcefully disable this idle pool sharing between multiple threads, and force
this option to "off". The default is on.

This could have been nice to have during the idle connections debugging,
but it's not too late to add it!
2020-07-01 19:07:37 +02:00
Willy Tarreau
83ca305ddc DOC: configuration: fix alphabetical ordering for tune.pool-{high,low}-fd-ratio
In addition they were in the wrong alphabetical order in the doc. They
were added in 2.0 by commit 88698d966 ("MEDIUM: connections: Add a way
to control the number of idling connections.") so this must be backported
to 2.0.
2020-07-01 18:30:16 +02:00
Willy Tarreau
a8e2d97905 DOC: configuration: add missing index entries for tune.pool-{low,high}-fd-ratio
These two keywords didn't have an entry in the index. They were added in
2.0 by commit 88698d966 ("MEDIUM: connections: Add a way to control the
number of idling connections.") so this must be backported to 2.0.
2020-07-01 18:29:44 +02:00
Willy Tarreau
2f3f4d3441 MEDIUM: server: add a new pool-low-conn server setting
The problem with the way idle connections currently work is that it's
easy for a thread to steal all of its siblings' connections, then release
them, then it's done by another one, etc. This happens even more easily
due to scheduling latencies, or merged events inside the same pool loop,
which, when dealing with a fast server responding in sub-millisecond
delays, can really result in one thread being fully at work at a time.

In such a case, we perform a huge amount of takeover() which consumes
CPU and requires quite some locking, sometimes resulting in lower
performance than expected.

In order to fight against this problem, this patch introduces a new server
setting "pool-low-conn", whose purpose is to dictate when it is allowed to
steal connections from a sibling. As long as the number of idle connections
remains at least as high as this value, it is permitted to take over another
connection. When the idle connection count becomes lower, a thread may only
use its own connections or create a new one. By proceeding like this even
with a low number (typically 2*nbthreads), we quickly end up in a situation
where all active threads have a few connections. It then becomes possible
to connect to a server without bothering other threads the vast majority
of the time, while still being able to use these connections when the
number of available FDs becomes low.

We also use this threshold instead of global.nbthread in the connection
release logic, allowing to keep more extra connections if needed.

A test performed with 10000 concurrent HTTP/1 connections, 16 threads
and 210 servers with 1 millisecond of server response time showed the
following numbers:

   haproxy 2.1.7:           185000 requests per second
   haproxy 2.2:             314000 requests per second
   haproxy 2.2 lowconn 32:  352000 requests per second

The takeover rate goes down from 300k/s to 13k/s. The difference is
further amplified as the response time shrinks.
2020-07-01 15:23:15 +02:00
William Lallemand
5d03639ba6 DOC: ssl: add "allow-0rtt" and "ciphersuites" in crt-list
Support for "allow-0rtt" and "ciphersuites" exists for crt-list.

Fix issue #721.

Should be backported as far as 1.8.
2020-06-30 16:15:44 +02:00
Willy Tarreau
4462af8da2 [RELEASE] Released version 2.2-dev11
Released version 2.2-dev11 with the following main changes :
    - REGTEST: Add a simple script to tests errorfile directives in proxy sections
    - BUG/MEDIUM: fcgi-app: Resolve the sink if a fcgi-app logs in a ring buffer
    - BUG/MINOR: spoe: correction of setting bits for analyzer
    - BUG/MINOR: cfgparse: Support configurations without newline at EOF
    - MINOR: cfgparse: Warn on truncated lines / files
    - BUG/MINOR: http_ana: clarify connection pointer check on L7 retry
    - MINOR: debug: add a new DEBUG_FD build option
    - BUG/MINOR: tasks: make sure never to exceed max_processed
    - MINOR: task: add a new pointer to current tasklet queue
    - BUG/MEDIUM: task: be careful not to run too many tasks at TL_URGENT
    - BUG/MINOR: cfgparse: Fix argument reference in PARSE_ERR_TOOMANY message
    - BUG/MINOR: cfgparse: Fix calculation of position for PARSE_ERR_TOOMANY message
    - BUG/MEDIUM: ssl: fix ssl_bind_conf double free
    - MINOR: ssl: free bind_conf_node in crtlist_free()
    - MINOR: ssl: free the crtlist and the ckch during the deinit()
    - BUG/MINOR: ssl: fix build with ckch_deinit() and crtlist_deinit()
    - BUG/MINOR: ssl/cli: certs added from the CLI can't be deleted
    - MINOR: ssl: move the ckch/crtlist deinit to ssl_sock.c
    - MEDIUM: tasks: apply a fair CPU distribution between tasklet classes
    - MINOR: tasks: make current_queue an index instead of a pointer
    - MINOR: tasks: add a mask of the queues with active tasklets
    - MINOR: tasks: pass the queue index to run_task_from_list()
    - MINOR: tasks: make run_tasks_from_lists() scan the queues itself
    - MEDIUM: tasks: add a tune.sched.low-latency option
    - BUG/MEDIUM: ssl/cli: 'commit ssl cert' crashes when no private key
    - BUG/MINOR: cfgparse: don't increment linenum on incomplete lines
    - MINOR: tools: make parse_line() always terminate the args list
    - BUG/MINOR: cfgparse: report extraneous args *after* the string is allocated
    - MINOR: cfgparse: sanitize the output a little bit
    - MINOR: cli/ssl: handle trailing slashes in crt-list commands
    - MINOR: ssl: add the ssl_s_* sample fetches for server side certificate
    - BUG/MEDIUM: http-ana: Don't loop trying to generate a malformed 500 response
    - BUG/MINOR: stream-int: Don't wait to send truncated HTTP messages
    - BUG/MINOR: http-ana: Set CF_EOI on response channel for generated responses
    - BUG/MINOR: http-ana: Don't wait to send 1xx responses generated by HAProxy
    - MINOR: spoe: Don't systematically create new applets if processing rate is low
    - DOC: fix some typos in the ssl_s_{s|i}_dn documentation
    - BUILD: fix ssl_sample.c when building against BoringSSL
    - CI: travis-ci: switch BoringSSL builds to ninja
    - CI: extend spellchecker whitelist
    - DOC: assorted typo fixes in the documentation
    - CLEANUP: assorted typo fixes in the code and comments
    - MINOR: http: Add support for http 413 status
    - REGTEST: ssl: tests the ssl_f_* sample fetches
    - REGTEST: ssl: add some ssl_c_* sample fetches test
    - DOC: ssl: update the documentation of "commit ssl cert"
    - BUG/MINOR: cfgparse: correctly deal with empty lines
    - BUG/MEDIUM: fetch: Fix hdr_ip misparsing IPv4 addresses due to missing NUL
2020-06-26 22:01:04 +02:00
Anthonin Bonnefoy
85048f80c9 MINOR: http: Add support for http 413 status
Add 413 http "payload too large" status code. This will allow 413 to be
used in deny_status and errorfile.
2020-06-26 11:30:02 +02:00
Ilya Shipitsin
11057a3590 DOC: assorted typo fixes in the documentation
this is 10th iteration of typo fixes
2020-06-26 11:27:10 +02:00
William Lallemand
8f600c833a DOC: fix some typos in the ssl_s_{s|i}_dn documentation
Fix some typos in the ssl_s_{s|i}_dn documentation.
2020-06-26 09:55:06 +02:00
William Lallemand
bfa3e81a7b MINOR: ssl: add the ssl_s_* sample fetches for server side certificate
This commit adds some sample fetches that were lacking on the server
side:

ssl_s_key_alg, ssl_s_notafter, ssl_s_notbefore, ssl_s_sig_alg,
ssl_s_i_dn, ssl_s_s_dn, ssl_s_serial, ssl_s_sha1, ssl_s_der,
ssl_s_version
2020-06-25 21:22:24 +02:00
Willy Tarreau
e7723bddd7 MEDIUM: tasks: add a tune.sched.low-latency option
Now that all tasklet queues are scanned at once by run_tasks_from_lists(),
it becomes possible to always check for lower priority classes and jump
back to them when they exist.

This patch adds tune.sched.low-latency global setting to enable this
behavior. What it does is stick to the lowest ranked priority list in
which tasks are still present with an available budget, and leave the
loop to refill the tasklet lists if the trees got new tasks or if new
work arrived into the shared urgent queue.

Doing so allows to cut the latency in half when running with extremely
deep run queues (10k-100k), thus allowing forwarding of small and large
objects to coexist better. It remains off by default since it does have
a small impact on large traffic by default (shorter batches).
2020-06-24 12:21:26 +02:00
Willy Tarreau
dc0936c255 [RELEASE] Released version 2.2-dev10
Released version 2.2-dev10 with the following main changes :
    - BUILD: include: add sys/types before netinet/tcp.h
    - BUG/MEDIUM: log: don't hold the log lock during writev() on a file descriptor
    - BUILD: Remove nowarn for warnings that do not trigger
    - BUG/MEDIUM: pattern: fix thread safety of pattern matching
    - BUILD: Re-enable -Wimplicit-fallthrough
    - BUG/MINOR: ssl: fix ssl-{min,max}-ver with openssl < 1.1.0
    - BUILD: thread: add parenthesis around values of locking macros
    - BUILD: proto_uxst: shut up yet another gcc's absurd warning
    - BUG/MEDIUM: checks: Fix off-by-one in allocation of SMTP greeting cmd
    - CI: travis-ci: use "-O1" for clang builds
    - MINOR: haproxy: Add void deinit_and_exit(int)
    - MINOR: haproxy: Make use of deinit_and_exit() for clean exits
    - BUG/MINOR: haproxy: Free rule->arg.vars.expr during deinit_act_rules
    - BUILD: compression: make gcc 10 happy with free_zlib()
    - BUILD: atomic: add string.h for memcpy() on ARM64
    - BUG/MINOR: http: make smp_fetch_body() report that the contents may change
    - BUG/MINOR: tcp-rules: tcp-response must check the buffer's fullness
    - BUILD: haproxy: mark deinit_and_exit() as noreturn
    - BUG/MAJOR: vars: Fix bogus free() during deinit() for http-request rules
    - BUG/MEDIUM: ebtree: use a byte-per-byte memcmp() to compare memory blocks
    - MINOR: tools: add a new configurable line parse, parse_line()
    - BUG/MEDIUM: cfgparse: use parse_line() to expand/unquote/unescape config lines
    - BUG/MEDIUM: cfgparse: stop after a reasonable amount of fatal error
    - MINOR: http: do not close connections anymore after internal responses
    - BUG/MINOR: cfgparse: Add missing fatal++ in PARSE_ERR_HEX case
    - BUG/MINOR: spoe: add missing key length check before checking key names
    - MINOR: version: put the compiler version output into version.c not haproxy.c
    - MINOR: compiler: always define __has_feature()
    - MINOR: version: report the presence of the compiler's address sanitizer
    - BUILD: Fix build by including haproxy/global.h
    - BUG/MAJOR: connection: always disable ready events once reported
    - CLEANUP: activity: remove unused counter fd_lock
    - DOC: fd: make it clear that some fields ordering must absolutely be respected
    - MINOR: activity: report the number of times poll() reports I/O
    - MINOR: activity: rename confusing poll_* fields in the output
    - MINOR: fd: Fix a typo in a coment.
    - BUG/MEDIUM: fd: Don't fd_stop_recv() a fd we don't own.
    - BUG/MEDIUM: fd: Call fd_stop_recv() when we just got a fd.
    - MINOR: activity: group the per-loop counters at the top
    - MINOR: activity: rename the "stream" field to "stream_calls"
    - MEDIUM: fd: refine the fd_takeover() migration lock
    - MINOR: fd: slightly optimize the fd_takeover double-CAS loop
    - MINOR: fd: factorize the fd_takeover() exit path to make it safer
    - MINOR: peers: do not use localpeer as an array anymore
    - MEDIUM: peers: add the "localpeer" global option
    - MEDIUM: fd: add experimental support for edge-triggered polling
    - CONTRIB: debug: add the missing flags CO_FL_SAFE_LIST and CO_FL_IDLE_LIST
    - MINOR: haproxy: process signals before runnable tasks
    - MEDIUM: tasks: clean up the front side of the wait queue in wake_expired_tasks()
    - MEDIUM: tasks: also process late wakeups in process_runnable_tasks()
    - BUG/MINOR: cli: allow space escaping on the CLI
    - BUG/MINOR: mworker/cli: fix the escaping in the master CLI
    - BUG/MINOR: mworker/cli: fix semicolon escaping in master CLI
    - REGTEST: http-rules: test spaces in ACLs
    - REGTEST: http-rules: test spaces in ACLs with master CLI
    - BUG/MAJOR: init: properly compute the default global.maxpipes value
    - MEDIUM: map: make the "clear map" operation yield
    - BUG/MEDIUM: stream-int: fix loss of CO_SFL_MSG_MORE flag in forwarding
    - MINOR: mux_h1: Set H1_F_CO_MSG_MORE if we know we have more to send.
    - BUG/MINOR: systemd: Wait for network to be online
    - DOC: configuration: Unindent non-code sentences in the protobuf example
    - DOC: configuration: http-check send was missing from matrix
2020-06-19 21:43:26 +02:00
Peter Gervai
8912ae6987 DOC: configuration: http-check send was missing from matrix
The new directive and its doc were added by commit 8acb1284b ("MINOR:
checks: Add a way to send custom headers and payload during http chekcs")
but the index was not updated.
2020-06-19 21:38:05 +02:00
Peter Gervai
df4c9d2a28 DOC: configuration: Unindent non-code sentences in the protobuf example
Unindent to make the explanation go back to text from code formatted
example in tyhe HTMLized version. Still it's not perfect since these
are not haproxy examples but protobuf config, but... way better.
2020-06-19 21:33:37 +02:00
Willy Tarreau
bc52bec163 MEDIUM: fd: add experimental support for edge-triggered polling
Some of the recent optimizations around the polling to save a few
epoll_ctl() calls have shown that they could also cause some trouble.
However, over time our code base has become totally asynchronous with
I/Os always attempted from the upper layers and only retried at the
bottom, making it look like we're getting closer to EPOLLET support.

There are showstoppers there such as the listeners which cannot support
this. But given that most of the epoll_ctl() dance comes from the
connections, we can try to enable edge-triggered polling on connections.

What this patch does is to add a new global tunable "tune.fd.edge-triggered",
that makes fd_insert() automatically set an et_possible bit on the fd if
the I/O callback is conn_fd_handler. When the epoll code sees an update
for such an FD, it immediately registers it in both directions the first
time and doesn't update it anymore.

On a few tests it proved quite useful with a 14% request rate increase in
a H2->H1 scenario, reducing the epoll_ctl() calls from 2 per request to
2 per connection.

The option is obviously disabled by default as bugs are still expected,
particularly around the subscribe() code where it is possible that some
layers do not always re-attempt reading data after being woken up.
2020-06-19 14:21:46 +02:00
Dragan Dosen
13cd54c08b MEDIUM: peers: add the "localpeer" global option
localpeer <name>
  Sets the local instance's peer name. It will be ignored if the "-L"
  command line argument is specified or if used after "peers" section
  definitions. In such cases, a warning message will be emitted during
  the configuration parsing.

  This option will also set the HAPROXY_LOCALPEER environment variable.
  See also "-L" in the management guide and "peers" section in the
  configuration manual.
2020-06-19 11:37:30 +02:00
Willy Tarreau
1385c88560 [RELEASE] Released version 2.2-dev9
Released version 2.2-dev9 with the following main changes :
    - BUG/MINOR: http-htx: Don't forget to release the http reply in release function
    - BUG/MINOR: http-htx: Fix a leak on error path during http reply parsing
    - MINOR: checks: Remove dead code from process_chk_conn()
    - REGTESTS: checks: Fix tls_health_checks when IPv6 addresses are used
    - REGTESTS: Add missing OPENSSL to REQUIRE_OPTIONS for lua/txn_get_priv
    - MINOR: lua: Use vars_unset_by_name_ifexist()
    - CLEANUP: vars: Remove void vars_unset_by_name(const char*, size_t, struct sample*)
    - MINOR: vars: Make vars_(un|)set_by_name(_ifexist|) return a success value
    - MINOR: lua: Make `set_var()` and `unset_var()` return success
    - MEDIUM: lua: Add `ifexist` parameter to `set_var`
    - MEDIUM: ring: new section ring to declare custom ring buffers.
    - REGTESTS: Add missing OPENSSL to REQUIRE_OPTIONS for compression/lua_validation
    - REGTESTS: Require the version 2.2 to execute lua/set_var
    - BUG/MEDIUM: checks: Refresh the conn-stream and the connection after a connect
    - MINOR: checks: Remove useless tests on the connection and conn-stream
    - BUG/MEDIUM: contrib/spoa: do not register python3.8 if --embed fail
    - BUG/MEDIUM: connection: Ignore PP2 unique ID for stream-less connections
    - BUG/MINOR: connection: Always get the stream when available to send PP2 line
    - BUG/MEDIUM: backend: set the connection owner to the session when using alpn.
    - MINOR: pools: compute an estimate of each pool's average needed objects
    - MEDIUM: pools: directly free objects when pools are too much crowded
    - REGTEST: Add connection/proxy_protocol_send_unique_id_alpn
    - MINOR: http-ana: Make the function http_reply_to_htx() public
    - MINOR: http-ana: Use proxy's error replies to emit 401/407 responses
    - MINOR: http-rules: Use an action function to eval http-request auth rules
    - CLEANUP: http: Remove unused HTTP message templates
    - BUG/MEDIUM: checks: Don't blindly subscribe for receive if waiting for connect
    - MINOR: checks: I/O callback function only rely on the data layer wake callback
    - BUG/MINOR: lua: Add missing string length for lua sticktable lookup
    - BUG/MEDIUM: logs: fix trailing zeros on log message.
    - CI: cirrus-ci: skip reg-tests/connection/proxy_protocol_send_unique_id_alpn.vtc on CentOS 6
    - BUG/MINOR: nameservers: fix error handling in parsing of resolv.conf
    - BUG/MEDIUM: checks: Don't add a tcpcheck ruleset twice in the shared tree
    - MEDIUM: ssl: use TLSv1.2 as the minimum default on bind lines
    - CLEANUP: pools: use the regular lock for the flush operation on lockless pools
    - SCRIPTS: publish-release: pass -n to gzip to remove timestamp
    - MINOR: ring: re-work ring attach generic API.
    - BUG/MINOR: error on unknown statement in ring section.
    - MEDIUM: ring: add server statement to forward messages from a ring
    - MEDIUM: ring: add new srv statement to support octet counting forward
    - MINOR: ssl: set ssl-min-ver in ambiguous configurations
    - CLEANUP: ssl: remove comment from dump_crtlist_sslconf()
    - BUILD: sink: address build warning on 32-bit architectures
    - BUG/MINOR: peers: fix internal/network key type mapping.
    - CLEANUP: regex: remove outdated support for regex actions
    - Revert "MINOR: ssl: rework add cert chain to CTX to be libssl independent"
    - MINOR: mux-h1/proxy: Add a proxy option to disable clear h2 upgrade
    - BUG/MEDIUM: lua: Reset analyse expiration timeout before executing a lua action
    - DOC: add a line about comments in crt-list
    - BUG/MEDIUM: hlua: Lock pattern references to perform set/add/del operations
    - BUG/MINOR: checks: Fix test on http-check rulesets during config validity check
    - BUG/MEDIUM: contrib/prometheus-exporter: Properly set flags to dump metrics
    - BUG/MEDIUM: mworker: fix the copy of options in copy_argv()
    - BUG/MINOR: init: -x can have a parameter starting with a dash
    - BUG/MINOR: init: -S can have a parameter starting with a dash
    - BUG/MEDIUM: mworker: fix the reload with an -- option
    - BUG/MINOR: ssl: fix a trash buffer leak in some error cases
    - BUG/MINOR: mworker: fix a memleak when execvp() failed
    - MINOR: sample: Add secure_memcmp converter
    - REORG: ebtree: move the C files from ebtree/ to src/
    - REORG: ebtree: move the include files from ebtree to include/import/
    - REORG: ebtree: clean up remains of the ebtree/ directory
    - REORG: include: create new file haproxy/api-t.h
    - REORG: include: create new file haproxy/api.h
    - REORG: include: update all files to use haproxy/api.h or api-t.h if needed
    - CLEANUP: include: remove common/config.h
    - CLEANUP: include: remove unused template.h
    - REORG: include: move MIN/MAX from tools.h to compat.h
    - REORG: include: move SWAP/MID_RANGE/MAX_RANGE from tools.h to standard.h
    - CLEANUP: include: remove unused common/tools.h
    - REORG: include: move the base files from common/ to haproxy/
    - REORG: include: move version.h to haproxy/
    - REORG: include: move base64.h, errors.h and hash.h from common to to haproxy/
    - REORG: include: move openssl-compat.h from common/ to haproxy/
    - REORG: include: move ist.h from common/ to import/
    - REORG: include: move the BUG_ON() code to haproxy/bug.h
    - REORG: include: move debug.h from common/ to haproxy/
    - CLEANUP: debug: drop unused function p_malloc()
    - REORG: include: split buf.h into haproxy/buf-t.h and haproxy/buf.h
    - REORG: include: move istbuf.h to haproxy/
    - REORG: include: split mini-clist into haproxy/list and list-t.h
    - REORG: threads: extract atomic ops from hathreads.h
    - CLEANUP: threads: remove a few needless includes of hathreads.h
    - REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h
    - CLEANUP: thread: rename __decl_hathreads() to __decl_thread()
    - REORG: include: move time.h from common/ to haproxy/
    - REORG: include: move integer manipulation functions from standard.h to intops.h
    - CLEANUP: include: remove excessive includes of common/standard.h
    - REORG: include: move freq_ctr to haproxy/
    - CLEANUP: pool: include freq_ctr.h and remove locally duplicated functions
    - REORG: memory: move the pool type definitions to haproxy/pool-t.h
    - REORG: memory: move the OS-level allocator to haproxy/pool-os.h
    - MINOR: memory: don't let __pool_get_first() pick from the cache
    - MEDIUM: memory: don't let pool_put_to_cache() free the objects itself
    - MINOR: memory: move pool-specific path of the locked pool_free() to __pool_free()
    - MEDIUM: memory: make local pools independent on lockless pools
    - REORG: include: move common/memory.h to haproxy/pool.h
    - REORG: include: move common/chunk.h to haproxy/chunk.h
    - REORG: include: move activity to haproxy/
    - REORG: include: move common/buffer.h to haproxy/dynbuf{,-t}.h
    - REORG: include: move common/net_helper.h to haproxy/net_helper.h
    - REORG: include: move common/namespace.h to haproxy/namespace{,-t}.h
    - REORG: include: split common/regex.h into haproxy/regex{,-t}.h
    - REORG: include: split common/xref.h into haproxy/xref{,-t}.h
    - REORG: include: move common/ticks.h to haproxy/ticks.h
    - REORG: include: split common/http.h into haproxy/http{,-t}.h
    - REORG: include: split common/http-hdr.h into haproxy/http-hdr{,-t}.h
    - REORG: include: move common/h1.h to haproxy/h1.h
    - REORG: include: split common/htx.h into haproxy/htx{,-t}.h
    - REORG: include: move hpack*.h to haproxy/ and split hpack-tbl
    - REORG: include: move common/h2.h to haproxy/h2.h
    - REORG: include: move common/fcgi.h to haproxy/
    - REORG: include: move protocol.h to haproxy/protocol{,-t}.h
    - REORG: tools: split common/standard.h into haproxy/tools{,-t}.h
    - REORG: include: move dict.h to hparoxy/dict{,-t}.h
    - REORG: include: move shctx to haproxy/shctx{,-t}.h
    - REORG: include: move port_range.h to haproxy/port_range{,-t}.h
    - REORG: include: move fd.h to haproxy/fd{,-t}.h
    - REORG: include: move ring to haproxy/ring{,-t}.h
    - REORG: include: move sink.h to haproxy/sink{,-t}.h
    - REORG: include: move pipe.h to haproxy/pipe{,-t}.h
    - CLEANUP: include: remove empty raw_sock.h
    - REORG: include: move proto_udp.h to haproxy/proto_udp{,-t}.h
    - REORG: include: move proto/proto_sockpair.h to haproxy/proto_sockpair.h
    - REORG: include: move compression.h to haproxy/compression{,-t}.h
    - REORG: include: move h1_htx.h to haproxy/h1_htx.h
    - REORG: include: move http_htx.h to haproxy/http_htx{,-t}.h
    - REORG: include: move hlua.h to haproxy/hlua{,-t}.h
    - REORG: include: move hlua_fcn.h to haproxy/hlua_fcn.h
    - REORG: include: move action.h to haproxy/action{,-t}.h
    - REORG: include: move arg.h to haproxy/arg{,-t}.h
    - REORG: include: move auth.h to haproxy/auth{,-t}.h
    - REORG: include: move dns.h to haproxy/dns{,-t}.h
    - REORG: include: move flt_http_comp.h to haproxy/
    - REORG: include: move counters.h to haproxy/counters-t.h
    - REORG: include: split mailers.h into haproxy/mailers{,-t}.h
    - REORG: include: move capture.h to haproxy/capture{,-t}.h
    - REORG: include: move frontend.h to haproxy/frontend.h
    - REORG: include: move obj_type.h to haproxy/obj_type{,-t}.h
    - REORG: include: move http_rules.h to haproxy/http_rules.h
    - CLEANUP: include: remove unused mux_pt.h
    - REORG: include: move mworker.h to haproxy/mworker{,-t}.h
    - REORG: include: move ssl_utils.h to haproxy/ssl_utils.h
    - REORG: include: move ssl_ckch.h to haproxy/ssl_ckch{,-t}.h
    - REORG: move ssl_crtlist.h to haproxy/ssl_crtlist{,-t}.h
    - REORG: include: move lb_chash.h to haproxy/lb_chash{,-t}.h
    - REORG: include: move lb_fas.h to haproxy/lb_fas{,-t}.h
    - REORG: include: move lb_fwlc.h to haproxy/lb_fwlc{,-t}.h
    - REORG: include: move lb_fwrr.h to haproxy/lb_fwrr{,-t}.h
    - REORG: include: move listener.h to haproxy/listener{,-t}.h
    - REORG: include: move pattern.h to haproxy/pattern{,-t}.h
    - REORG: include: move map to haproxy/map{,-t}.h
    - REORG: include: move payload.h to haproxy/payload.h
    - REORG: include: move sample.h to haproxy/sample{,-t}.h
    - REORG: include: move protocol_buffers.h to haproxy/protobuf{,-t}.h
    - REORG: include: move vars.h to haproxy/vars{,-t}.h
    - REORG: include: split global.h into haproxy/global{,-t}.h
    - REORG: include: move task.h to haproxy/task{,-t}.h
    - REORG: include: move proto_tcp.h to haproxy/proto_tcp.h
    - REORG: include: move signal.h to haproxy/signal{,-t}.h
    - REORG: include: move tcp_rules.h to haproxy/tcp_rules.h
    - REORG: include: move connection.h to haproxy/connection{,-t}.h
    - REORG: include: move checks.h to haproxy/check{,-t}.h
    - REORG: include: move http_fetch.h to haproxy/http_fetch.h
    - REORG: include: move peers.h to haproxy/peers{,-t}.h
    - REORG: include: move stick_table.h to haproxy/stick_table{,-t}.h
    - REORG: include: move session.h to haproxy/session{,-t}.h
    - REORG: include: move trace.h to haproxy/trace{,-t}.h
    - REORG: include: move acl.h to haproxy/acl.h{,-t}.h
    - REORG: include: split common/uri_auth.h into haproxy/uri_auth{,-t}.h
    - REORG: move applet.h to haproxy/applet{,-t}.h
    - REORG: include: move stats.h to haproxy/stats{,-t}.h
    - REORG: include: move cli.h to haproxy/cli{,-t}.h
    - REORG: include: move lb_map.h to haproxy/lb_map{,-t}.h
    - REORG: include: move ssl_sock.h to haproxy/ssl_sock{,-t}.h
    - REORG: include: move stream_interface.h to haproxy/stream_interface{,-t}.h
    - REORG: include: move channel.h to haproxy/channel{,-t}.h
    - REORG: include: move http_ana.h to haproxy/http_ana{,-t}.h
    - REORG: include: move filters.h to haproxy/filters{,-t}.h
    - REORG: include: move fcgi-app.h to haproxy/fcgi-app{,-t}.h
    - REORG: include: move log.h to haproxy/log{,-t}.h
    - REORG: include: move proxy.h to haproxy/proxy{,-t}.h
    - REORG: include: move spoe.h to haproxy/spoe{,-t}.h
    - REORG: include: move backend.h to haproxy/backend{,-t}.h
    - REORG: include: move queue.h to haproxy/queue{,-t}.h
    - REORG: include: move server.h to haproxy/server{,-t}.h
    - REORG: include: move stream.h to haproxy/stream{,-t}.h
    - REORG: include: move cfgparse.h to haproxy/cfgparse.h
    - CLEANUP: hpack: export debug functions and move inlines to .h
    - REORG: check: move the e-mail alerting code to mailers.c
    - REORG: check: move tcpchecks away from check.c
    - REORG: check: move email_alert* from proxy-t.h to mailers-t.h
    - REORG: check: extract the external checks from check.{c,h}
    - CLEANUP: include: don't include stddef.h directly
    - CLEANUP: include: don't include proxy-t.h in global-t.h
    - CLEANUP: include: move sample_data out of sample-t.h
    - REORG: include: move the error reporting functions to from log.h to errors.h
    - BUILD: reorder objects in the Makefile for faster builds
    - CLEANUP: compiler: add a THREAD_ALIGNED macro and use it where appropriate
    - CLEANUP: include: make atomic.h part of the base API
    - REORG: include: move MAX_THREADS to defaults.h
    - REORG: include: move THREAD_LOCAL and __decl_thread() to compiler.h
    - CLEANUP: include: tree-wide alphabetical sort of include files
    - REORG: include: make list-t.h part of the base API
    - REORG: dgram: rename proto_udp to dgram
2020-06-11 10:22:10 +02:00
Tim Duesterhus
f38175cf6e MINOR: sample: Add secure_memcmp converter
secure_memcmp compares two binary strings in constant time. It's only
available when haproxy is compiled with USE_OPENSSL.
2020-06-09 22:04:13 +02:00
William Lallemand
7c26ed73f7 DOC: add a line about comments in crt-list
Add a line about comments in crt-list.

Fix issue #514.
2020-06-03 17:36:39 +02:00
Christopher Faulet
89aed32bff MINOR: mux-h1/proxy: Add a proxy option to disable clear h2 upgrade
By default, HAProxy is able to implicitly upgrade an H1 client connection to an
H2 connection if the first request it receives from a given HTTP connection
matches the HTTP/2 connection preface. This way, it is possible to support H1
and H2 clients on a non-SSL connections. It could be a problem if for any
reason, the H2 upgrade is not acceptable. "option disable-h2-upgrade" may now be
used to disable it, per proxy. The main puprose of this option is to let an
admin to totally disable the H2 support for security reasons. Recently, a
critical issue in the HPACK decoder was fixed, forcing everyone to upgrade their
HAProxy version to fix the bug. It is possible to disable H2 for SSL
connections, but not on clear ones. This option would have been a viable
workaround.
2020-06-03 10:23:39 +02:00
William Lallemand
50df1cb1e5 MINOR: ssl: set ssl-min-ver in ambiguous configurations
Using ssl-max-ver without ssl-min-ver is ambiguous.

When the ssl-min-ver is not configured, and ssl-max-ver is set to a
value lower than the default ssl-min-ver (which is TLSv1.2 currently),
set the ssl-min-ver to the value of ssl-max-ver, and emit a warning.
2020-06-02 11:13:12 +02:00
Emeric Brun
975564784f MEDIUM: ring: add new srv statement to support octet counting forward
log-proto <logproto>
  The "log-proto" specifies the protocol used to forward event messages to
  a server configured in a ring section. Possible values are "legacy"
  and "octet-count" corresponding respectively to "Non-transparent-framing"
  and "Octet counting" in rfc6587. "legacy" is the default.

Notes: a separated io_handler was created to avoid per messages test
and to prepare code to set different log protocols such as
request- response based ones.
2020-05-31 10:49:43 +02:00
Emeric Brun
494c505703 MEDIUM: ring: add server statement to forward messages from a ring
This patch adds new statement "server" into ring section, and the
related "timeout connect" and "timeout server".

server <name> <address> [param*]
  Used to configure a syslog tcp server to forward messages from ring buffer.
  This supports for all "server" parameters found in 5.2 paragraph.
  Some of these parameters are irrelevant for "ring" sections.

timeout connect <timeout>
  Set the maximum time to wait for a connection attempt to a server to succeed.

  Arguments :
    <timeout> is the timeout value specified in milliseconds by default, but
              can be in any other unit if the number is suffixed by the unit,
              as explained at the top of this document.

timeout server <timeout>
  Set the maximum time for pending data staying into output buffer.

  Arguments :
    <timeout> is the timeout value specified in milliseconds by default, but
              can be in any other unit if the number is suffixed by the unit,
              as explained at the top of this document.

  Example:
    global
        log ring@myring local7

    ring myring
        description "My local buffer"
        format rfc3164
        maxlen 1200
        size 32764
        timeout connect 5s
        timeout server 10s
        server mysyslogsrv 127.0.0.1:6514
2020-05-31 10:46:13 +02:00
Christopher Faulet
612f2eafe9 MINOR: http-ana: Use proxy's error replies to emit 401/407 responses
There is no reason to not use proxy's error replies to emit 401/407
responses. The function http_reply_40x_unauthorized(), responsible to emit those
responses, is not really complex. It only adds a
WWW-Authenticate/Proxy-Authenticate header to a generic message.

So now, error replies can be defined for 401 and 407 status codes, using
errorfile or http-error directives. When an http-request auth rule is evaluated,
the corresponding error reply is used. For 401 responses, all occurrences of the
WWW-Authenticate header are removed and replaced by a new one with a basic
authentication challenge for the configured realm. For 407 responses, the same
is done on the Proxy-Authenticate header. If the error reply must not be
altered, "http-request return" rule must be used instead.
2020-05-28 15:07:20 +02:00
Emeric Brun
99c453df9d MEDIUM: ring: new section ring to declare custom ring buffers.
It is possible to globally declare ring-buffers, to be used as target for log
servers or traces.

ring <ringname>
  Creates a new ring-buffer with name <ringname>.

description <text>
  The descritpition is an optional description string of the ring. It will
  appear on CLI. By default, <name> is reused to fill this field.

format <format>
  Format used to store events into the ring buffer.

  Arguments:
    <format> is the log format used when generating syslog messages. It may be
             one of the following :

      iso     A message containing only the ISO date, followed by the text.
              The PID, process name and system name are omitted. This is
              designed to be used with a local log server.

      raw     A message containing only the text. The level, PID, date, time,
              process name and system name are omitted. This is designed to be
              used in containers or during development, where the severity
              only depends on the file descriptor used (stdout/stderr). This
              is the default.

      rfc3164 The RFC3164 syslog message format. This is the default.
              (https://tools.ietf.org/html/rfc3164)

      rfc5424 The RFC5424 syslog message format.
              (https://tools.ietf.org/html/rfc5424)

      short   A message containing only a level between angle brackets such as
              '<3>', followed by the text. The PID, date, time, process name
              and system name are omitted. This is designed to be used with a
              local log server. This format is compatible with what the systemd
              logger consumes.

      timed   A message containing only a level between angle brackets such as
              '<3>', followed by ISO date and by the text. The PID, process
              name and system name are omitted. This is designed to be
              used with a local log server.

maxlen <length>
  The maximum length of an event message stored into the ring,
  including formatted header. If an event message is longer than
  <length>, it will be truncated to this length.

size <size>
  This is the optional size in bytes for the ring-buffer. Default value is
  set to BUFSIZE.

  Example:
    global
        log ring@myring local7

    ring myring
        description "My local buffer"
        format rfc3164
        maxlen 1200

Note: ring names are resolved during post configuration processing.
2020-05-26 08:03:15 +02:00
Willy Tarreau
73b943b964 [RELEASE] Released version 2.2-dev8
Released version 2.2-dev8 with the following main changes :
    - MINOR: checks: Improve report of unexpected errors for expect rules
    - MEDIUM: checks: Add matching on log-format string for expect rules
    - DOC: Fix req.body and co documentation to be accurate
    - MEDIUM: checks: Remove dedicated sample fetches and use response ones instead
    - CLEANUP: checks: sort and rename tcpcheck_expect_type types
    - MINOR: checks: Use dedicated actions to send log-format strings in send rules
    - MINOR: checks: Simplify matching on HTTP headers in HTTP expect rules
    - MINOR: checks/sample: Remove unnecessary tests on the sample session
    - REGTEST: checks: Adapt SSL error message reported when connection is rejected
    - MINOR: mworker: replace ha_alert by ha_warning when exiting successfuly
    - MINOR: checks: Support log-format string to set the URI for HTTP send rules
    - MINOR: checks: Support log-format string to set the body for HTTP send rules
    - DOC: Be more explicit about configurable check ok/error/timeout status
    - MINOR: checks: Make matching on HTTP headers for expect rules less obscure
    - BUG/MEDIUM: lua: Fix dumping of stick table entries for STD_T_DICT
    - BUG/MINOR: config: Make use_backend and use-server post-parsing less obscur
    - REGTESTS: make the http-check-send test require version 2.2
    - BUG/MINOR: http-ana: fix NTLM response parsing again
    - BUG/MEDIUM: http_ana: make the detection of NTLM variants safer
    - BUG/MINOR: cfgparse: Abort parsing the current line if an invalid \x sequence is encountered
    - MINOR: cfgparse: Improve error message for invalid \x sequences
    - CI: travis-ci: enable arm64 builds again
    - MEDIUM: ssl: increase default-dh-param to 2048
    - CI: travis-ci: skip pcre2 on arm64 build
    - CI: travis-ci: extend the build time for SSL to 60 minutes
    - CLEANUP: config: drop unused setting CONFIG_HAP_MEM_OPTIM
    - CLEANUP: config: drop unused setting CONFIG_HAP_INLINE_FD_SET
    - CLENAUP: config: move CONFIG_HAP_LOCKLESS_POOLS out of config.h
    - CLEANUP: remove THREAD_LOCAL from config.h
    - CI: travis-ci: upgrade LibreSSL versions
    - DOC: assorted typo fixes in the documentation
    - CI: extend spellchecker whitelist
    - CLEANUP: assorted typo fixes in the code and comments
    - MAJOR: contrib: porting spoa_server to support python3
    - BUG/MEDIUM: checks: Subscribe to I/O events on an unfinished connect
    - BUG/MINOR: checks: Don't subscribe to I/O events if it is already done
    - BUG/MINOR: checks: Rely on next I/O oriented rule when waiting for a connection
    - MINOR: checks: Don't try to send outgoing data if waiting to be able to send
    - MINOR: sample: Move aes_gcm_dec implementation into sample.c
    - MINOR: sample: Add digest and hmac converters
    - BUG/MEDIUM: checks: Subscribe to I/O events only if a mux was installed
    - BUG/MINOR: sample/ssl: Fix digest converter for openssl < 1.1.0
    - BUG/MINOR: pools: use %u not %d to report pool stats in "show pools"
    - BUG/MINOR: pollers: remove uneeded free in global init
    - CLEANUP: select: enhance readability in init
    - BUG/MINOR: soft-stop: always wake up waiting threads on stopping
    - MINOR: soft-stop: let the first stopper only signal other threads
    - BUILD: select: only declare existing local labels to appease clang
    - BUG/MEDIUM: streams: Remove SF_ADDR_SET if we're retrying due to L7 retry.
    - BUG/MEDIUM: stream: Only allow L7 retries when using HTTP.
    - DOC: retry-on can only be used with mode http
    - MEDIUM: ssl: allow to register callbacks for SSL/TLS protocol messages
    - MEDIUM: ssl: split ssl_sock_msgcbk() and use a new callback mechanism
    - MINOR: ssl: add a new function ssl_sock_get_ssl_object()
    - MEDIUM: ssl: use ssl_sock_get_ssl_object() in fetchers where appropriate
    - REORG: ssl: move macros and structure definitions to ssl_sock.h
    - CLEANUP: ssl: remove the shsess_* macros
    - REORG: move the crt-list structures in their own .h
    - REORG: ssl: move the ckch structures to types/ssl_ckch.h
    - CLEANUP: ssl: add ckch prototypes in proto/ssl_ckch.h
    - REORG: ssl: move crtlist functions to src/ssl_crtlist.c
    - CLEANUP: ssl: avoid circular dependencies in ssl_crtlist.h
    - REORG: ssl: move the ckch_store related functions to src/ssl_ckch.c
    - REORG: ssl: move ckch_inst functions to src/ssl_ckch.c
    - REORG: ssl: move the crt-list CLI functions in src/ssl_crtlist.c
    - REORG: ssl: move the CLI 'cert' functions to src/ssl_ckch.c
    - REORG: ssl: move ssl configuration to cfgparse-ssl.c
    - MINOR: ssl: remove static keyword in some SSL utility functions
    - REORG: ssl: move ssl_sock_ctx and fix cross-dependencies issues
    - REORG: ssl: move sample fetches to src/ssl_sample.c
    - REORG: ssl: move utility functions to src/ssl_utils.c
    - DOC: ssl: update MAINTAINERS file
    - CI: travis-ci: switch arm64 builds to use openssl from distro
    - MINOR: stats: Prepare for more accurate moving averages
    - MINOR: stats: Expose native cum_req metric for a server
    - MEDIUM: stats: Enable more accurate moving average calculation for stats
    - BUILD: ssl: include buffer common headers for ssl_sock_ctx
    - BUILD: ssl: include errno.h in ssl_crtlist.c
    - CLEANUP: acl: remove unused assignment
    - DOC/MINOR: halog: Add long help info for ic flag
    - BUILD: ssl: fix build without OPENSSL_NO_ENGINE
    - DOC: SPOE is no longer experimental
    - BUG/MINOR: cache: Don't needlessly test "cache" keyword in parse_cache_flt()
    - MINOR: config: Don't dump keywords if argument is NULL
    - MEDIUM: checks: Make post-41 the default mode for mysql checks
    - BUG/MINOR: logs: prevent double line returns in some events.
    - MEDIUM: sink: build header in sink_write for log formats
    - MEDIUM: logs: buffer targets now rely on new sink_write
    - MEDIUM: sink: add global statement to create a new ring (sink buffer)
    - MEDIUM: hpack: use a pool for the hpack table
    - BUG/MAJOR: mux-fcgi: Stop sending loop if FCGI stream is blocked for any reason
    - BUG/MEDIUM: ring: write-lock the ring while attaching/detaching
    - MINOR: applet: adopt the wait list entry from the CLI
    - MINOR: ring: make the applet code not depend on the CLI
    - Revert "MEDIUM: sink: add global statement to create a new ring (sink buffer)"
    - CI: travis-ci: fix libslz download URL
    - MINOR: ssl: split config and runtime variable for ssl-{min,max}-ver
    - CLEANUP: http_ana: Remove unused TXN flags
    - BUG/MINOR: http-rules: Mark http return rules as final
    - MINOR: http-htx: Add http_reply type based on what is used for http return rules
    - CLEANUP: http-htx: Rename http_error structure into http_error_msg
    - MINOR: http-rules: Use http_reply structure for http return rules
    - MINOR: http-htx: Use a dedicated function to release http_reply objects
    - MINOR: http-htx: Use a dedicated function to parse http reply arguments
    - MINOR: http-htx: Use a dedicated function to check http reply validity
    - MINOR: http-ana: Use a dedicated function to send a response from an http reply
    - MEDIUM: http-rules: Rely on http reply for http deny/tarpit rules
    - MINOR: http-htx: Store default error messages in a global http reply array
    - MINOR: http-htx: Store messages of an http-errors section in a http reply array
    - MINOR: http-htx: Store errorloc/errorfile messages in http replies
    - MINOR: proxy: Add references on http replies for proxy error messages
    - MINOR: http-htx: Use http reply from the http-errors section
    - MINOR: http-ana: Use a TXN flag to prevent after-response ruleset evaluation
    - MEDIUM: http-ana: Use http replies for HTTP error messages
    - CLEANUP: http-htx: Remove unused storage of error messages in buffers
    - MINOR: htx: Add a function to copy a buffer in an HTX message
    - CLEANUP: channel: Remove channel_htx_copy_msg() function
    - MINOR: http-ana: Add a function to write an http reply in an HTX message
    - MINOR: http-htx/proxy: Add http-error directive using http return syntax
    - DOC: Fix "errorfile" description in the configuration manual
    - BUG/MINOR: checks: Respect check-ssl param when a port or an addr is specified
    - BUILD: hpack: make sure the hpack table can still be built standalone
    - CONTRIB: hpack: make use of the simplified standalone HPACK API
    - MINOR: connection: add pp2-never-send-local to support old PP2 behavior
2020-05-22 16:19:04 +02:00
Willy Tarreau
119e50e0cc MINOR: connection: add pp2-never-send-local to support old PP2 behavior
A bug in the PROXY protocol v2 implementation was present in HAProxy up to
version 2.1, causing it to emit a PROXY command instead of a LOCAL command
for health checks. This is particularly minor but confuses some servers'
logs. Sadly, the bug was discovered very late and revealed that some servers
which possibly only tested their PROXY protocol implementation against
HAProxy fail to properly handle the LOCAL command, and permanently remain in
the "down" state when HAProxy checks them. When this happens, it is possible
to enable this global option to revert to the older (bogus) behavior for the
time it takes to contact the affected components' vendors and get them fixed.
This option is disabled by default and acts on all servers having the
"send-proxy-v2" statement.

Older versions were reverted to the old behavior and should not attempt to
be fixed by default again. However a variant of this patch could possibly
be implemented to ask to explicitly send LOCAL if needed by some servers.

More context here:
   https://www.mail-archive.com/haproxy@formilux.org/msg36890.html
   https://www.mail-archive.com/haproxy@formilux.org/msg37218.html
2020-05-22 13:55:32 +02:00
Christopher Faulet
7017067d68 DOC: Fix "errorfile" description in the configuration manual
"errorfile" description is outdated. Now, error messages are parsed at startup
and converted to HTX messages. So they must be valid according to the HTTP
standards. In addition, there is now a comment about the reserved buffer space
that should remain available to allow header rewrites.
2020-05-20 18:27:14 +02:00
Christopher Faulet
3b967c1210 MINOR: http-htx/proxy: Add http-error directive using http return syntax
The http-error directive can now be used instead of errorfile to define an error
message in a proxy section (including default sections). This directive uses the
same syntax that http return rules. The only real difference is the limitation
on status code that may be specified. Only status codes supported by errorfile
directives are supported for this new directive. Parsing of errorfile directive
remains independent from http-error parsing. But functionally, it may be
expressed in terms of http-errors :

  errorfile <status> <file> ==> http-errror status <status> errorfile <file>
2020-05-20 18:27:14 +02:00
Christopher Faulet
5cb513abeb MEDIUM: http-rules: Rely on http reply for http deny/tarpit rules
"http-request deny", "http-request tarpit" and "http-response deny" rules now
use the same syntax than http return rules and internally rely on the http
replies. The behaviour is not the same when no argument is specified (or only
the status code). For http replies, a dummy response is produced, with no
payload. For old deny/tarpit rules, the proxy's error messages are used. Thus,
to be compatible with existing configuration, the "default-errorfiles" parameter
is implied. For instance :

  http-request deny deny_status 404

is now an alias of

  http-request deny status 404 default-errorfiles
2020-05-20 18:27:13 +02:00
Willy Tarreau
d68a6927f7 Revert "MEDIUM: sink: add global statement to create a new ring (sink buffer)"
This reverts commit 957ec59571.

As discussed with Emeric, the current syntax is not extensible enough,
this will be turned to a section instead in a forthcoming patch.
2020-05-20 12:06:16 +02:00
Emeric Brun
957ec59571 MEDIUM: sink: add global statement to create a new ring (sink buffer)
This patch adds the new global statement:
ring <name> [desc <desc>] [format <format>] [size <size>] [maxlen <length>]
  Creates a named ring buffer which could be used on log line for instance.

  <desc> is an optionnal description string of the ring. It will appear on
         CLI. By default, <name> is reused to fill this field.

  <format> is the log format used when generating syslog messages. It may be
           one of the following :

    iso       A message containing only the ISO date, followed by the text.
              The PID, process name and system name are omitted. This is
              designed to be used with a local log server.

    raw       A message containing only the text. The level, PID, date, time,
              process name and system name are omitted. This is designed to be
              used in containers or during development, where the severity only
              depends on the file descriptor used (stdout/stderr). This is
              the default.

    rfc3164   The RFC3164 syslog message format. This is the default.
              (https://tools.ietf.org/html/rfc3164)

    rfc5424   The RFC5424 syslog message format.
              (https://tools.ietf.org/html/rfc5424)

    short     A message containing only a level between angle brackets such as
              '<3>', followed by the text. The PID, date, time, process name
              and system name are omitted. This is designed to be used with a
              local log server. This format is compatible with what the systemd
              logger consumes.

    timed     A message containing only a level between angle brackets such as
              '<3>', followed by ISO date and by the text. The PID, process
              name and system name are omitted. This is designed to be
              used with a local log server.

  <length> is the maximum length of event message stored into the ring,
           including formatted header. If the event message is longer
           than <length>, it would be truncated to this length.

  <name> is the ring identifier, which follows the same naming convention as
         proxies and servers.

  <size> is the optionnal size in bytes. Default value is set to BUFSIZE.

Note: Historically sink's name and desc were refs on const strings. But with new
configurable rings a dynamic allocation is needed.
2020-05-19 11:04:11 +02:00