1229 Commits

Author SHA1 Message Date
Willy Tarreau
b149eedd5a CLEANUP: da: register the deinitialization function
deinit_deviceatlas() is not called anymore from haproxy.c, removing 2
still includes other parts of the Deviceatlas library so it was not
touched.
2016-12-21 21:30:54 +01:00
Willy Tarreau
7ac4c20509 CLEANUP: 51d: register the deinitialization function
deinit_51degrees() is not called anymore from haproxy.c, removing
2 #ifdefs and one include. The function was made static. The include
file still includes 51Degrees.h which is needed by global.h and 51d.c
so it was not touched beyond this last function removal.
2016-12-21 21:30:54 +01:00
Willy Tarreau
800f93f375 CLEANUP: wurfl: register the deinit function via the dedicated list
By registering the deinit function we avoid another #ifdef in haproxy.c.
The ha_wurfl_deinit() function has been made static and unexported. Now
proto/wurfl.h is totally empty, the code being self-contained in wurfl.c,
so the useless .h has been removed.
2016-12-21 21:30:54 +01:00
Willy Tarreau
05554e6bf1 MINOR: haproxy: add a registration for post-deinit functions
The 3 device detection engines stop at the same place in deinit()
with the usual #ifdefs. Similar to the other functions we can have
some late deinitialization functions. These functions do not return
anything however so we have to use a different type.
2016-12-21 21:30:54 +01:00
Willy Tarreau
876054df96 CLEANUP: da: make use of the late init registration code
Instead of having a #ifdef in the main init code we now use the registered
init functions. Doing so also enables error checking as errors were previously
reported as alerts but ignored. Also they were incorrect as the 'status'
variable was hidden by a second one and was always reporting DA_SYS (which
is apparently an error) in every case including the case where no file was
loaded. The init_deviceatlas() function was unexported since it's not used
outside of this place anymore.
2016-12-21 21:30:54 +01:00
Willy Tarreau
9f3f2549fb CLEANUP: 51d: make use of the late init registration
This removes some #ifdefs from the main haproxy code path. Function
init_51degrees() now returns ERR_* instead of exit(1) on error, and
this function was made static and is not exported anymore.
2016-12-21 21:30:54 +01:00
Willy Tarreau
dc2ed47163 CLEANUP: wurfl: make use of the late init registration
This removes some #ifdefs from the main haproxy code path and enables
error checking. The current code only makes use of warnings even for
some errors that look serious. While this choice is questionnable, it
has been kept as-is, and only the return codes were adapted to ERR_WARN
to at least report that some warnings were emitted. ha_wurfl_init() was
unexported as it's not needed anymore.
2016-12-21 21:30:54 +01:00
Willy Tarreau
64bca599d9 CLEANUP: filters: use the function registration to initialize all proxies
Function flt_init() was called in the main init code path, now we move
it to the list of initializers and we can unexport flt_init().
2016-12-21 21:30:54 +01:00
Willy Tarreau
865c5148e6 CLEANUP: checks: make use of the post-init registration to start checks
Instead of calling the checks directly from the init code, we now
register the start_checks() function to be run at this point. This
also allows to unexport the check init function and to remove one
include from haproxy.c.
2016-12-21 21:30:54 +01:00
Willy Tarreau
e694573fa0 MINOR: haproxy: add a registration for post-check functions
There's a significant amount of late initialization calls which are
performed after the point where we exit in check mode. These calls
are used to allocate resource and perform certain slow operations.
Let's have a way to register some functions which need to be called
there instead of having this multitude of #ifdef in the init path.
2016-12-21 21:30:54 +01:00
Willy Tarreau
e8692b41e5 CLEANUP: auth: use the build options list to report its support
This removes 1 #ifdef from haproxy.c.
2016-12-21 21:30:54 +01:00
Willy Tarreau
b97c6fb59e CLEANUP: compression: use the build options list to report the algos
This removes 2 #ifdef, an include, an ugly construct and a wild "extern"
declaration from haproxy.c. The message indicating that compression is
*not* enabled is not there anymore.
2016-12-21 21:30:54 +01:00
Willy Tarreau
c2c0b61274 CLEANUP: ssl: use the build options list to report the SSL details
This removes 7 #ifdef from haproxy.c. The message indicating that
openssl is *not* enabled is not there anymore.
2016-12-21 21:30:54 +01:00
Willy Tarreau
7a9ac6dac6 CLEANUP: regex: use the build options list to report the regex type
This removes 3 #ifdef from haproxy.c.
2016-12-21 21:30:54 +01:00
Willy Tarreau
bb57d94a96 CLEANUP: lua: use the build options list to report it
This removes 1 #ifdef from haproxy.c. The "build without" version
is not reported anymore now.
2016-12-21 21:30:54 +01:00
Willy Tarreau
ba96291600 CLEANUP: tcp: use the build options list to report transparent modes
This removes 6 #ifdef from haproxy.c.
2016-12-21 21:30:54 +01:00
Willy Tarreau
dba5002c4c CLEANUP: namespaces: use the build options list to report it
This removes one #ifdef from haproxy.c.
2016-12-21 21:30:54 +01:00
Willy Tarreau
3dd483e727 CLEANUP: da: use the build options list to report it
This removes one #ifdef from haproxy.c.
2016-12-21 21:30:54 +01:00
Willy Tarreau
b5e58d6ba1 CLEANUP: 51d: use the build options list to report it
This removes one #ifdef from haproxy.c.
2016-12-21 21:30:54 +01:00
Willy Tarreau
770042d3c6 CLEANUP: wurfl: use the build options list to report it
This removes one #ifdef from haproxy.c.
2016-12-21 21:30:54 +01:00
Willy Tarreau
cdb737e5a2 MINOR: haproxy: add a registration for build options
Many extensions now report some build options to ease debugging, but
this is now being done at the expense of code maintainability. Let's
provide a registration function to do this so that we can start to
remove most of the #ifdefs from haproxy.c (18 currently just for a
single function).
2016-12-21 21:30:54 +01:00
Willy Tarreau
1b5af7cd42 CLEANUP: haproxy: statify unexported functions
haproxy.c is a real mess. Let's start to clean it up by declaring static
all functions which are not exported (ie almost all of them).
2016-12-21 18:19:57 +01:00
Christopher Faulet
34c5cc98da MINOR: task: Rename run_queue and run_queue_cur counters
<run_queue> is used to track the number of task in the run queue and
<run_queue_cur> is a copy used for the reporting purpose. These counters has
been renamed, respectively, <tasks_run_queue> and <tasks_run_queue_cur>. So the
naming is consistent between tasks and applets.

[wt: needed for next fixes, backport to 1.7 and 1.6]
2016-12-12 19:10:54 +01:00
Christopher Faulet
1cbe74cd83 MINOR: applet: Count number of (active) applets
As for tasks, 2 counters has been added to track :
  * the total number of applets : nb_applets
  * the number of active applets : applets_active_queue

[wt: needed for next fixes, to backport to 1.7 and 1.6]
2016-12-12 19:10:46 +01:00
Thierry FOURNIER / OZON.IO
4ed1c9585d MINOR: http/conf: store the use_backend configuration file and line for logs
The error log of the directive use_backend doesn't provide the
file and line containing the declaration. This patch stores
theses informations.
2016-11-25 07:15:09 +01:00
Christopher Faulet
ff2613ed7a MEDIUM: vars: Add a per-process scope for variables
Now it is possible to use variables attached to a process. The scope name is
'proc'. These variables are released only when HAProxy is stopped.

'tune.vars.proc-max-size' directive has been added to confiure the maximum
amount of memory used by "proc" variables. And because memory accounting is
hierachical for variables, memory for "proc" vars includes memory for "sess"
vars.
2016-11-09 22:57:00 +01:00
Christopher Faulet
476e5d0e03 REORG: sample: move code to release a sample expression in sample.c
This code has been moved from haproxy.c to sample.c and the function
release_sample_expr can now be called from anywhere to release a sample
expression. This function will be used by the stream processing offload engine
(SPOE).
2016-11-09 22:57:00 +01:00
Willy Tarreau
3eed10e54b MINOR: init: add -dr to ignore server address resolution failures
It is very common when validating a configuration out of production not to
have access to the same resolvers and to fail on server address resolution,
making it difficult to test a configuration. This option simply appends the
"none" method to the list of address resolution methods for all servers,
ensuring that even if the libc fails to resolve an address, the startup
sequence is not interrupted.
2016-11-09 15:33:52 +01:00
Baptiste Assmann
83cbaa531f MAJOR: server: postpone address resolution
Server addresses are not resolved anymore upon the first pass so that we
don't fail if an address cannot be resolved by the libc. Instead they are
processed all at once after the configuration is fully loaded, by the new
function srv_init_addr(). This function only acts on the server's address
if this address uses an FQDN, which appears in server->hostname.

For now the function does two things, to followup with HAProxy's historical
default behavior:

  1. apply server IP address found in server-state file if runtime DNS
     resolution is enabled for this server

  2. use the DNS resolver provided by the libc

If none of the 2 options above can find an IP address, then an error is
returned.

All of this will be needed to support the new server parameter "init-addr".
For now, the biggest user-visible change is that all server resolution errors
are dumped at once instead of causing a startup failure one by one.
2016-11-09 14:24:20 +01:00
Baptiste Assmann
4215d7d033 MINOR: init: move apply_server_state in haproxy.c before MODE_CHECK
Currently, the function which applies server states provided by the
"old" process is applied after configuration sanity check. This results
in the impossibility to check the validity of the state file during a
regular config check, implying a full start is required, which can be
a problem sometimes.

This patch moves the loading of server_state file before MODE_CHECK.
2016-11-09 14:24:20 +01:00
Willy Tarreau
e5d3169e1c CLEANUP: wurfl: reduce exposure in the rest of the code
The only reason wurfl/wurfl.h was needed outside of wurfl.c was to expose
wurfl_handle which is a pointer to a structure, referenced by global.h.
By just storing a void* there instead, we can confine all wurfl code to
wurfl.c, which is really nice.
2016-11-08 18:47:25 +01:00
scientiamobile
d0027ed5b1 MEDIUM: wurfl: add Scientiamobile WURFL device detection module
WURFL is a high-performance and low-memory footprint mobile device
detection software component that can quickly and accurately detect
over 500 capabilities of visiting devices. It can differentiate between
portable mobile devices, desktop devices, SmartTVs and any other types
of devices on which a web browser can be installed.

In order to add WURFL device detection support, you would need to
download Scientiamobile InFuze C API and install it on your system.
Refer to www.scientiamobile.com to obtain a valid InFuze license.

Any useful information on how to configure HAProxy working with WURFL
may be found in:

  doc/WURFL-device-detection.txt
  doc/configuration.txt
  examples/wurfl-example.cfg

Please find more information about WURFL device detection API detection
at https://docs.scientiamobile.com/documentation/infuze/infuze-c-api-user-guide
2016-11-08 14:21:43 +01:00
Willy Tarreau
757478e900 BUG/MEDIUM: servers: properly propagate the maintenance states during startup
Right now there is an issue with the way the maintenance flags are
propagated upon startup. They are not propagate, just copied from the
tracked server. This implies that depending on the server's order, some
tracking servers may not be marked down. For example this configuration
does not work as expected :

        server s1 1.1.1.1:8000 track s2
        server s2 1.1.1.1:8000 track s3
        server s3 1.1.1.1:8000 track s4
        server s4 wtap:8000 check inter 1s disabled

It results in s1/s2 being up, and s3/s4 being down, while all of them
should be down.

The only clean way to process this is to run through all "root" servers
(those not tracking any other server), and to propagate their state down
to all their trackers. This is the same algorithm used to propagate the
state changes. It has to be done both to compute the IDRAIN flag and the
IMAINT flag. However, doing so requires that tracking servers are not
marked as inherited maintenance anymore while parsing the configuration
(and given that it is wrong, better drop it).

This fix also addresses another side effect of the bug above which is
that the IDRAIN/IMAINT flags are stored in the state files, and if
restored while the tracked server doesn't have the equivalent flag,
the servers may end up in a situation where it's impossible to remove
these flags. For example in the configuration above, after removing
"disabled" on server s4, the other servers would have remained down,
and not anymore with this fix. Similarly, the combination of IMAINT
or IDRAIN with their respective forced modes was not accepted on
reload, which is wrong as well.

This bug has been present at least since 1.5, maybe even 1.4 (it came
with tracking support). The fix needs to be backported there, though
the srv-state parts are irrelevant.

This commit relies on previous patch to silence warnings on startup.
2016-11-07 14:31:52 +01:00
Willy Tarreau
b957109727 BUG/MEDIUM: systemd: let the wrapper know that haproxy has completed or failed
Pierre Cheynier found that there's a persistent issue with the systemd
wrapper. Too fast reloads can lead to certain old processes not being
signaled at all and continuing to run. The problem was tracked down as
a race between the startup and the signal processing : nothing prevents
the wrapper from starting new processes while others are still starting,
and the resulting pid file will only contain the latest pids in this
case. This can happen with large configs and/or when a lot of SSL
certificates are involved.

In order to solve this we want the wrapper to wait for the new processes
to complete their startup. But we also want to ensure it doesn't wait for
nothing in case of error.

The solution found here is to create a pipe between the wrapper and the
sub-processes. The wrapper waits on the pipe and the sub-processes are
expected to close this pipe once they completed their startup. That way
we don't queue up new processes until the previous ones have registered
their pids to the pid file. And if anything goes wrong, the wrapper is
immediately released. The only thing is that we need the sub-processes
to know the pipe's file descriptor. We pass it in an environment variable
called HAPROXY_WRAPPER_FD.

It was confirmed both by Pierre and myself that this completely solves
the "zombie" process issue so that only the new processes continue to
listen on the sockets.

It seems that in the future this stuff could be moved to the haproxy
master process, also getting rid of an environment variable.

This fix needs to be backported to 1.6 and 1.5.
2016-10-25 17:43:45 +02:00
Lukas Tribus
a0bcbdcb04 MEDIUM: make SO_REUSEPORT configurable
With Linux officially introducing SO_REUSEPORT support in 3.9 and
its mainstream adoption we have seen more people running into strange
SO_REUSEPORT related issues (a process management issue turning into
hard to diagnose problems because the kernel load-balances between the
new and an obsolete haproxy instance).

Also some people simply want the guarantee that the bind fails when
the old process is still bound.

This change makes SO_REUSEPORT configurable, introducing the command
line argument "-dR" and the noreuseport configuration directive.

A backport to 1.6 should be considered.
2016-09-13 07:56:03 +02:00
Lukas Tribus
255cc5184d MINOR: show Running on zlib version 2016-09-13 07:55:59 +02:00
Lukas Tribus
dcbc5c5ecf MINOR: show Built with PCRE version
Inspired by PCRE's pcre_version.c and improved with Willy's
suggestions. Reusable parts have been added to
include/common/standard.h.
2016-09-13 07:55:51 +02:00
Lukas Tribus
d64788d9c6 BUG/MINOR: displayed PCRE version is running release
pcre_version() returns the running PCRE release, not the release
haproxy was built with.

This simple string fix should be backported to supported releases,
as the output may be confusing.
2016-09-13 07:55:46 +02:00
Willy Tarreau
164dd0b6e4 BUG/MINOR: init: ensure that FD limit is raised to the max allowed
When the requested amount of FDs cannot be allocated, setrlimit() fails.
That's bad because if the limit is set to 1024 and we need 10000, we
stay on 1024 while we could possibly raise it to 4096 thanks to rlim_max.
This patch takes care of trying to assign rlim_cur to rlim_max on failure
so that we get as much as possible if we can't get all we need. The case
is particularly visible when starting haproxy as a non-privileged user
and a large maxconn is specified in the configuration.

Another point of doing this is that it is the only way to allow us to
close inherited FDs upon fork(), ie those between rlim_cur and rlim_max.

This patch may be backported to 1.6 and 1.5.
2016-06-21 18:10:50 +02:00
Willy Tarreau
ef6354719b BUG/MINOR: init: always ensure that global.rlimit_nofile matches actual limits
global.rlimit_nofile contains the mxa number of file descriptors that
can be allocated, except if the user is not allowed to reach this limit,
where it still contains the initially requested value. It is important
that this value always matches what is really configured so that it is
properly reported in the stats and that we can use it later to close
all FDs without wasting time closing impossible FDs.

This fix may be backported to 1.6 and 1.5.
2016-06-21 18:10:50 +02:00
Maxime de Roucy
e3841395ad BUG/MEDIUM: init: don't use environment locale
This patch removes setlocale from the main function. It was introduced
by commit 379d9c7 ("MEDIUM: init: allow directory as argument of -f")
in 1.7-dev a few commits ago after a discussion on the mailing list.

Some regex may have different behaviours depending on the
locale. Some LUA scripts may change their behaviour too
(http://lua-users.org/wiki/LuaLocales).

Without this patch (haproxy is using setlocale) :

	$ cat locale.cfg
	defaults
	  mode http

	frontend test
	  bind :9000
	  mode http
	  use_backend testbk if { hdr_reg(X-Test) ^\w+$ }

	backend testbk
	  mode http
	  server s 127.0.0.1:80

	$ LANG=fr_FR.UTF-8 ./haproxy -f locale.cfg
	$ curl -i -H "X-Test: échec" localhost:9000
	HTTP/1.1 200 OK
	...

	$ LANG=C ./haproxy -f locale.cfg
	$ curl -i -H "X-Test: échec" localhost:9000
	HTTP/1.0 503 Service Unavailable
	...
2016-05-19 07:19:19 +02:00
Maxime de Roucy
379d9c7c14 MEDIUM: init: allow directory as argument of -f
If -f argument is a directory add all the files (and only files) it
containes to the config files list.
These files are added in lexical order (respecting LC_COLLATE).
Only files with ".cfg" extension are added.
Only non hidden files (not prefixed with ".") are added.
Symlink are followed.
The -f order is still respected:

        $ tree -a rootdir
        rootdir
        |-- dir1
        |   |-- .6.cfg
        |   |-- 1.cfg
        |   |-- 2
        |   |-- 3.cfg
        |   |-- 4.cfg -> 1.cfg
        |   |-- 5 -> 1.cfg
        |   |-- 7.cfg -> .
        |   `-- dir4
        |       `-- 8.cfg
        |-- dir2
        |   |-- 10.cfg
        |   `-- 9.cfg
        |-- dir3
        |   `-- 11.cfg
        |-- link -> dir3/
        |-- root1
        |-- root2
        `-- root3

        $ ./haproxy -C rootdir -f root2 -f dir2 -f root3 -f dir1 \
                               -f link -f root1
        root2
        dir2/10.cfg
        dir2/9.cfg
        root3
        dir1/1.cfg
        dir1/3.cfg
        dir1/4.cfg
        link/11.cfg
        root1

This can be useful on systemd where you can't change the haproxy
commande line options on service reload.
2016-05-14 07:09:33 +02:00
Maxime de Roucy
0f503925f0 MEDIUM: init: use list_append_word in haproxy.c
replace LIST_ADDQ with list_append_word
2016-05-14 00:00:54 +02:00
Willy Tarreau
7d1b48fae0 [RELEASE] Released version 1.7-dev3
Released version 1.7-dev3 with the following main changes :
    - MINOR: sample: Moves ARGS underlying type from 32 to 64 bits.
    - BUG/MINOR: log: Don't use strftime() which can clobber timezone if chrooted
    - BUILD: namespaces: fix a potential build warning in namespaces.c
    - MINOR: da: Using ARG12 macro for the sample fetch and the convertor.
    - DOC: add encoding to json converter example
    - BUG/MINOR: conf: "listener id" expects integer, but its not checked
    - DOC: Clarify tunes.vars.xxx-max-size settings
    - CLEANUP: chunk: adding NULL check to chunk_dup allocation.
    - CLEANUP: connection: fix double negation on memcmp()
    - BUG/MEDIUM: peers: fix incorrect age in frequency counters
    - BUG/MEDIUM: Fix RFC5077 resumption when more than TLS_TICKETS_NO are present
    - BUG/MAJOR: Fix crash in http_get_fhdr with exactly MAX_HDR_HISTORY headers
    - BUG/MINOR: lua: can't load external libraries
    - BUG/MINOR: prevent the dump of uninitialized vars
    - CLEANUP: map: it seems that the map were planed to be chained
    - MINOR: lua: move class registration facilities
    - MINOR: lua: remove some useless checks
    - CLEANUP: lua: Remove two same functions
    - MINOR: lua: refactor the Lua object registration
    - MINOR: lua: precise message when a critical error is catched
    - MINOR: lua: post initialization
    - MINOR: lua: Add internal function which strip spaces
    - MINOR: lua: convert field to lua type
    - DOC: "addr" parameter applies to both health and agent checks
    - DOC: timeout client: pointers to timeout http-request
    - DOC: typo on stick-store response
    - DOC: stick-table: amend paragraph blaming the loss of table upon reload
    - DOC: typo: ACL subdir match
    - DOC: typo: maxconn paragraph is wrong due to a wrong buffer size
    - DOC: regsub: parser limitation about the inability to use closing square brackets
    - DOC: typo: req.uri is now replaced by capture.req.uri
    - DOC: name set-gpt0 mismatch with the expected keyword
    - MINOR: http: sample fetch which returns unique-id
    - MINOR: dumpstats: extract stats fields enum and names
    - MINOR: dumpstats: split stats_dump_info_to_buffer() in two parts
    - MINOR: dumpstats: split stats_dump_fe_stats() in two parts
    - MINOR: dumpstats: split stats_dump_li_stats() in two parts
    - MINOR: dumpstats: split stats_dump_sv_stats() in two parts
    - MINOR: dumpstats: split stats_dump_be_stats() in two parts
    - MINOR: lua: dump general info
    - MINOR: lua: add class proxy
    - MINOR: lua: add class server
    - MINOR: lua: add class listener
    - BUG/MEDIUM: stick-tables: some sample-fetch doesn't work in the connection state.
    - MEDIUM: proxy: use dynamic allocation for error dumps
    - CLEANUP: remove unneeded casts
    - CLEANUP: uniformize last argument of malloc/calloc
    - DOC: fix "needed" typo
    - BUG/MINOR: dumpstats: fix write to global chunk
    - BUG/MINOR: dns: inapropriate way out after a resolution timeout
    - BUG/MINOR: dns: trigger a DNS query type change on resolution timeout
    - CLEANUP: proto_http: few corrections for gcc warnings.
    - BUG/MINOR: DNS: resolution structure change
    - BUG/MINOR : allow to log cookie for tarpit and denied request
    - BUG/MEDIUM: ssl: rewind the BIO when reading certificates
    - OPTIM/MINOR: session: abort if possible before connecting to the backend
    - DOC: http: rename the unique-id sample and add the documentation
    - BUG/MEDIUM: trace.c: rdtsc() is defined in two files
    - BUG/MEDIUM: channel: fix miscalculation of available buffer space (2nd try)
    - BUG/MINOR: server: risk of over reading the pref_net array.
    - BUG/MINOR: cfgparse: couple of small memory leaks.
    - BUG/MEDIUM: sample: initialize the pointer before parse_binary call.
    - DOC: fix discrepancy in the example for http-request redirect
    - MINOR: acl: Add predefined METH_DELETE, METH_PUT
    - CLEANUP: .gitignore cleanup
    - DOC: Clarify IPv4 address / mask notation rules
    - CLEANUP: fix inconsistency between fd->iocb, proto->accept and accept()
    - BUG/MEDIUM: fix maxaccept computation on per-process listeners
    - BUG/MINOR: listener: stop unbound listeners on startup
    - BUG/MINOR: fix maxaccept computation according to the frontend process range
    - TESTS: add blocksig.c to run tests with all signals blocked
    - MEDIUM: unblock signals on startup.
    - MINOR: filters: Print the list of existing filters during HA startup
    - MINOR: filters: Typo in an error message
    - MINOR: filters: Filters must define the callbacks struct during config parsing
    - DOC: filters: Add filters documentation
    - BUG/MEDIUM: channel: don't allow to overwrite the reserve until connected
    - BUG/MEDIUM: channel: incorrect polling condition may delay event delivery
    - BUG/MEDIUM: channel: fix miscalculation of available buffer space (3rd try)
    - BUG/MEDIUM: log: fix risk of segfault when logging HTTP fields in TCP mode
    - MINOR: Add ability for agent-check to set server maxconn
    - CLEANUP: Use server_parse_maxconn_change_request for maxconn CLI updates
    - MINOR: filters: add opaque data
    - BUG/MEDIUM: lua: protects the upper boundary of the argument list for converters/fetches.
    - MINOR: lua: migrate the argument mask to 64 bits type.
    - BUG/MINOR: dumpstats: Fix the "Total bytes saved" counter in backends stats
    - BUG/MINOR: log: fix a typo that would cause %HP to log <BADREQ>
    - BUG/MEDIUM: http: fix incorrect reporting of server errors
    - MINOR: channel: add new function channel_congested()
    - BUG/MEDIUM: http: fix risk of CPU spikes with pipelined requests from dead client
    - BUG/MAJOR: channel: fix miscalculation of available buffer space (4th try)
    - BUG/MEDIUM: stream: ensure the SI_FL_DONT_WAKE flag is properly cleared
    - BUG/MEDIUM: channel: fix inconsistent handling of 4GB-1 transfers
    - BUG/MEDIUM: stats: show servers state may show an empty or incomplete result
    - BUG/MEDIUM: stats: show backend may show an empty or incomplete result
    - MINOR: stats: fix typo in help messages
    - MINOR: stats: show stat resolvers missing in the help message
    - BUG/MINOR: dns: fix DNS header definition
    - BUG/MEDIUM: dns: fix alignment issue when building DNS queries
    - CLEANUP: don't ignore scripts in .gitignore
    - BUILD: add a few release and backport scripts in scripts/
2016-05-10 15:36:58 +02:00
Christopher Faulet
b3f4e14932 MINOR: filters: Print the list of existing filters during HA startup
This is done  in verbose/debug mode and when build options are reported.
2016-04-21 06:58:08 +02:00
Vincent Bernat
3c2f2f207f CLEANUP: remove unneeded casts
In C89, "void *" is automatically promoted to any pointer type. Casting
the result of malloc/calloc to the type of the LHS variable is therefore
unneeded.

Most of this patch was built using this Coccinelle patch:

@@
type T;
@@

- (T *)
  (\(lua_touserdata\|malloc\|calloc\|SSL_get_app_data\|hlua_checkudata\|lua_newuserdata\)(...))

@@
type T;
T *x;
void *data;
@@

  x =
- (T *)
  data

@@
type T;
T *x;
T *data;
@@

  x =
- (T *)
  data

Unfortunately, either Coccinelle or I is too limited to detect situation
where a complex RHS expression is of type "void *" and therefore casting
is not needed. Those cases were manually examined and corrected.
2016-04-03 14:17:42 +02:00
Willy Tarreau
8234f6dae8 [RELEASE] Released version 1.7-dev2
Released version 1.7-dev2 with the following main changes :
    - DOC: lua: fix lua API
    - DOC: mailers: typo in 'hostname' description
    - DOC: compression: missing mention of libslz for compression algorithm
    - BUILD/MINOR: regex: missing header
    - BUG/MINOR: stream: bad return code
    - DOC: lua: fix somme errors and add implicit types
    - MINOR: lua: add set/get priv for applets
    - BUG/MINOR: http: fix several off-by-one errors in the url_param parser
    - BUG/MINOR: http: Be sure to process all the data received from a server
    - MINOR: filters/http: Use a wrapper function instead of stream_int_retnclose
    - BUG/MINOR: chunk: make chunk_dup() always check and set dst->size
    - DOC: ssl: fixed some formatting errors in crt tag
    - MINOR: chunks: ensure that chunk_strcpy() adds a trailing zero
    - MINOR: chunks: add chunk_strcat() and chunk_newstr()
    - MINOR: chunk: make chunk_initstr() take a const string
    - MEDIUM: tools: add csv_enc_append() to preserve the original chunk
    - MINOR: tools: make csv_enc_append() always start at the first byte of the chunk
    - MINOR: lru: new function to delete <nb> least recently used keys
    - DOC: add Ben Shillito as the maintainer of 51d
    - BUG/MINOR: 51d: Ensures a unique domain for each configuration
    - BUG/MINOR: 51d: Aligns Pattern cache implementation with HAProxy best practices.
    - BUG/MINOR: 51d: Releases workset back to pool.
    - BUG/MINOR: 51d: Aligned const pointers to changes in 51Degrees.
    - CLEANUP: 51d: Aligned if statements with HAProxy best practices and removed casts from malloc.
    - MINOR: rename master process name in -Ds (systemd mode)
    - DOC: fix a few spelling mistakes
    - DOC: fix "workaround" spelling
    - BUG/MINOR: examples: Fixing haproxy.spec to remove references to .cfg files
    - MINOR: fix the return type for dns_response_get_query_id() function
    - MINOR: server state: missing LF (\n) on error message printed when parsing server state file
    - BUG/MEDIUM: dns: no DNS resolution happens if no ports provided to the nameserver
    - BUG/MAJOR: servers state: server port is erased when dns resolution is enabled on a server
    - BUG/MEDIUM: servers state: server port is used uninitialized
    - BUG/MEDIUM: config: Adding validation to stick-table expire value.
    - BUG/MEDIUM: sample: http_date() doesn't provide the right day of the week
    - BUG/MEDIUM: channel: fix miscalculation of available buffer space.
    - MEDIUM: pools: add a new flag to avoid rounding pool size up
    - BUG/MEDIUM: buffers: do not round up buffer size during allocation
    - BUG/MINOR: stream: don't force retries if the server is DOWN
    - BUG/MINOR: counters: make the sc-inc-gpc0 and sc-set-gpt0 touch the table
    - MINOR: unix: don't mention free ports on EAGAIN
    - BUG/CLEANUP: CLI: report the proper field states in "show sess"
    - MINOR: stats: send content-length with the redirect to allow keep-alive
    - BUG: stream_interface: Reuse connection even if the output channel is empty
    - DOC: remove old tunnel mode assumptions
    - BUG/MAJOR: http-reuse: fix risk of orphaned connections
    - BUG/MEDIUM: http-reuse: do not share private connections across backends
    - BUG/MINOR: ssl: Be sure to use unique serial for regenerated certificates
    - BUG/MINOR: stats: fix missing comma in stats on agent drain
    - MAJOR: filters: Add filters support
    - MINOR: filters: Do not reset stream analyzers if the client is gone
    - REORG: filters: Prepare creation of the HTTP compression filter
    - MAJOR: filters/http: Rewrite the HTTP compression as a filter
    - MEDIUM: filters: Use macros to call filters callbacks to speed-up processing
    - MEDIUM: filters: remove http_start_chunk, http_last_chunk and http_chunk_end
    - MEDIUM: filters: Replace filter_http_headers callback by an analyzer
    - MEDIUM: filters/http: Move body parsing of HTTP messages in dedicated functions
    - MINOR: filters: Add stream_filters structure to hide filters info
    - MAJOR: filters: Require explicit registration to filter HTTP body and TCP data
    - MINOR: filters: Remove unused or useless stuff and do small optimizations
    - MEDIUM: filters: Optimize the HTTP compression for chunk encoded response
    - MINOR: filters/http: Slightly update the parsing of chunks
    - MINOR: filters/http: Forward remaining data when a channel has no "data" filters
    - MINOR: filters: Add an filter example
    - MINOR: filters: Extract proxy stuff from the struct filter
    - MINOR: map: Add regex matching replacement
    - BUG/MINOR: lua: unsafe initialization
    - DOC: lua: fix somme errors
    - MINOR: lua: file dedicated to unsafe functions
    - MINOR: lua: add "now" time function
    - MINOR: standard: add RFC HTTP date parser
    - MINOR: lua: Add date functions
    - MINOR: lua: move common function
    - MINOR: lua: merge function
    - MINOR: lua: Add concat class
    - MINOR: standard: add function "escape_chunk"
    - MEDIUM: log: add a new log format flag "E"
    - DOC: add server name at rate-limit sessions example
    - BUG/MEDIUM: ssl: fix off-by-one in ALPN list allocation
    - BUG/MEDIUM: ssl: fix off-by-one in NPN list allocation
    - DOC: LUA: fix some typos and syntax errors
    - MINOR: cli: add a new "show env" command
    - MEDIUM: config: allow to manipulate environment variables in the global section
    - MEDIUM: cfgparse: reject incorrect 'timeout retry' keyword spelling in resolvers
    - MINOR: mailers: increase default timeout to 10 seconds
    - MINOR: mailers: use <CRLF> for all line endings
    - BUG/MAJOR: lua: segfault using Concat object
    - DOC: lua: copyrights
    - MINOR: common: mask conversion
    - MEDIUM: dns: extract options
    - MEDIUM: dns: add a "resolve-net" option which allow to prefer an ip in a network
    - MINOR: mailers: make it possible to configure the connection timeout
    - BUG/MAJOR: lua: applets can't sleep.
    - BUG/MINOR: server: some prototypes are renamed
    - BUG/MINOR: lua: Useless copy
    - BUG/MEDIUM: stats: stats bind-process doesn't propagate the process mask correctly
    - BUG/MINOR: server: fix the format of the warning on address change
    - CLEANUP: server: add "const" to some message strings
    - MINOR: server: generalize the "updater" source
    - BUG/MEDIUM: chunks: always reject negative-length chunks
    - BUG/MINOR: systemd: ensure we don't miss signals
    - BUG/MINOR: systemd: report the correct signal in debug message output
    - BUG/MINOR: systemd: propagate the correct signal to haproxy
    - MINOR: systemd: ensure a reload doesn't mask a stop
    - BUG/MEDIUM: cfgparse: wrong argument offset after parsing server "sni" keyword
    - CLEANUP: stats: Avoid computation with uninitialized bits.
    - CLEANUP: pattern: Ignore unknown samples in pat_match_ip().
    - CLEANUP: map: Avoid memory leak in out-of-memory condition.
    - BUG/MINOR: tcpcheck: fix incorrect list usage resulting in failure to load certain configs
    - BUG/MAJOR: samples: check smp->strm before using it
    - MINOR: sample: add a new helper to initialize the owner of a sample
    - MINOR: sample: always set a new sample's owner before evaluating it
    - BUG/MAJOR: vars: always retrieve the stream and session from the sample
    - CLEANUP: payload: remove useless and confusing nullity checks for channel buffer
    - BUG/MINOR: ssl: fix usage of the various sample fetch functions
    - MINOR: stats: create fields types suitable for all CSV output data
    - MINOR: stats: add all the "show info" fields in a table
    - MEDIUM: stats: fill all the show info elements prior to displaying them
    - MINOR: stats: add a function to emit fields into a chunk
    - MINOR: stats: add stats_dump_info_fields() to dump one field per line
    - MEDIUM: stats: make use of stats_dump_info_fields() for "show info"
    - MINOR: stats: add a declaration of all stats fields
    - MINOR: stats: don't hard-code the CSV fields list anymore
    - MINOR: stats: create stats fields storage and CSV dump function
    - MEDIUM: stats: convert stats_dump_fe_stats() to use stats_dump_fields_csv()
    - MEDIUM: stats: make stats_dump_fe_stats() use stats fields for HTML dump
    - MEDIUM: stats: convert stats_dump_li_stats() to use stats_dump_fields_csv()
    - MEDIUM: stats: make stats_dump_li_stats() use stats fields for HTML dump
    - MEDIUM: stats: convert stats_dump_be_stats() to use stats_dump_fields_csv()
    - MEDIUM: stats: make stats_dump_be_stats() use stats fields for HTML dump
    - MEDIUM: stats: convert stats_dump_sv_stats() to use stats_dump_fields_csv()
    - MEDIUM: stats: make stats_dump_sv_stats() use the stats field for HTML
    - MEDIUM: stats: move the server state coloring logic to the server dump function
    - MINOR: stats: do not use srv->admin & STATS_ADMF_MAINT in HTML dumps
    - MINOR: stats: do not check srv->state for SRV_ST_STOPPED in HTML dumps
    - MINOR: stats: make CSV report server check status only when enabled
    - MINOR: stats: only report backend's down time if it has servers
    - MINOR: stats: prepend '*' in front of the check status when in progress
    - MINOR: stats: make HTML stats dump rely on the table for the check status
    - MINOR: stats: add agent_status, agent_code, agent_duration to output
    - MINOR: stats: add check_desc and agent_desc to the output fields
    - MINOR: stats: add check and agent's health values in the output
    - MEDIUM: stats: make the HTML server state dump use the CSV states
    - MEDIUM: stats: only report observe errors when observe is set
    - MEDIUM: stats: expose the same flags for CLI and HTTP accesses
    - MEDIUM: stats: report server's address in the CSV output
    - MEDIUM: stats: report the cookie value in the server & backend CSV dumps
    - MEDIUM: stats: compute the color code only in the HTML form
    - MEDIUM: stats: report the listeners' address in the CSV output
    - MEDIUM: stats: make it possible to report the WAITING state for listeners
    - REORG: stats: dump the frontend's HTML stats via a generic function
    - REORG: stats: dump the socket stats via the generic function
    - REORG: stats: dump the server stats via the generic function
    - REORG: stats: dump the backend stats via the generic function
    - MEDIUM: stats: add a new "mode" column to report the proxy mode
    - MINOR: stats: report the load balancing algorithm in CSV output
    - MINOR: stats: add 3 fields to report the frontend-specific connection stats
    - MINOR: stats: report number of intercepted requests for frontend and backends
    - MINOR: stats: introduce stats_dump_one_line() to dump one stats line
    - CLEANUP: stats: make stats_dump_fields_html() not rely on proxy anymore
    - MINOR: stats: add ST_SHOWADMIN to pass the admin info in the regular flags
    - MINOR: stats: make stats_dump_fields_html() not use &trash by default
    - MINOR: stats: add functions to emit typed fields into a chunk
    - MEDIUM: stats: support "show info typed" on the CLI
    - MEDIUM: stats: implement a typed output format for stats
    - DOC: document the "show info typed" and "show stat typed" output formats
    - MINOR: cfgparse: warn when uid parameter is not a number
    - MINOR: cfgparse: warn when gid parameter is not a number
    - BUG/MINOR: standard: Avoid free of non-allocated pointer
    - BUG/MINOR: pattern: Avoid memory leak on out-of-memory condition
    - CLEANUP: http: fix a build warning introduced by a recent fix
    - BUG/MINOR: log: GMT offset not updated when entering/leaving DST
2016-03-14 00:10:05 +01:00
Benoit GARNIER
b413c2a759 BUG/MINOR: log: GMT offset not updated when entering/leaving DST
GMT offset used in local time formats was computed at startup, but was not updated when DST status changed while running.

For example these two RFC5424 syslog traces where emitted 5 seconds apart, just before and after DST changed:
  <14>1 2016-03-27T01:59:58+01:00 bunch-VirtualBox haproxy 2098 - - Connect ...
  <14>1 2016-03-27T03:00:03+01:00 bunch-VirtualBox haproxy 2098 - - Connect ...

It looked like they were emitted more than 1 hour apart, unlike with the fix:
  <14>1 2016-03-27T01:59:58+01:00 bunch-VirtualBox haproxy 3381 - - Connect ...
  <14>1 2016-03-27T03:00:03+02:00 bunch-VirtualBox haproxy 3381 - - Connect ...

This patch should be backported to 1.6 and partially to 1.5 (no fix needed in log.c).
2016-03-13 23:48:05 +01:00
Dragan Dosen
835b9212f6 MEDIUM: log: add a new log format flag "E"
The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It
mostly makes sense to use it in the RFC5424 structured-data log formats.

Example:

log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 13:36:47 +01:00
Christopher Faulet
92d3638d2d MAJOR: filters/http: Rewrite the HTTP compression as a filter
HTTP compression has been rewritten to use the filter API. This is more a PoC
than other thing for now. It allocates memory to work. So, if only for that, it
should be rewritten.

In the mean time, the implementation has been refactored to allow its use with
other filters. However, there are limitations that should be respected:

  - No filter placed after the compression one is allowed to change input data
    (in 'http_data' callback).
  - No filter placed before the compression one is allowed to change forwarded
    data (in 'http_forward_data' callback).

For now, these limitations are informal, so you should be careful when you use
several filters.

About the configuration, 'compression' keywords are still supported and must be
used to configure the HTTP compression behavior. In absence of a 'filter' line
for the compression filter, it is added in the filter chain when the first
compression' line is parsed. This is an easy way to do when you do not use other
filters. But another filter exists, an error is reported so that the user must
explicitly declare the filter.

For example:

  listen tst
      ...
      compression algo gzip
      compression offload
      ...
      filter flt_1
      filter compression
      filter flt_2
      ...
2016-02-09 14:53:15 +01:00