mirrors/haproxy
1
0
Fork 0
mirror of https://git.haproxy.org/git/haproxy.git/ synced 2025-08-12 18:16:58 +02:00
Code Issues Projects Releases Wiki Activity
8,448 Commits 18 Branches 394 Tags 184 MiB
e548974ca8
Commit Graph

76 Commits

Author SHA1 Message Date
Willy Tarreau
4a4e6bca60 BUG/MEDIUM: compression: fix the output type of the compressor name 
smp_fetch_res_comp_algo() returns the name of the compression algorithm
in use. The output type is set to SMP_T_STR instead of SMP_T_CSTR, which
causes any transformation to be operated without a cast. Fortunately,
the current converters do not overwrite a zero-sized area, so the result
is an empty string. Fix this to have SMP_T_CSTR instead so that the cast
is always performed using a copy before any transformation is done.
 2014-03-11 16:23:05 +01:00
Willy Tarreau
ef38c39287 MEDIUM: sample: systematically pass the keyword pointer to the keyword 
We're having a lot of duplicate code just because of minor variants between
fetch functions that could be dealt with if the functions had the pointer to
the original keyword, so let's pass it as the last argument. An earlier
version used to pass a pointer to the sample_fetch element, but this is not
the best solution for two reasons :
  - fetch functions will solely rely on the keyword string
  - some other smp_fetch_* users do not have the pointer to the original
    keyword and were forced to pass NULL.

So finally we're passing a pointer to the keyword as a const char *, which
perfectly fits the original purpose.
 2013-08-01 21:17:13 +02:00
Willy Tarreau
dc13c11c1e BUG/MEDIUM: prevent gcc from moving empty keywords lists into BSS 
Benoit Dolez reported a failure to start haproxy 1.5-dev19. The
process would immediately report an internal error with missing
fetches from some crap instead of ACL names.

The cause is that some versions of gcc seem to trim static structs
containing a variable array when moving them to BSS, and only keep
the fixed size, which is just a list head for all ACL and sample
fetch keywords. This was confirmed at least with gcc 3.4.6. And we
can't move these structs to const because they contain a list element
which is needed to link all of them together during the parsing.

The bug indeed appeared with 1.5-dev19 because it's the first one
to have some empty ACL keyword lists.

One solution is to impose -fno-zero-initialized-in-bss to everyone
but this is not really nice. Another solution consists in ensuring
the struct is never empty so that it does not move there. The easy
solution consists in having a non-null list head since it's not yet
initialized.

A new "ILH" list head type was thus created for this purpose : create
an Initialized List Head so that gcc cannot move the struct to BSS.
This fixes the issue for this version of gcc and does not create any
burden for the declarations.
 2013-06-21 23:29:02 +02:00
Willy Tarreau
6d4e4e8dd2 MEDIUM: acl: remove a lot of useless ACLs that are equivalent to their fetches 
The following 116 ACLs were removed because they're redundant with their
fetch function since last commit which allows the fetch function to be
used instead for types BOOL, INT and IP. Most places are now left with
an empty ACL keyword list that was not removed so that it's easier to
add other ACLs later.

always_false, always_true, avg_queue, be_conn, be_id, be_sess_rate, connslots,
nbsrv, queue, srv_conn, srv_id, srv_is_up, srv_sess_rate, res.comp, fe_conn,
fe_id, fe_sess_rate, dst_conn, so_id, wait_end, http_auth, http_first_req,
status, dst, dst_port, src, src_port, sc1_bytes_in_rate, sc1_bytes_out_rate,
sc1_clr_gpc0, sc1_conn_cnt, sc1_conn_cur, sc1_conn_rate, sc1_get_gpc0,
sc1_gpc0_rate, sc1_http_err_cnt, sc1_http_err_rate, sc1_http_req_cnt,
sc1_http_req_rate, sc1_inc_gpc0, sc1_kbytes_in, sc1_kbytes_out, sc1_sess_cnt,
sc1_sess_rate, sc1_tracked, sc1_trackers, sc2_bytes_in_rate,
sc2_bytes_out_rate, sc2_clr_gpc0, sc2_conn_cnt, sc2_conn_cur, sc2_conn_rate,
sc2_get_gpc0, sc2_gpc0_rate, sc2_http_err_cnt, sc2_http_err_rate,
sc2_http_req_cnt, sc2_http_req_rate, sc2_inc_gpc0, sc2_kbytes_in,
sc2_kbytes_out, sc2_sess_cnt, sc2_sess_rate, sc2_tracked, sc2_trackers,
sc3_bytes_in_rate, sc3_bytes_out_rate, sc3_clr_gpc0, sc3_conn_cnt,
sc3_conn_cur, sc3_conn_rate, sc3_get_gpc0, sc3_gpc0_rate, sc3_http_err_cnt,
sc3_http_err_rate, sc3_http_req_cnt, sc3_http_req_rate, sc3_inc_gpc0,
sc3_kbytes_in, sc3_kbytes_out, sc3_sess_cnt, sc3_sess_rate, sc3_tracked,
sc3_trackers, src_bytes_in_rate, src_bytes_out_rate, src_clr_gpc0,
src_conn_cnt, src_conn_cur, src_conn_rate, src_get_gpc0, src_gpc0_rate,
src_http_err_cnt, src_http_err_rate, src_http_req_cnt, src_http_req_rate,
src_inc_gpc0, src_kbytes_in, src_kbytes_out, src_sess_cnt, src_sess_rate,
src_updt_conn_cnt, table_avl, table_cnt, ssl_c_ca_err, ssl_c_ca_err_depth,
ssl_c_err, ssl_c_used, ssl_c_verify, ssl_c_version, ssl_f_version, ssl_fc,
ssl_fc_alg_keysize, ssl_fc_has_crt, ssl_fc_has_sni, ssl_fc_use_keysize,
 2013-06-11 21:22:58 +02:00
Willy Tarreau
c5599e7c49 BUG/MEDIUM: compression: the deflate algorithm must use global settings as well 
Global compression settings (windowsize and memlevel) were only considered
for the gzip algorithm but not the deflate algorithm. Since a single allocator
is used for both algos, if gzip was first initialized the memory with parameters
smaller than default, then initializing deflate after with default settings
would result in overusing the small allocated areas.

To fix this, we make use of deflateInit2() for deflate_init() as well.

Thanks to Godbach for reporting this bug, introduced by in 1.5-dev13 by commit
8b52bb38. No backport is needed.
 2013-04-28 09:01:11 +02:00
Willy Tarreau
7f6fa69221 BUG/MINOR: fix unterminated ACL array in compression 
Recent commit 727db8b4 was lacking a NULL ACL descriptor to terminate
the array, causing a random behaviour upon startup. No backport is needed.
 2013-04-23 19:39:43 +02:00
William Lallemand
727db8b4ea MINOR: compression: acl "res.comp" and fetch "res.comp_algo" 
Implements the "res.comp" ACL which is a boolean returning 1 when a
response has been compressed by HAProxy or 0 otherwise.

Implements the "res.comp_algo" fetch which contains the name of the
algorithm HAProxy used to compress the response.
 2013-04-20 23:53:33 +02:00
William Lallemand
00bf1dee9c BUG/MEDIUM: compression: does not forward trailers 
The commit bf3ae617 introduced a regression about the forward of the
trailers in compression mode.
 2012-11-23 11:12:33 +01:00
Willy Tarreau
55058a7c1e MINOR: stats: report HTTP compression stats per frontend and per backend 
It was a bit frustrating to have no idea about the bandwidth saved by
HTTP compression. Now we have per-frontend and per-backend stats. The
stats on the HTTP interface are shown in a hover title in the "bytes out"
column if at least something was fed to the compressor. 3 new columns
appeared in the CSV stats output.
 2012-11-22 01:07:40 +01:00
William Lallemand
072a2bf537 MINOR: compression: CPU usage limit 
New option 'maxcompcpuusage' in global section.
Sets the maximum CPU usage HAProxy can reach before stopping the
compression for new requests or decreasing the compression level of
current requests.  It works like 'maxcomprate' but with the Idle.
 2012-11-21 02:15:16 +01:00
William Lallemand
c71407657d BUG/MINOR: compression: dynamic level increase 
Using compression rate limit, the compression level wasn't taking care
of the max compression level during a session because the test was done
on the wrong variable.
 2012-11-21 02:15:16 +01:00
William Lallemand
e3a7d99062 MINOR: compression: report zlib memory usage 
Show the memory usage and the max memory available for zlib.
The value stored is now the memory used instead of the remaining
available memory.
 2012-11-21 02:15:16 +01:00
William Lallemand
8b52bb3878 MEDIUM: compression: use pool for comp_ctx 
Use pool for comp_ctx, it is allocated during the comp_algo->init().
The allocation of comp_ctx is accounted for in the zlib_memory_available.
 2012-11-21 01:56:47 +01:00
William Lallemand
bf3ae61789 MEDIUM: compression: don't compress when no data 
This patch makes changes in the http_response_forward_body state
machine. It checks if the compress algorithm had consumed data before
swapping the temporary and the input buffer. So it prevents null sized
zlib chunks.
 2012-11-19 14:57:29 +01:00
Willy Tarreau
4690985fca BUG: compression: do not always increment the round counter on allocation failure 
Zlib (at least 1.2 and 1.3) aborts when it fails to allocate the state, so we
must not count a round on this event. If the state succeeds, then it allocates
all the 4 remaining counters at once.
 2012-11-15 15:00:55 +01:00
Cyril Bonté
6162c43a0a BUILD: report zlib support in haproxy -vv 
Compression algorithms are not always supported depending on build options.
"haproxy -vv" now reports if zlib is supported and lists compression algorithms
also supported.
 2012-11-10 20:36:46 +01:00
Willy Tarreau
b1fbd050ec BUILD: compression: remove a build warning 
gcc emits this warning while building free_zlib() :
  src/compression.c: In function `free_zlib':
  src/compression.c:403: warning: 'pool' might be used uninitialized in this function

This is not a bug as the pool cannot take other values, but let's
pre-initialize is to null to fix the warning.
 2012-11-10 17:49:37 +01:00
William Lallemand
d85f917daf MINOR: compression: maximum compression rate limit 
This patch adds input and output rate calcutation on the HTTP compresion
feature.

Compression can be limited with a maximum rate value in kilobytes per
second. The rate is set with the global 'maxcomprate' option. You can
change this value dynamicaly with 'set rate-limit http-compression
global' on the UNIX socket.
 2012-11-10 17:47:27 +01:00
William Lallemand
9d5f5480fd MEDIUM: compression: limit RAM usage 
With the global maxzlibmem option, you are able ton control the maximum
amount of RAM usable for HTTP compression.

A test is done before each zlib allocation, if the there isn't available
memory, the test fail and so the zlib initialization, so data won't be
compressed.
 2012-11-08 15:23:30 +01:00
William Lallemand
2b50247695 MEDIUM: use pool for zlib 
Don't use the zlib allocator anymore, 5 pools are used for the zlib
compression. Their sizes depends of the window size and the memLevel in
deflateInit2.
 2012-11-08 15:23:29 +01:00
William Lallemand
a509e4c332 MINOR: compression: memlevel and windowsize 
The window size and the memlevel of the zlib are now configurable using
global options tune.zlib.memlevel and tune.zlib.windowsize.

It affects the memory consumption of the zlib.
 2012-11-08 15:23:29 +01:00
William Lallemand
08289f12f9 BUILD: remove dependency to zlib.h 
The build was dependent of the zlib.h header, regardless of the USE_ZLIB
option. The fix consists of several #ifdef in the source code.

It removes the overhead of the zstream structure in the session when you
don't use the option.
 2012-11-05 10:23:16 +01:00
William Lallemand
1c2d622d82 CLEANUP: use struct comp_ctx instead of union 
Replace union comp_ctx by struct comp_ctx.

Use struct comp_ctx * in the init/add_data/flush/reset/end prototypes of
compression.h functions.
 2012-11-05 10:23:16 +01:00
Willy Tarreau
3476364ce9 BUILD: fix coexistence of openssl and zlib 
The crappy zlib and openssl libs both define a free_func as a different typedef.
That's a very clever idea to use such a generic name in general purpose libraries,
really... The zlib one is easier to redefine than openssl's, so let's only fix this
one.
 2012-10-26 15:07:59 +02:00
Willy Tarreau
7e488d781c MINOR: compression: optimize memLevel to improve byte rate 
Decreasing the deflateInit2's memLevel parameter from 9 to 8 does not
affect the compression ratio and increases the compression speed by 12%.
Lower values do not increase transfer speed but decrease the compression
ratio so it looks like 8 is optimal.
 2012-10-26 11:36:40 +02:00
William Lallemand
82fe75c1a7 MEDIUM: HTTP compression (zlib library support) 
This commit introduces HTTP compression using the zlib library.

http_response_forward_body has been modified to call the compression
functions.

This feature includes 3 algorithms: identity, gzip and deflate:

  * identity: this is mostly for debugging, and it was useful for
  developping the compression feature. With Content-Length in input, it
  is making each chunk with the data available in the current buffer.
  With chunks in input, it is rechunking, the output chunks will be
  bigger or smaller depending of the size of the input chunk and the
  size of the buffer. Identity does not apply any change on data.

  * gzip: same as identity, but applying a gzip compression. The data
  are deflated using the Z_NO_FLUSH flag in zlib. When there is no more
  data in the input buffer, it flushes the data in the output buffer
  (Z_SYNC_FLUSH). At the end of data, when it receives the last chunk in
  input, or when there is no more data to read, it writes the end of
  data with Z_FINISH and the ending chunk.

  * deflate: same as gzip, but with deflate algorithm and zlib format.
  Note that this algorithm has ambiguous support on many browsers and
  no support at all from recent ones. It is strongly recommended not
  to use it for anything else than experimentation.

You can't choose the compression ratio at the moment, it will be set to
Z_BEST_SPEED (1), as tests have shown very little benefit in terms of
compression ration when going above for HTML contents, at the cost of
a massive CPU impact.

Compression will be activated depending of the Accept-Encoding request
header. With identity, it does not take care of that header.

To build HAProxy with zlib support, use USE_ZLIB=1 in the make
parameters.

This work was initially started by David Du Colombier at Exceliance.
 2012-10-26 02:30:48 +02:00