Commit Graph

758 Commits

Author SHA1 Message Date
Willy Tarreau
71241abfd3 MINOR: http: move redirect rule processing to its own function
We now have http_apply_redirect_rule() which does all the redirect-specific
job instead of having this inside http_process_req_common().

Also one of the benefit gained from uniformizing this code is that both
keep-alive and close response do emit the PR-- flags. The fix for the
flags could probably be backported to 1.4 though it's very minor.

The previous function http_perform_redirect() was becoming confusing
so it was renamed http_perform_server_redirect() since it only applies
to server-based redirection.
2012-12-28 14:47:19 +01:00
Willy Tarreau
96257ec5c8 CLEANUP: http: rename the misleading http_check_access_rule
Several bugs were introduced recently due to a misunderstanding of how
this function works and what it was supposed to do. Since it's supposed
to only return the pointer to a rule which aborts further processing of
the request, let's rename it to avoid further issues.

The function was also slightly cleaned up without any functional change.
2012-12-28 14:47:19 +01:00
Willy Tarreau
d79a3b248e BUG/MINOR: log: make log-format, unique-id-format and add-header more independant
It happens that all of them call parse_logformat_line() which sets
proxy->to_log with a number of flags affecting the line format for
all three users. For example, having a unique-id specified disables
the default log-format since fe->to_log is tested when the session
is established.

Similarly, having "option logasap" will cause "+" to be inserted in
unique-id or headers referencing some of the fields depending on
LW_BYTES.

This patch first removes most of the dependency on fe->to_log whenever
possible. The first possible cleanup is to stop checking fe->to_log
for being null, considering that it always contains at least LW_INIT
when any such usage is made of the log-format!

Also, some checks are wrong. s->logs.logwait cannot be nulled by
"logwait &= ~LW_*" since LW_INIT is always there. This results in
getting the wrong log at the end of a request or session when a
unique-id or add-header is set, because logwait is still not null
but the log-format is not checked.

Further cleanups are required. Most LW_* flags should be removed or at
least replaced with what they really mean (eg: depend on client-side
connection, depend on server-side connection, etc...) and this should
only affect logging, not other mechanisms.

This patch fixes the default log-format and tries to limit interferences
between the log formats, but does not pretend to do more for the moment,
since it's the most visible breakage.
2012-12-28 09:51:00 +01:00
Willy Tarreau
cbc743e36c BUG/MEDIUM: stats: disable request analyser when processing POST or HEAD
After the response headers are sent and the request processing is done,
the buffers are wiped out and the stream interface is closed. We must
then disable the request analysers, otherwise some processing will
happen on a closed stream interface and empty buffers which do not
match, causing all sort of crashes. This issue was introduced with
recent work on the stats, and was reported by Seri.
2012-12-28 08:36:50 +01:00
Willy Tarreau
1a1e8072f9 BUG/MINOR: stats: http-request rules still don't cope with stats
Since commit 20b0de5, we also had another remaining issue : an
"http-request allow" rule would prevent a stats rule from being
processed.
2012-12-27 10:34:21 +01:00
Willy Tarreau
8b80f0c9a2 BUG/MINOR: stats: last fix was still wrong
Previous commit was still wrong, it broke add-header and set-header
because we don't want to leave on these actions.

The http_check_access_rule() function should be redesigned, it was
initially thought for allow/deny rules but now it is executing other
non-final rules and at the same time returning a pointer to the last
final rule. That becomes a bit confusing and will need to be addressed
before we implement redirect and return.
2012-12-25 21:55:37 +01:00
Willy Tarreau
418c1a0a95 BUG/MEDIUM: stats: fix stats page regression introduced by commit 20b0de5
This commit adding http-request add-header/set-header unfortunately introduced
a regression to the handling of the stats page which is not matched anymore.

Thanks to Dmitry Sivachenko for reporting this.
2012-12-25 20:52:58 +01:00
Willy Tarreau
20b0de56d4 MEDIUM: http: add http-request 'add-header' and 'set-header' to build headers
These two new statements allow to pass information extracted from the request
to the server. It's particularly useful for passing SSL information to the
server, but may be used for various other purposes such as combining headers
together to emulate internal variables.
2012-12-24 15:56:20 +01:00
Willy Tarreau
5c2e198390 MINOR: http: prepare to support more http-request actions
We'll need to support per-action arguments, so we need to have an
"arg" union in http_req_rule.
2012-12-24 12:26:26 +01:00
Willy Tarreau
354898bba9 MINOR: stats: replace STAT_FMT_CSV with STAT_FMT_HTML
We need to switch the default mode if we want to add new output formats
later. Let CSV be the default and HTML be an option.
2012-12-23 21:46:30 +01:00
Willy Tarreau
47ca54505c MINOR: chunks: centralize the trash chunk allocation
At the moment, we need trash chunks almost everywhere and the only
correctly implemented one is in the sample code. Let's move this to
the chunks so that all other places can use this allocator.

Additionally, the get_trash_chunk() function now really returns two
different chunks. Previously it used to always overwrite the same
chunk and point it to a different buffer, which was a bit tricky
because it's not obvious that two consecutive results do alias each
other.
2012-12-23 21:46:07 +01:00
Willy Tarreau
1facd6d67e REORG: stats: move the HTTP header injection to proto_http
The HTTP header injection that are performed in dumpstats when responding
or when redirecting a POST request have nothing to do in dumpstats. They
do not use any state from the stats, and are 100% HTTP. Let's make the
headers there in the HTTP core, and have dumpstats only produce stats.
2012-12-22 22:50:01 +01:00
Willy Tarreau
d9bdcd5139 REORG: stats: massive code reorg and cleanup
The dumpstats code looks like a spaghetti plate. Several functions are
supposed to be able to do several things but rely on complex states to
dispatch the work to independant functions. Most of the HTML output is
performed within the switch/case statements of the whole state machine.

Let's clean this up by adding new functions to emit the data and have
a few more iterators to avoid relying on so complex states.

The new stats dump sequence looks like this for CLI and for HTTP :

  cli_io_handler()
      -> stats_dump_sess_to_buffer()      // "show sess"
      -> stats_dump_errors_to_buffer()    // "show errors"
      -> stats_dump_raw_info_to_buffer()  // "show info"
         -> stats_dump_raw_info()
      -> stats_dump_raw_stat_to_buffer()  // "show stat"
         -> stats_dump_csv_header()
         -> stats_dump_proxy()
            -> stats_dump_px_hdr()
            -> stats_dump_fe_stats()
            -> stats_dump_li_stats()
            -> stats_dump_sv_stats()
            -> stats_dump_be_stats()
            -> stats_dump_px_end()

  http_stats_io_handler()
      -> stats_http_redir()
      -> stats_dump_http()              // also emits the HTTP headers
         -> stats_dump_html_head()      // emits the HTML headers
         -> stats_dump_csv_header()     // emits the CSV headers (same as above)
         -> stats_dump_http_info()      // note: ignores non-HTML output
         -> stats_dump_proxy()          // same as above
         -> stats_dump_http_end()       // emits HTML trailer
2012-12-22 20:45:02 +01:00
Willy Tarreau
40f151aa79 BUG/MINOR: http: don't abort client connection on premature responses
When a server responds prematurely to a POST request, haproxy used to
cause the transfer to be aborted before the end. This is problematic
because this causes the client to receive a TCP reset when it tries to
push more data, generally preventing it from receiving the response
which contain the reason for the premature reponse (eg: "entity too
large" or an authentication request).

From now on we take care of allowing the upload traffic to flow to the
server even when the response has been received, since the server is
supposed to drain it. That way the client receives the server response.

This bug has been present since 1.4 and the fix should probably be
backported there.
2012-12-20 12:10:09 +01:00
Willy Tarreau
f26b252ee4 MINOR: http: make resp_ver and status ACLs check for the presence of a response
The two ACL fetches "resp_ver" and "status", if used in a request despite
the warning, would return a match of zero length. This is inappropriate,
better return a non-match to be more consistent with other ACL processing.
2012-12-14 08:35:45 +01:00
Willy Tarreau
4a55060aa6 MINOR: http: add the "base32+src" fetch method.
This returns the concatenation of the base32 fetch and the src fetch.
The resulting type is of type binary, with a size of 8 or 20 bytes
depending on the source address family. This can be used to track
per-IP, per-URL counters.
2012-12-09 14:53:32 +01:00
Willy Tarreau
ab1f7b72fb MINOR: http: add the "base32" pattern fetch function
This returns a 32-bit hash of the value returned by the "base"
fetch method above. This is useful to track per-URL activity on
high traffic sites without having to store all URLs. Instead a
shorter hash is stored, saving a lot of memory. The output type
is an unsigned integer.
2012-12-09 14:08:48 +01:00
Willy Tarreau
5d5b5d8eaf MEDIUM: proto_tcp: add support for tracking L7 information
Until now it was only possible to use track-sc1/sc2 with "src" which
is the IPv4 source address. Now we can use track-sc1/sc2 with any fetch
as well as any transformation type. It works just like the "stick"
directive.

Samples are automatically converted to the correct types for the table.

Only "tcp-request content" rules may use L7 information, and such information
must already be present when the tracking is set up. For example it becomes
possible to track the IP address passed in the X-Forwarded-For header.

HTTP request processing now also considers tracking from backend rules
because we want to be able to update the counters even when the request
was already parsed and tracked.

Some more controls need to be performed (eg: samples do not distinguish
between L4 and L6).
2012-12-09 14:08:47 +01:00
Willy Tarreau
dc979f2492 BUG/MINOR: http: don't log a 503 on client errors while waiting for requests
If a client aborts a request with an error (typically a TCP reset), we must
log a 400. Till now we did not set the status nor close the stream interface,
causing the request to attempt to be forwarded and logging a 503.

Should be backported to 1.4 which is affected as well.
2012-12-04 10:52:22 +01:00
Willy Tarreau
14cba4b0b1 MEDIUM: connection: add an error code in connections
This will be needed to improve error reporting, especially for SSL.
2012-12-03 14:22:13 +01:00
Willy Tarreau
8139b9959f MINOR: compression: make the stats a bit more robust
To ensure that we only count when a response was compressed, we also
check for the SN_COMP_READY flag which indicates that the compression
was effectively initialized. Comp_algo alone is meaningless.
2012-11-27 09:34:00 +01:00
Willy Tarreau
9101535038 BUG/MINOR: http: disable compression when message has no body
Compression was not disabled on 1xx, 204, 304 nor HEAD requests. This
is not really a problem, but it reports more compressed responses than
really done.
2012-11-27 09:34:00 +01:00
Willy Tarreau
0a80a8dbb2 MINOR: http: factor out the content-type checks
Let's only look up the content-type header once. This involves
inverting the condition which is not dramatic.

Also, we now always check the value length before comparing it, and we
always reset the ctx.idx before looking a header up. Otherwise that
could make header lookups depend on their on-wire order. It would be
a minor issue however since at worst it would cause some responses not
to be compressed.
2012-11-26 16:36:00 +01:00
William Lallemand
d300261bab MINOR: compression: disable on multipart or status != 200
The compression is disabled when the HTTP status code is not 200, indeed
compression on some HTTP code can create issues (ex: 206, 416).

Multipart message should not be compressed eitherway.
2012-11-26 16:02:58 +01:00
William Lallemand
859550e068 BUG/MINOR: compression: Content-Type is case insensitive
The Content-Type parameter must be case insensitive.
2012-11-26 16:02:58 +01:00
Willy Tarreau
f003d375ec BUG/MINOR: http: don't report client aborts as server errors
If a client aborts with an abortonclose flag, the close is forwarded
to the server and when server response is processed, the analyser thinks
it's the server who has closed first, and logs flags "SD" or "SH" and
counts a server error. In order to avoid this, we now first detect that
the client has closed and log a client abort instead.

This likely is the reason why many people have been observing a small rate
of SD/SH flags without being able to find what the error was.

This fix should probably be backported to 1.4.
2012-11-26 13:50:02 +01:00
Willy Tarreau
5e16cbc3bd MINOR: stats: report the total number of compressed responses per front/back
Depending on the content-types and accept-encoding fields, some responses
might or might not be compressed. Let's have a counter of the number of
compressed responses and report it in the stats to help improve compression
usage.

Some cosmetic issues were fixed in the CSV output too (missing commas at the
end).
2012-11-24 14:54:13 +01:00
William Lallemand
00bf1dee9c BUG/MEDIUM: compression: does not forward trailers
The commit bf3ae617 introduced a regression about the forward of the
trailers in compression mode.
2012-11-23 11:12:33 +01:00
Willy Tarreau
193b8c6168 MINOR: http: allow the cookie capture size to be changed
Some users need more than 64 characters to log large cookies. The limit
was set to 63 characters (and not 64 as previously documented). Now it
is possible to change this using the global "tune.http.cookielen" setting
if required.
2012-11-22 00:44:27 +01:00
William Lallemand
072a2bf537 MINOR: compression: CPU usage limit
New option 'maxcompcpuusage' in global section.
Sets the maximum CPU usage HAProxy can reach before stopping the
compression for new requests or decreasing the compression level of
current requests.  It works like 'maxcomprate' but with the Idle.
2012-11-21 02:15:16 +01:00
William Lallemand
8b52bb3878 MEDIUM: compression: use pool for comp_ctx
Use pool for comp_ctx, it is allocated during the comp_algo->init().
The allocation of comp_ctx is accounted for in the zlib_memory_available.
2012-11-21 01:56:47 +01:00
William Lallemand
bf3ae61789 MEDIUM: compression: don't compress when no data
This patch makes changes in the http_response_forward_body state
machine. It checks if the compress algorithm had consumed data before
swapping the temporary and the input buffer. So it prevents null sized
zlib chunks.
2012-11-19 14:57:29 +01:00
Willy Tarreau
b97b6190e1 BUG: compression: properly disable compression when content-type does not match
Disabling compression based on the content-type was improperly done since the
introduction of the COMP_READY flag, sometimes resulting in truncated responses.
2012-11-19 14:55:02 +01:00
Willy Tarreau
543db62e1f BUG/MEDIUM: compression: release the zlib pools between keep-alive requests
There was a possible memory leak in the zlib code when the first response of
a keep-alive session was compressed, because the next request would reset the
compression algo, preventing a later call to session_free() from releasing it.
The reason is that it is necessary to release the assigned resources in
http_end_txn_clean_session().
2012-11-15 16:41:22 +01:00
William Lallemand
ec3e3890f0 BUG/MINOR: compression: deinit zlib only when required
The zlib stream was deinitialized even when the init failed.
2012-11-15 15:42:17 +01:00
William Lallemand
c04ca58222 BUG/MEDIUM: compression: no Content-Type header but type in configuration
HAProxy was compressing data when there was no Content-Type header in
the response but a compression type specified in the configuration.
2012-11-15 15:42:11 +01:00
Willy Tarreau
3fdb366885 MAJOR: connection: replace struct target with a pointer to an enum
Instead of storing a couple of (int, ptr) in the struct connection
and the struct session, we use a different method : we only store a
pointer to an integer which is stored inside the target object and
which contains a unique type identifier. That way, the pointer allows
us to retrieve the object type (by dereferencing it) and the object's
address (by computing the displacement in the target structure). The
NULL pointer always corresponds to OBJ_TYPE_NONE.

This reduces the size of the connection and session structs. It also
simplifies target assignment and compare.

In order to improve the generated code, we try to put the obj_type
element at the beginning of all the structs (listener, server, proxy,
si_applet), so that the original and target pointers are always equal.

A lot of code was touched by massive replaces, but the changes are not
that important.
2012-11-12 00:42:33 +01:00
Willy Tarreau
50fc7777c6 MEDIUM: http: refrain from sending "Connection: close" when Upgrade is present
Some servers are not totally HTTP-compliant when it comes to parsing the
Connection header. This is particularly true with WebSocket where it happens
from time to time that a server doesn't support having a "close" token along
with the "Upgrade" token in the Connection header. This broken behaviour has
also been noticed on some clients though the problem is less frequent on the
response path.

Sometimes the workaround consists in enabling "option http-pretend-keepalive"
to leave the request Connection header untouched, but this is not always the
most convenient solution. This patch introduces a new solution : haproxy now
also looks for the "Upgrade" token in the Connection header and if it finds
it, then it refrains from adding any other token to the Connection header
(though "keep-alive" and "close" may still be removed if found). The same is
done for the response headers.

This way, WebSocket much with less changes even when facing non-compliant
clients or servers. At least it fixes the DISCONNECT issue that was seen
on the websocket.org test.

Note that haproxy does not change its internal mode, it just refrains from
adding new tokens to the connection header.
2012-11-11 22:40:00 +01:00
Willy Tarreau
7f7ad91056 BUILD: stream_interface: remove si_fd() and its references
si_fd() is not used a lot, and breaks builds on OpenBSD 5.2 which
defines this name for its own purpose. It's easy enough to remove
this one-liner function, so let's do it.
2012-11-11 20:53:29 +01:00
William Lallemand
d85f917daf MINOR: compression: maximum compression rate limit
This patch adds input and output rate calcutation on the HTTP compresion
feature.

Compression can be limited with a maximum rate value in kilobytes per
second. The rate is set with the global 'maxcomprate' option. You can
change this value dynamicaly with 'set rate-limit http-compression
global' on the UNIX socket.
2012-11-10 17:47:27 +01:00
William Lallemand
f3747837e5 MINOR: compression: tune.comp.maxlevel
This option allows you to set the maximum compression level usable by
the compression algorithm. It affects CPU usage.
2012-11-10 17:47:07 +01:00
Finn Arne Gangstad
0a410e81fb BUG: http: revert broken optimisation from 82fe75c1a7
This optimisation causes haproxy to time out requests that result
in two TCP packets, one packet containing the header, and one
packet containing the actual data. This is a very typical type
of response from a lot of servers.

[Willy: I suspect the fix might have an impact on the compression code
 which I'm not sure completely handles calls with 0 bytes to forward]
2012-11-10 17:38:36 +01:00
William Lallemand
4c49fae985 MINOR: compression: init before deleting headers
Init the compression algorithm before modifying the response headers. So
if the compression init fail, the headers won't be modified.
2012-11-08 15:23:30 +01:00
William Lallemand
1c2d622d82 CLEANUP: use struct comp_ctx instead of union
Replace union comp_ctx by struct comp_ctx.

Use struct comp_ctx * in the init/add_data/flush/reset/end prototypes of
compression.h functions.
2012-11-05 10:23:16 +01:00
Finn Arne Gangstad
cbb9a4b128 MINOR: compression: Enable compression for IE6 w/SP2, IE7 and IE8
Some old browsers that have a user-agent starting with "Mozilla/4" do
not support compressison correctly, so disable compression for those.

Internet explorer 6 after Windows XP service pack 2, IE 7, and IE 8,
do however support compression and still have a user agent starting
with Mozilla/4, so we try to enable compression for those.

MSIE has a user-agent on this form:
Mozilla/4.0 (compatible; MSIE <version>; ...)

98% of MSIE 6 SP2 user agents start with
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1
The remaining 2% have additional flags before "SV1".

This simplified matching looking for MSIE at exactly position 25
and SV1 at exacly position 51 gives a few false negatives, so sometimes
a compression opportunity is lost.

A test against 3 hours of traffic to around 3000 news sites worldwide
gives less than 0.007% (70ppm) missed compression opportunities.
2012-10-29 22:03:14 +01:00
Willy Tarreau
7e2c647ee7 MEDIUM: remove remains of BUFSIZE in HTTP auth and sample conversions
Sample conversions rely on two alternative buffers which were previously
allocated as static bufs of size BUFSIZE. Now they're initialized to the
global buffer size. It was the same for HTTP authentication. Note that it
seems that none of them was prone to any mistake when dealing with the
buffer size, but better stay on the safe side by maintaining the old
assumption that a trash buffer is always "large enough".
2012-10-29 20:44:36 +01:00
Willy Tarreau
19d14ef104 MEDIUM: make the trash be a chunk instead of a char *
The trash is used everywhere to store the results of temporary strings
built out of s(n)printf, or as a storage for a chunk when chunks are
needed.

Using global.tune.bufsize is not the most convenient thing either.

So let's replace trash with a chunk and directly use it as such. We can
then use trash.size as the natural way to get its size, and get rid of
many intermediary chunks that were previously used.

The patch is huge because it touches many areas but it makes the code
a lot more clear and even outlines places where trash was used without
being that obvious.
2012-10-29 16:57:30 +01:00
Willy Tarreau
08b4d79d31 BUG: compression: disable auto-close and enable MSG_MORE during transfer
We don't want the lower layer to forward a close while we're compressing,
and we want the system to fuse outgoing TCP segments using MSG_MORE as
much as possible to save round trips that can emerge from sending short
packets with a PUSH flag.

A test on a remote busy DSL line consisting in compressing a 100MB file
on the fly full of zeroes only showed a transfer rate of a few kB/s due
to these round trips.
2012-10-27 01:36:34 +02:00
Willy Tarreau
70737d142f MINOR: compression: add an offload option to remove the Accept-Encoding header
This is used when it is desired that backend servers don't compress
(eg: because of buggy implementations).
2012-10-27 01:13:24 +02:00
Willy Tarreau
f2943dccd0 MAJOR: session: detach the connections from the stream interfaces
We will need to be able to switch server connections on a session and
to keep idle connections. In order to achieve this, the preliminary
requirement is that the connections can survive the session and be
detached from them.

Right now they're still allocated at exactly the same place, so when
there is a session, there are always 2 connections. We could soon
improve on this by allocating the outgoing connection only during a
connect().

This current patch touches a lot of code and intentionally does not
change any functionnality. Performance tests show no regression (even
a very minor improvement). The doc has not yet been updated.
2012-10-26 20:15:20 +02:00
Willy Tarreau
c919dc66a3 CLEANUP: remove trashlen
trashlen is a copy of global.tune.bufsize, so let's stop using it as
a duplicate, fall back to the original bufsize, it's less confusing
this way.
2012-10-26 20:04:27 +02:00
Willy Tarreau
3c7b97b9f9 BUG/MINOR: http: compression should consider all Accept-Encoding header values
Right now commit 82fe75c1 came with a minor bug limiting the check to the first
accept-encoding header value only.
2012-10-26 14:52:02 +02:00
Willy Tarreau
05d846092f MINOR: compression: automatically disable compression for older browsers
A number of older browsers have many issues with compressed contents. It
happens that all these older browsers announce themselves as "Mozilla/4"
and that despite not being all broken, the amount of working browsers
announcing themselves this way compared to all other ones is so tiny
that it's not worth wasting cycles trying to adapt to every specific
one.

So let's simply disable compression for these older browsers.

More information on this very detailed article :

   http://zoompf.com/2012/02/lose-the-wait-http-compression
2012-10-26 02:54:31 +02:00
William Lallemand
82fe75c1a7 MEDIUM: HTTP compression (zlib library support)
This commit introduces HTTP compression using the zlib library.

http_response_forward_body has been modified to call the compression
functions.

This feature includes 3 algorithms: identity, gzip and deflate:

  * identity: this is mostly for debugging, and it was useful for
  developping the compression feature. With Content-Length in input, it
  is making each chunk with the data available in the current buffer.
  With chunks in input, it is rechunking, the output chunks will be
  bigger or smaller depending of the size of the input chunk and the
  size of the buffer. Identity does not apply any change on data.

  * gzip: same as identity, but applying a gzip compression. The data
  are deflated using the Z_NO_FLUSH flag in zlib. When there is no more
  data in the input buffer, it flushes the data in the output buffer
  (Z_SYNC_FLUSH). At the end of data, when it receives the last chunk in
  input, or when there is no more data to read, it writes the end of
  data with Z_FINISH and the ending chunk.

  * deflate: same as gzip, but with deflate algorithm and zlib format.
  Note that this algorithm has ambiguous support on many browsers and
  no support at all from recent ones. It is strongly recommended not
  to use it for anything else than experimentation.

You can't choose the compression ratio at the moment, it will be set to
Z_BEST_SPEED (1), as tests have shown very little benefit in terms of
compression ration when going above for HTML contents, at the cost of
a massive CPU impact.

Compression will be activated depending of the Accept-Encoding request
header. With identity, it does not take care of that header.

To build HAProxy with zlib support, use USE_ZLIB=1 in the make
parameters.

This work was initially started by David Du Colombier at Exceliance.
2012-10-26 02:30:48 +02:00
Willy Tarreau
54d23dfc07 CLEANUP: http: rename HTTP_MSG_DATA_CRLF state
This state's name is confusing as it is only used with chunked encoding
and makes newcomers think it's also related to the content-length. Let's
call it CHUNK_CRLF to clear any doubt on this.
2012-10-26 01:13:52 +02:00
Willy Tarreau
24e6d972aa OPTIM: http: inline http_parse_chunk_size() and http_skip_chunk_crlf()
These functions are not that long and the compiler inlines them well. Doing
so has sped up the chunked encoding parser by 41% !

Note that http_forward_trailers was also declared static because it's not
exported.
2012-10-26 01:12:40 +02:00
Cyril Bonté
69fa99292e MEDIUM: http: accept IPv6 values with (s)hdr_ip acl
Commit ceb4ac9c states that IPv6 values are accepted by "hdr_ip" acl,
but the code didn't allow it. This patch provides the ability to accept IPv6
values.
2012-10-25 14:41:33 +02:00
Willy Tarreau
fc47f91c9c BUG/MEDIUM: http: set DONTWAIT on data when switching to tunnel mode
Jaroslaw Bojar diagnosed an issue when haproxy switches to tunnel mode
after a transfer. The response data are sent with the MSG_MORE flag,
causing them to be needlessly queued in the kernel. In order to fix this,
we set the CF_NEVER_WAIT flag on the channels when switching to tunnel
mode.

One issue remained with client-side keep-alive : if the response is sent
before the end of the request, it suffers the same issue for the same
reason. This is easily addressed by setting the CF_SEND_DONTWAIT flag
on the channel when the response has been parsed and we're waiting for
the other side.

The same issue is present in 1.4 so the fix must be backported.
2012-10-20 10:41:37 +02:00
Willy Tarreau
9b28e03b66 MAJOR: channel: replace the struct buffer with a pointer to a buffer
With this commit, we now separate the channel from the buffer. This will
allow us to replace buffers on the fly without touching the channel. Since
nobody is supposed to keep a reference to a buffer anymore, doing so is not
a problem and will also permit some copy-less data manipulation.

Interestingly, these changes have shown a 2% performance increase on some
workloads, probably due to a better cache placement of data.
2012-10-13 09:07:52 +02:00
Willy Tarreau
cdbdd52a38 CLEANUP: http: use 'chn' to name channel variables, not 'buf'
These "buf" were confusing as they were really refering to channels. At
most places, a buffer was really all what was needed, so a struct buffer
was used instead. It is possible that the performance has slightly increased
by the removal of pointer offset in many pointer operations by directly
using the buffer pointer instead of the channel pointer.
2012-10-12 23:02:51 +02:00
Willy Tarreau
394db379eb REORG: http: rename msg->buf to msg->chn since it's a channel
It's extremely confusing to have all those msg->buf->buf everywhere after
the extraction of the buffer from the channel. Let's clean this up.
2012-10-12 22:40:39 +02:00
Willy Tarreau
1b6c00cb99 BUG/MAJOR: ensure that hdr_idx is always reserved when L7 fetches are used
Baptiste Assmann reported a bug causing a crash on recent versions when
sticking rules were set on layer 7 in a TCP proxy. The bug is easier to
reproduce with the "defer-accept" option on the "bind" line in order to
have some contents to parse when the connection is accepted. The issue
is that the acl_prefetch_http() function called from HTTP fetches relies
on hdr_idx to be preinitialized, which is not the case if there is no L7
ACL.

The solution consists in adding a new SMP_CAP_L7 flag to fetches to indicate
that they are expected to work on L7 data, so that the proxy knows that the
hdr_idx has to be initialized. This is already how ACL and HTTP mode are
handled.

The bug was present since 1.5-dev9.
2012-10-05 22:46:09 +02:00
bedis
4c75cca8ba MINOR: samples: update the url_param fetch to match parameters in the path
It now supports an optional delimiter to allow to look for the parameter before
the query string.
2012-10-05 15:17:23 +02:00
Willy Tarreau
f7bc57ca6e REORG: connection: rename the data layer the "transport layer"
While working on the changes required to make the health checks use the
new connections, it started to become obvious that some naming was not
logical at all in the connections. Specifically, it is not logical to
call the "data layer" the layer which is in charge for all the handshake
and which does not yet provide a data layer once established until a
session has allocated all the required buffers.

In fact, it's more a transport layer, which makes much more sense. The
transport layer offers a medium on which data can transit, and it offers
the functions to move these data when the upper layer requests this. And
it is the upper layer which iterates over the transport layer's functions
to move data which should be called the data layer.

The use case where it's obvious is with embryonic sessions : an incoming
SSL connection is accepted. Only the connection is allocated, not the
buffers nor stream interface, etc... The connection handles the SSL
handshake by itself. Once this handshake is complete, we can't use the
data functions because the buffers and stream interface are not there
yet. Hence we have to first call a specific function to complete the
session initialization, after which we'll be able to use the data
functions. This clearly proves that SSL here is only a transport layer
and that the stream interface constitutes the data layer.

A similar change will be performed to rename app_cb => data, but the
two could not be in the same commit for obvious reasons.
2012-10-04 22:26:09 +02:00
Willy Tarreau
b8ffd378f0 BUG/MAJOR: http: chunk parser was broken with buffer changes
Since at least commit a458b679, msg->sov could become negative in
http_parse_chunk_size() if a chunk size wrapped around the buffer.
The effect is that at some point channel_forward() was called with
a negative size, causing all data to be transferred without being
analyzed anymore.

Since haproxy does not support keep-alive with the server yet, this
issue is not really noticeable, as the server closes the connection
in response. Still, when tunnel mode is used or when pretent-keepalive
is used, it is possible to see the problem.

This issue was reported and diagnosed by William Lallemand at
Exceliance.
2012-09-27 15:08:56 +02:00
Willy Tarreau
e92693af26 BUG: http: do not print garbage on invalid requests in debug mode
Cyril Bonté reported a mangled debug output when an invalid request
was sent with a faulty request line. The reason was the use of the
msg->sl.rq.l offset which was not yet initialized in this case. So
we change the way to report such an error so that first we initialize
it to zero before parsing a message, then we use that to know whether
we can trust it or not. If it's still zero, then we display the whole
buffer, truncated by debug_hdr() to the first CR or LF character, which
results in the first line only.

The same operation was performed for the response, which was wrong too.
2012-09-24 21:16:42 +02:00
Willy Tarreau
eb6cead1de MINOR: standard: make memprintf() support a NULL destination
Doing so removes many checks that were systematically made because
the callees don't know if the caller passed a valid pointer.
2012-09-24 10:53:16 +02:00
Willy Tarreau
2e1dca8f52 MEDIUM: http: add "redirect scheme" to ease HTTP to HTTPS redirection
For instance :

   redirect scheme https if !{ is_ssl }
2012-09-12 08:43:15 +02:00
Willy Tarreau
783f25800c BUILD: http: rename error_message http_error_message to fix conflicts on RHEL
Duncan Hall reported a build issue on CentOS where error_message conflicts
with another system declaration when SSL is enabled. Rename the function.
2012-09-04 12:19:04 +02:00
Willy Tarreau
74172ff9c3 CLEANUP: frontend: remove the old proxy protocol decoder
This one used to rely on a stream analyser which was inappropriate.
It's not used anymore.
2012-09-03 20:47:35 +02:00
Willy Tarreau
986a9d2d12 MAJOR: connection: move the addr field from the stream_interface
We need to have the source and destination addresses in the connection.
They were lying in the stream interface so let's move them. The flags
SI_FL_FROM_SET and SI_FL_TO_SET have been moved as well.

It's worth noting that tcp_connect_server() almost does not use the
stream interface anymore except for a few flags.

It has been identified that once we detach the connection from the SI,
it will probably be needed to keep a copy of the server-side addresses
in the SI just for logging purposes. This has not been implemented right
now though.
2012-09-03 20:47:34 +02:00
Willy Tarreau
3cefd521fa REORG: connection: move the target pointer from si to connection
The target is per connection and is directly used by the connection, so
we need it there. It's not needed anymore in the SI however.
2012-09-03 20:47:34 +02:00
Willy Tarreau
8263d2b259 CLEANUP: channel: use "channel" instead of "buffer" in function names
This is a massive rename of most functions which should make use of the
word "channel" instead of the word "buffer" in their names.

In concerns the following ones (new names) :

unsigned long long channel_forward(struct channel *buf, unsigned long long bytes);
static inline void channel_init(struct channel *buf)
static inline int channel_input_closed(struct channel *buf)
static inline int channel_output_closed(struct channel *buf)
static inline void channel_check_timeouts(struct channel *b)
static inline void channel_erase(struct channel *buf)
static inline void channel_shutr_now(struct channel *buf)
static inline void channel_shutw_now(struct channel *buf)
static inline void channel_abort(struct channel *buf)
static inline void channel_stop_hijacker(struct channel *buf)
static inline void channel_auto_connect(struct channel *buf)
static inline void channel_dont_connect(struct channel *buf)
static inline void channel_auto_close(struct channel *buf)
static inline void channel_dont_close(struct channel *buf)
static inline void channel_auto_read(struct channel *buf)
static inline void channel_dont_read(struct channel *buf)
unsigned long long channel_forward(struct channel *buf, unsigned long long bytes)

Some functions provided by channel.[ch] have kept their "buffer" name because
they are really designed to act on the buffer according to some information
gathered from the channel. They have been moved together to the same place in
the file for better readability but they were not changed at all.

The "buffer" memory pool was also renamed "channel".
2012-09-03 20:47:33 +02:00
Willy Tarreau
03cdb7c678 CLEANUP: channel: usr CF_/CHN_ prefixes instead of BF_/BUF_
Get rid of these confusing BF_* flags. Now channel naming should clearly
be used everywhere appropriate.

No code was changed, only a renaming was performed. The comments about
channel operations was updated.
2012-09-03 20:47:33 +02:00
Willy Tarreau
af81935b82 REORG: channel: move buffer_{replace,insert_line}* to buffer.{c,h}
These functions do not depend on the channel flags anymore thus they're
much better suited to be used on plain buffers. Move them from channel
to buffer.
2012-09-03 20:47:33 +02:00
Willy Tarreau
3bf1b2b816 MAJOR: channel: stop relying on BF_FULL to take action
This flag is quite complex to get right and updating it everywhere is a
major pain, especially since the buffer/channel split. This is the first
step of getting rid of it. Instead now it's dynamically computed whenever
needed.
2012-09-03 20:47:33 +02:00
Willy Tarreau
a75bcef867 REORG: buffer: move buffer_flush, b_adv and b_rew to buffer.h
These one now operate over real buffers, not channels anymore.
2012-09-03 20:47:32 +02:00
Willy Tarreau
8e21bb9e52 MAJOR: channel: remove the BF_OUT_EMPTY flag
This flag was very problematic because it was composite in that both changes
to the pipe or to the buffer had to cause this flag to be updated, which is
not always simple (eg: there may not even be a channel attached to a buffer
at all).

There were not that many users of this flags, mostly setters. So the flag got
replaced with a macro which reports whether the channel is empty or not, by
checking both the pipe and the buffer.

One part of the change is sensible : the flag was also part of BF_MASK_STATIC,
which is used by process_session() to rescan all analysers in case the flag's
status changes. At first glance, none of the analysers seems to change its
mind base on this flag when it is subject to change, so it seems fine not to
add variation checks here. Otherwise it's possible that checking the buffer's
output size is more useful than checking the flag's replacement.
2012-09-03 20:47:32 +02:00
Willy Tarreau
c7e4238df0 REORG: buffers: split buffers into chunk,buffer,channel
Many parts of the channel definition still make use of the "buffer" word.
2012-09-03 20:47:32 +02:00
Willy Tarreau
75bf2c925f REORG: sock_raw: rename the files raw_sock*
The "raw_sock" prefix will be more convenient for naming functions as
it will be prefixed with the data layer and suffixed with the data
direction. So let's rename the files now to avoid any further confusion.

The #include directive was also removed from a number of files which do
not need it anymore.
2012-09-02 21:54:56 +02:00
Willy Tarreau
572bf9095d REORG/MAJOR: extract "struct buffer" from "struct channel"
At the moment, the struct is still embedded into the struct channel, but
all the functions have been updated to use struct buffer only when possible,
otherwise struct channel. Some functions would likely need to be splitted
between a buffer-layer primitive and a channel-layer function.

Later the buffer should become a pointer in the struct buffer, but doing so
requires a few changes to the buffer allocation calls.
2012-09-02 21:54:56 +02:00
Willy Tarreau
7421efb85f REORG/MAJOR: use "struct channel" instead of "struct buffer"
This is a massive rename. We'll then split channel and buffer.

This change needs a lot of cleanups. At many locations, the parameter
or variable is still called "buf" which will become ambiguous. Also,
the "struct channel" is still defined in buffers.h.
2012-09-02 21:54:55 +02:00
Willy Tarreau
505e34a36d MAJOR: get rid of fdtab[].state and use connection->flags instead
fdtab[].state was only used to know whether a connection was in progress
or an error was encountered. Instead we now use connection->flags to store
a flag for both. This way, connection management will be able to update the
connection status on I/O.
2012-09-02 21:51:26 +02:00
Willy Tarreau
a9fddca778 MINOR: http: add the urlp_val ACL match
It's derived from other urlp_* matches, but there was no way to check for
an integer value and it seems like it's significantly used.
2012-07-31 07:55:32 +02:00
Willy Tarreau
dae2a8a5a5 BUG/MINOR: tarpit: fix condition to return the HTTP 500 message
Commit fa7e1025 (1.3.16-rc1) introduced a minor bug by comparing req->flags
with BF_READ_ERROR instead of checking for the bit. The result is that the
error message is always returned even in case of client error. This has no
real impact but this must be fixed.

It may be backported to 1.4 and 1.3.
2012-07-31 07:55:31 +02:00
Willy Tarreau
a7ad50cdb1 MEDIUM: pattern: add the "base" sample fetch method
This one returns the concatenation of the first Host header entry with
the path. It can make content-switching rules easier, help with fighting
DDoS on certain URLs and improve shared caches efficiency.
2012-07-26 19:08:38 +02:00
Willy Tarreau
6812bcfc94 MINOR: replace acl_fetch_{path,url}* with smp_fetch_*
Doing so allows us to support sticking on URL, URL's IP, URL's port and
path.

Both fetch functions should be improved to support an optional depth
allowing to stick to a server depending on just a few directory
components. This would help with portals, some prefetch-capable
caches and with outgoing connections using multiple internet links.
2012-07-26 19:06:40 +02:00
Willy Tarreau
a05903174f BUG/MAJOR: cookie prefix doesn't support cookie-less servers
Commit 827aee91 merged in 1.5-dev5 introduced a regression causing
the srv pointer to be tested twice instead of srv then srv->cookie.
The result is that if a server has no cookie in prefix mode, haproxy
will crash when trying to modify it.

Such a config is very unlikely to happen, except maybe with a backup
server, which would cause haproxy to die with the last server in the
farm.

No backport is needed, only 1.5-dev was affected.
2012-06-06 16:07:00 +02:00
Willy Tarreau
4f8a83cb6e MEDIUM: stats: add the ability to kill sessions from the admin interface
It was not possible to kill remaining sessions from the admin interface,
which is annoying especially when switching to maintenance mode. Now it's
possible.
2012-06-04 00:26:23 +02:00
Willy Tarreau
d72822442d MEDIUM: stats: add support for soft stop/soft start in the admin interface
One important missing feature on the web interface is the ability to perform
a soft stop/soft start. This is now possible.
2012-06-04 00:22:44 +02:00
Willy Tarreau
4992dd2d30 MINOR: http: add support for "httponly" and "secure" cookie attributes
httponly  This option tells haproxy to add an "HttpOnly" cookie attribute
             when a cookie is inserted. This attribute is used so that a
             user agent doesn't share the cookie with non-HTTP components.
             Please check RFC6265 for more information on this attribute.

   secure    This option tells haproxy to add a "Secure" cookie attribute when
             a cookie is inserted. This attribute is used so that a user agent
             never emits this cookie over non-secure channels, which means
             that a cookie learned with this flag will be presented only over
             SSL/TLS connections. Please check RFC6265 for more information on
             this attribute.
2012-05-31 21:02:17 +02:00
Willy Tarreau
674021329c REORG/MINOR: use dedicated proxy flags for the cookie handling
Cookies were mixed with many other options while they're not used as options.
Move them to a dedicated bitmask (ck_opts). This has released 7 flags in the
proxy options and leaves some room for new proxy flags.
2012-05-31 20:40:20 +02:00
Willy Tarreau
cde18fc1ba BUG/MINOR: perform_http_redirect also needs to rewind the buffer
Commit d1de8af362 was incomplete, because
perform_http_redirect() also needs to rewind the buffer since it's called
after data are scheduled for forwarding.

No backport needed.
2012-05-30 08:00:56 +02:00
Cyril Bonté
a32d275ab0 BUG/MEDIUM: option forwardfor if-none doesn't work with some configurations
When "option forwardfor" is enabled in a frontend that uses backends,
"if-none" ignores the header name provided in the frontend.
This prevents haproxy to add the X-Forwarded-For header if the option is not
used in the backend.

This may introduce security issues for servers/applications that rely on the
header provided by haproxy.

A minimal configuration which can reproduce the bug:
defaults
	mode http

listen OK
	bind :9000

	option forwardfor if-none
	server s1 127.0.0.1:80

listen BUG-frontend
	bind :9001

	option forwardfor if-none

	default_backend BUG-backend

backend BUG-backend
	server s1 127.0.0.1:80
2012-05-30 06:43:24 +02:00
Willy Tarreau
949811319b REORG/MEDIUM: stream_interface: move applet->state and private to connection
The state and the private pointer are not specific to the applets, since SSL
will require exactly both of them. Move them to the connection layer now and
rename them. We also now ensure that both are NULL on first call.
2012-05-21 17:09:48 +02:00
Willy Tarreau
fb7508aefb REORG/MINOR: stream_interface: move si->fd to struct connection
The socket fd is used only when in socket mode and with a connection.
2012-05-21 16:47:54 +02:00
Willy Tarreau
73b013b070 MINOR: stream_interface: introduce a new "struct connection" type
We start to move everything needed to manage a connection to a special
entity "struct connection". We have the data layer operations and the
control operations there. We'll also have more info in the future such
as file descriptors and applet contexts, so that in the end it becomes
detachable from the stream interface, which will allow connections to
be reused between sessions.

For now on, we start with minimal changes.
2012-05-21 16:31:45 +02:00
Willy Tarreau
ea95316bf1 MEDIUM: http: msg->sov and msg->sol will never wrap
These ones are offsets now, so they cannot wrap. Let's remove the useless
wrapping detection and simplify the forwarding code.
2012-05-18 23:50:43 +02:00
Willy Tarreau
2692736aa3 MEDIUM: http: get rid of msg->som which is not used anymore
msg->som was zero before the body and was used to carry the beginning
of a chunk size for chunked-encoded messages, at a moment when msg->sol
is always zero.

Remove msg->som and replace it with msg->sol where needed.
2012-05-18 23:50:43 +02:00
Willy Tarreau
06a000f56e CLEANUP: http: make it more obvious that msg->som is always null outside of chunks
Since the recent buffer reorg, msg->som is redundant with buf->p but still
appears at a number of places. This tiny patch allows to confirm that som
follows two states :
  - 0 from the moment the message starts to be parsed
  - relative offset to ->p for start of chunk when parsing chunks

During this second state, ->sol is never used, so we should probably merge
the two.
2012-05-18 23:04:32 +02:00