Commit Graph

381 Commits

Author SHA1 Message Date
Willy Tarreau
ba2ffd18b5 MEDIUM: counters: add a new "gpc0_rate" counter in stick-tables
This counter is special in that instead of reporting the gpc0 cumulative
count, it returns its increase rate over the configured period.
2013-05-29 15:54:14 +02:00
Willy Tarreau
e25c917af8 MEDIUM: counters: add support for tracking a third counter
We're often missin a third counter to track base, src and base+src at
the same time. Here we introduce track_sc3 to have this third counter.
It would be wise not to add much more counters because that slightly
increases the session size and processing time though the real issue
is more the declaration of the keywords in the code and in the doc.
2013-05-29 00:37:16 +02:00
Willy Tarreau
d5ca9abb0d MINOR: counters: make it easier to extend the amount of tracked counters
By properly affecting the flags and values, it becomes easier to add
more tracked counters, for example for experimentation. It also slightly
reduces the code and the number of tests. No counters were added with
this patch.
2013-05-28 17:43:40 +02:00
Willy Tarreau
1e5dfdad77 MINOR: session: only call http_send_name_header() when changing the server
Till now we used to call the function until the connection established, which
means that the header rewriting was performed for nothing upon each even (eg:
uploaded contents) until the server responded or timed out.

Now we only call the function when we assign the server.
2013-04-11 18:18:01 +02:00
Willy Tarreau
d86e29d2a1 CLEANUP: acl: remove unused references to ACL_USE_*
Now that acl->requires is not used anymore, we can remove all references
to it as well as all ACL_USE_* flags.
2013-04-03 02:13:00 +02:00
Willy Tarreau
c48c90dfa5 MAJOR: acl: remove the arg_mask from the ACL definition and use the sample fetch's
Now that ACLs solely rely on sample fetch functions, make them use the
same arg mask. All inconsistencies have been fixed separately prior to
this patch, so this patch almost only adds a new pointer indirection
and removes all references to ARG*() in the definitions.

The parsing is still performed by the ACL code though.
2013-04-03 02:12:58 +02:00
Willy Tarreau
8ed669b12a MAJOR: acl: make all ACLs reference the fetch function via a sample.
ACL fetch functions used to directly reference a fetch function. Now
that all ACL fetches have their sample fetches equivalent, we can make
ACLs reference a sample fetch keyword instead.

In order to simplify the code, a sample keyword name may be NULL if it
is the same as the ACL's, which is the most common case.

A minor change appeared, http_auth always expects one argument though
the ACL allowed it to be missing and reported as such afterwards, so
fix the ACL to match this. This is not really a bug.
2013-04-03 02:12:58 +02:00
Willy Tarreau
281c799e25 MINOR: session: rename sample fetch functions and declare the sample keywords
The following sample fetch functions were only usable by ACLs but are now
usable by sample fetches too :

  sc1_bytes_in_rate, sc1_bytes_out_rate, sc1_clr_gpc0, sc1_conn_cnt,
  sc1_conn_cur, sc1_conn_rate, sc1_get_gpc0, sc1_http_err_cnt,
  sc1_http_err_rate, sc1_http_req_cnt, sc1_http_req_rate, sc1_inc_gpc0,
  sc1_kbytes_in, sc1_kbytes_out, sc1_sess_cnt, sc1_sess_rate, sc1_trackers,
  sc2_bytes_in_rate, sc2_bytes_out_rate, sc2_clr_gpc0, sc2_conn_cnt,
  sc2_conn_cur, sc2_conn_rate, sc2_get_gpc0, sc2_http_err_cnt,
  sc2_http_err_rate, sc2_http_req_cnt, sc2_http_req_rate, sc2_inc_gpc0,
  sc2_kbytes_in, sc2_kbytes_out, sc2_sess_cnt, sc2_sess_rate, sc2_trackers,
  src_bytes_in_rate, src_bytes_out_rate, src_clr_gpc0, src_conn_cnt,
  src_conn_cur, src_conn_rate, src_get_gpc0, src_http_err_cnt,
  src_http_err_rate, src_http_req_cnt, src_http_req_rate, src_inc_gpc0,
  src_kbytes_in, src_kbytes_out, src_sess_cnt, src_sess_rate,
  src_updt_conn_cnt, table_avl, table_cnt,

The fetch functions have been renamed "smp_fetch_*".
2013-04-03 02:12:58 +02:00
Willy Tarreau
a7a7ebc382 BUG/MINOR: http: don't process abortonclose when request was sent
option abortonclose may cause a valid connection to be aborted just
after the request has been sent. This is because we check for it
during the session establishment sequence before checking for write
activity. So if the abort and the connect complete at the same time,
the abort is still considered. Let's check for an explicity partial
write before aborting.

This fix should be backported to 1.4 too.
2012-12-30 00:50:35 +01:00
Willy Tarreau
71241abfd3 MINOR: http: move redirect rule processing to its own function
We now have http_apply_redirect_rule() which does all the redirect-specific
job instead of having this inside http_process_req_common().

Also one of the benefit gained from uniformizing this code is that both
keep-alive and close response do emit the PR-- flags. The fix for the
flags could probably be backported to 1.4 though it's very minor.

The previous function http_perform_redirect() was becoming confusing
so it was renamed http_perform_server_redirect() since it only applies
to server-based redirection.
2012-12-28 14:47:19 +01:00
Willy Tarreau
d79a3b248e BUG/MINOR: log: make log-format, unique-id-format and add-header more independant
It happens that all of them call parse_logformat_line() which sets
proxy->to_log with a number of flags affecting the line format for
all three users. For example, having a unique-id specified disables
the default log-format since fe->to_log is tested when the session
is established.

Similarly, having "option logasap" will cause "+" to be inserted in
unique-id or headers referencing some of the fields depending on
LW_BYTES.

This patch first removes most of the dependency on fe->to_log whenever
possible. The first possible cleanup is to stop checking fe->to_log
for being null, considering that it always contains at least LW_INIT
when any such usage is made of the log-format!

Also, some checks are wrong. s->logs.logwait cannot be nulled by
"logwait &= ~LW_*" since LW_INIT is always there. This results in
getting the wrong log at the end of a request or session when a
unique-id or add-header is set, because logwait is still not null
but the log-format is not checked.

Further cleanups are required. Most LW_* flags should be removed or at
least replaced with what they really mean (eg: depend on client-side
connection, depend on server-side connection, etc...) and this should
only affect logging, not other mechanisms.

This patch fixes the default log-format and tries to limit interferences
between the log formats, but does not pretend to do more for the moment,
since it's the most visible breakage.
2012-12-28 09:51:00 +01:00
Willy Tarreau
20d46a5a95 CLEANUP: session: use an array for the stick counters
The stick counters were in two distinct sets of struct members,
causing some code to be duplicated. Now we use an array, which
enables some processing to be performed in loops. This allowed
the code to be shrunk by 700 bytes.
2012-12-09 15:57:16 +01:00
Willy Tarreau
2406db4b39 MEDIUM: counters: add sc1_trackers/sc2_trackers
Returns the current amount of concurrent connections tracking the same
tracked counters. This number is automatically incremented when tracking
begins and decremented when tracking stops. It differs from sc1_conn_cur in
that it does not rely on any stored information but on the table's reference
count (the "use" value which is returned by "show table" on the CLI). This
may sometimes be more suited for layer7 tracking.
2012-12-09 14:08:47 +01:00
Willy Tarreau
5d5b5d8eaf MEDIUM: proto_tcp: add support for tracking L7 information
Until now it was only possible to use track-sc1/sc2 with "src" which
is the IPv4 source address. Now we can use track-sc1/sc2 with any fetch
as well as any transformation type. It works just like the "stick"
directive.

Samples are automatically converted to the correct types for the table.

Only "tcp-request content" rules may use L7 information, and such information
must already be present when the tracking is set up. For example it becomes
possible to track the IP address passed in the X-Forwarded-For header.

HTTP request processing now also considers tracking from backend rules
because we want to be able to update the counters even when the request
was already parsed and tracked.

Some more controls need to be performed (eg: samples do not distinguish
between L4 and L6).
2012-12-09 14:08:47 +01:00
Willy Tarreau
0ede5a3318 BUG/MEDIUM: session: fix FD leak when transport layer logging is enabled
Commit 2b199c9a attempted to fix all places where the transport layer
is improperly closed, but it missed one place in session_free(). If
SSL ciphers are logged, the close() is delayed post-log and performed
in session_free(). However, conn_xprt_close() only closes the transport
layer but not the file descriptor, resulting in a slow FD leak which is
hardly noticeable until the process cannot accept any new connection.

A workaround consisted in disabling %sslv/%sslc in log-format.

So use conn_full_close() instead of conn_xprt_close() to fix this there
too.

A similar pending issue existed in the close during outgoing connection
failure, though on this side, the transport layer is never tracked at the
moment.
2012-12-08 08:48:04 +01:00
Willy Tarreau
20879a0233 MEDIUM: connection: add error reporting for the SSL
Get a bit more info in the logs when client-side SSL handshakes fail.
2012-12-03 17:21:52 +01:00
Willy Tarreau
8e3bf699db MEDIUM: connection: add error reporting for the PROXY protocol header
When the PROXY protocol header is expected and fails, leading to an
abort of the incoming connection, we now emit a log message. If option
dontlognull is set and it was just a port probe, then nothing is logged.
2012-12-03 17:21:51 +01:00
Willy Tarreau
0af2912fd1 MEDIUM: connection: add minimal error reporting in logs for incomplete connections
Since the introduction of SSL, it became quite annoying not to get any useful
info in logs about handshake failures. Let's improve reporting for embryonic
sessions by checking a per-connection error code and reporting it into the logs
if an error happens before the session is completely instanciated.

The "dontlognull" option is supported in that if a connection does not talk
before being aborted, nothing will be emitted.

At the moment, only timeouts are considered for SSL and the PROXY protocol,
but next patches will handle more errors.
2012-12-03 15:38:23 +01:00
Willy Tarreau
14cba4b0b1 MEDIUM: connection: add an error code in connections
This will be needed to improve error reporting, especially for SSL.
2012-12-03 14:22:13 +01:00
Willy Tarreau
8139b9959f MINOR: compression: make the stats a bit more robust
To ensure that we only count when a response was compressed, we also
check for the SN_COMP_READY flag which indicates that the compression
was effectively initialized. Comp_algo alone is meaningless.
2012-11-27 09:34:00 +01:00
Willy Tarreau
5e16cbc3bd MINOR: stats: report the total number of compressed responses per front/back
Depending on the content-types and accept-encoding fields, some responses
might or might not be compressed. Let's have a counter of the number of
compressed responses and report it in the stats to help improve compression
usage.

Some cosmetic issues were fixed in the CSV output too (missing commas at the
end).
2012-11-24 14:54:13 +01:00
Willy Tarreau
2b199c9ac3 MEDIUM: connection: provide a common conn_full_close() function
Several places got the connection close sequence wrong because it
was not obvious. In practice we always need the same sequence when
aborting, so let's have a common function for this.
2012-11-23 17:32:21 +01:00
Willy Tarreau
543db62e1f BUG/MEDIUM: compression: release the zlib pools between keep-alive requests
There was a possible memory leak in the zlib code when the first response of
a keep-alive session was compressed, because the next request would reset the
compression algo, preventing a later call to session_free() from releasing it.
The reason is that it is necessary to release the assigned resources in
http_end_txn_clean_session().
2012-11-15 16:41:22 +01:00
William Lallemand
ec3e3890f0 BUG/MINOR: compression: deinit zlib only when required
The zlib stream was deinitialized even when the init failed.
2012-11-15 15:42:17 +01:00
Willy Tarreau
3fdb366885 MAJOR: connection: replace struct target with a pointer to an enum
Instead of storing a couple of (int, ptr) in the struct connection
and the struct session, we use a different method : we only store a
pointer to an integer which is stored inside the target object and
which contains a unique type identifier. That way, the pointer allows
us to retrieve the object type (by dereferencing it) and the object's
address (by computing the displacement in the target structure). The
NULL pointer always corresponds to OBJ_TYPE_NONE.

This reduces the size of the connection and session structs. It also
simplifies target assignment and compare.

In order to improve the generated code, we try to put the obj_type
element at the beginning of all the structs (listener, server, proxy,
si_applet), so that the original and target pointers are always equal.

A lot of code was touched by massive replaces, but the changes are not
that important.
2012-11-12 00:42:33 +01:00
Willy Tarreau
b31c971bef CLEANUP: channel: remove any reference of the hijackers
Hijackers were functions designed to inject data into channels in the
distant past. They became unused around 1.3.16, and since there has
not been any user of this mechanism to date, it's uncertain whether
the mechanism still works (and it's not really useful anymore). So
better remove it as well as the pointer it uses in the channel struct.
2012-11-11 23:05:39 +01:00
Willy Tarreau
7f7ad91056 BUILD: stream_interface: remove si_fd() and its references
si_fd() is not used a lot, and breaks builds on OpenBSD 5.2 which
defines this name for its own purpose. It's easy enough to remove
this one-liner function, so let's do it.
2012-11-11 20:53:29 +01:00
Willy Tarreau
815f5ecffa BUG/MINOR: session: mark the handshake as complete earlier
There is a small waste of CPU cycles when no handshake is required on an
accepted connection, because we had to perform one call to conn_fd_handler()
to mark the connection CONNECTED and to call process_session() again to say
that nothing happened.

By marking the connection CONNECTED when there is no pending handshake, we
avoid this extra call to process_session().
2012-11-09 22:09:08 +01:00
Willy Tarreau
798f4325fa OPTIM: session: don't process the whole session when only timers need a refresh
Having a global expiration timer for a task means that the tasks are regularly
woken up (at least after each expiration timer). It's totally useless and counter
productive to process the whole session upon each such wakeup, and it's fairly
easy to detect such wakeups, so let's just update the task's timer and return
to sleep when this happens.

For 100k concurrent connections with 10s of timeouts, this can save 10k wakeups
per second, which is not bad.
2012-11-08 16:55:07 +01:00
William Lallemand
1c2d622d82 CLEANUP: use struct comp_ctx instead of union
Replace union comp_ctx by struct comp_ctx.

Use struct comp_ctx * in the init/add_data/flush/reset/end prototypes of
compression.h functions.
2012-11-05 10:23:16 +01:00
Willy Tarreau
e3224e870f BUG/MINOR: session: ensure that we don't retry connection if some data were sent
With extra-large buffers, it is possible that a lot of data are sent upon
connection establishment before the session is notified. The issue is how
to handle a send() error after some data were actually sent.

At the moment, only a connection error is reported, causing a new connection
attempt and send() to restart after the last data. We absolutely don't want
to retry the connect() if at least one byte was sent, because those data are
lost.

The solution consists in reporting exactly what happens, which is :
  - a successful connection attempt
  - a read/write error on the channel

That way we go on with sess_establish(), the response analysers are called
and report the appropriate connection state for the error (typically a server
abort while waiting for a response). This mechanism also guarantees that we
won't retry since it's a success. The logs also report the correct connect
time.

Note that 1.4 is not directly affected because it only attempts one send(),
so it cannot detect a send() failure here and distinguish it form a failed
connection attempt. So no backport is needed. Also, this is just a safe belt
we're taking, since this issue should not happen anymore since previous commit.
2012-10-29 23:31:04 +01:00
Willy Tarreau
19d14ef104 MEDIUM: make the trash be a chunk instead of a char *
The trash is used everywhere to store the results of temporary strings
built out of s(n)printf, or as a storage for a chunk when chunks are
needed.

Using global.tune.bufsize is not the most convenient thing either.

So let's replace trash with a chunk and directly use it as such. We can
then use trash.size as the natural way to get its size, and get rid of
many intermediary chunks that were previously used.

The patch is huge because it touches many areas but it makes the code
a lot more clear and even outlines places where trash was used without
being that obvious.
2012-10-29 16:57:30 +01:00
Willy Tarreau
f2943dccd0 MAJOR: session: detach the connections from the stream interfaces
We will need to be able to switch server connections on a session and
to keep idle connections. In order to achieve this, the preliminary
requirement is that the connections can survive the session and be
detached from them.

Right now they're still allocated at exactly the same place, so when
there is a session, there are always 2 connections. We could soon
improve on this by allocating the outgoing connection only during a
connect().

This current patch touches a lot of code and intentionally does not
change any functionnality. Performance tests show no regression (even
a very minor improvement). The doc has not yet been updated.
2012-10-26 20:15:20 +02:00
Willy Tarreau
c919dc66a3 CLEANUP: remove trashlen
trashlen is a copy of global.tune.bufsize, so let's stop using it as
a duplicate, fall back to the original bufsize, it's less confusing
this way.
2012-10-26 20:04:27 +02:00
William Lallemand
82fe75c1a7 MEDIUM: HTTP compression (zlib library support)
This commit introduces HTTP compression using the zlib library.

http_response_forward_body has been modified to call the compression
functions.

This feature includes 3 algorithms: identity, gzip and deflate:

  * identity: this is mostly for debugging, and it was useful for
  developping the compression feature. With Content-Length in input, it
  is making each chunk with the data available in the current buffer.
  With chunks in input, it is rechunking, the output chunks will be
  bigger or smaller depending of the size of the input chunk and the
  size of the buffer. Identity does not apply any change on data.

  * gzip: same as identity, but applying a gzip compression. The data
  are deflated using the Z_NO_FLUSH flag in zlib. When there is no more
  data in the input buffer, it flushes the data in the output buffer
  (Z_SYNC_FLUSH). At the end of data, when it receives the last chunk in
  input, or when there is no more data to read, it writes the end of
  data with Z_FINISH and the ending chunk.

  * deflate: same as gzip, but with deflate algorithm and zlib format.
  Note that this algorithm has ambiguous support on many browsers and
  no support at all from recent ones. It is strongly recommended not
  to use it for anything else than experimentation.

You can't choose the compression ratio at the moment, it will be set to
Z_BEST_SPEED (1), as tests have shown very little benefit in terms of
compression ration when going above for HTML contents, at the cost of
a massive CPU impact.

Compression will be activated depending of the Accept-Encoding request
header. With identity, it does not take care of that header.

To build HAProxy with zlib support, use USE_ZLIB=1 in the make
parameters.

This work was initially started by David Du Colombier at Exceliance.
2012-10-26 02:30:48 +02:00
Willy Tarreau
c93f7959e5 CLEANUP: session: remove term_trace which is not used anymore
This field was used to trace precisely where a session was terminated
but it did not survive code rearchitecture and was not used at all
anymore. Let's get rid of it.
2012-10-13 11:10:30 +02:00
Willy Tarreau
9b28e03b66 MAJOR: channel: replace the struct buffer with a pointer to a buffer
With this commit, we now separate the channel from the buffer. This will
allow us to replace buffers on the fly without touching the channel. Since
nobody is supposed to keep a reference to a buffer anymore, doing so is not
a problem and will also permit some copy-less data manipulation.

Interestingly, these changes have shown a 2% performance increase on some
workloads, probably due to a better cache placement of data.
2012-10-13 09:07:52 +02:00
Willy Tarreau
394db379eb REORG: http: rename msg->buf to msg->chn since it's a channel
It's extremely confusing to have all those msg->buf->buf everywhere after
the extraction of the buffer from the channel. Let's clean this up.
2012-10-12 22:40:39 +02:00
Willy Tarreau
93dbc2bc0e MEDIUM: log: add a new LW_XPRT flag to pin the transport layer
This flag will have to be set on log tags which require transport layer
information. They will prevent the conn_xprt_close() call from releasing
the transport layer too early.
2012-10-12 20:30:51 +02:00
Willy Tarreau
1e954913de MEDIUM: connection: add a flag to hold the transport layer
When we start logging SSL information, we need the SSL struct to be
present even past the conn_xprt_close() call. In order to achieve this,
we should use refcounting on the connection and the transport layer. At
the moment it's not worth using plain refcounting as only the logs require
this, so instead of real refcounting we just use a flag which will be set
by the log subsystem when SSL data need to be logged.

What happens then is that the xprt->close() call is ignored and the
transport layer is closed again during session_free(), after the log
line is emitted.
2012-10-12 20:30:50 +02:00
Willy Tarreau
91083f5c8f BUG/MEDIUM: session: enable the conn_session_update() callback
This callback was introduced by commit 9683e9a0 but never enabled because
the CO_FL_WAKE_DATA flag was not set. The result is that this function is
never called when an SSL handshake fails, so the connection is only closed
on timeout.
2012-10-12 20:30:38 +02:00
Willy Tarreau
e9909f4e50 BUG/MINOR: session: fix some leftover from debug code
Commit 82569f91 moved the health and monitor-net checks to session.c
but a debug test introduced 0& to disable MSG_DONTWAIT in the recv()
call and this debug code remained there. Since the socket is marked
non-blocking, there should be no effect but it's dangerous to keep
such a thing here.
2012-10-12 17:36:40 +02:00
Willy Tarreau
1bc4aab290 MEDIUM: listener: add support for linux's accept4() syscall
On Linux, accept4() does the same as accept() except that it allows
the caller to specify some flags to set on the resulting socket. We
use this to set the O_NONBLOCK flag and thus to save one fcntl()
call in each connection. The effect is a small performance gain of
around 1%.

The option is automatically enabled when target linux2628 is set, or
when the USE_ACCEPT4 Makefile variable is set. If the libc is too old
to provide the equivalent function, this is automatically detected and
our own function is used instead. In any case it is possible to force
the use of our implementation with USE_MY_ACCEPT4.
2012-10-08 20:11:03 +02:00
Willy Tarreau
9683e9a05f MEDIUM: session: register a data->wake callback to process errors
The connection layer will soon call ->wake() only when errors happen, and
not ->init(). So make the session layer use this callback to detect errors
and abort connections.
2012-10-04 22:26:10 +02:00
Willy Tarreau
071e137ec2 MEDIUM: connection: use a generic data-layer init() callback
The generic data-layer init callback is now used after the transport
layer is complete and before calling the data layer recv/send callbacks.

This allows the session to switch from the embryonic session data layer
to the complete stream interface data layer, by making conn_session_complete()
the data layer's init callback.

It sill looks awkwards that the init() callback must be used opon error,
but except by adding yet another one, it does not seem to be mergeable
into another function (eg: it should probably not be merged with ->wake
to avoid unneeded calls during the handshake, though semantically that
would make sense).
2012-10-04 22:26:10 +02:00
Willy Tarreau
5e75e2755e MEDIUM: session: use a specific data_cb for embryonic sessions
We don't want to have the recv or send callbacks in embryonic
sessions, and we want the stream interface to be referenced as
the connection owner only once the session is instanciated. So
let's first have the embryonic session be the owner, then replaced
later by the stream interface once the transport layer is ready.
2012-10-04 22:26:10 +02:00
Willy Tarreau
4aa3683b2d MINOR: connection: provide a generic data layer wakeup callback
Instead of calling conn_notify_si() from the connection handler, we
now call data->wake(), which will allow us to use a different callback
with health checks.

Note that we still rely on a flag in order to decide whether or not
to call this function. The reason is that with embryonic sessions,
the callback is already initialized to si_conn_cb without the flag,
and we can't call the SI notify function in the leave path before
the stream interface is initialized.

This issue should be addressed by involving a different data_cb for
embryonic sessions and for stream interfaces, that would be changed
during session_complete() for the final data_cb.
2012-10-04 22:26:10 +02:00
Willy Tarreau
f7bc57ca6e REORG: connection: rename the data layer the "transport layer"
While working on the changes required to make the health checks use the
new connections, it started to become obvious that some naming was not
logical at all in the connections. Specifically, it is not logical to
call the "data layer" the layer which is in charge for all the handshake
and which does not yet provide a data layer once established until a
session has allocated all the required buffers.

In fact, it's more a transport layer, which makes much more sense. The
transport layer offers a medium on which data can transit, and it offers
the functions to move these data when the upper layer requests this. And
it is the upper layer which iterates over the transport layer's functions
to move data which should be called the data layer.

The use case where it's obvious is with embryonic sessions : an incoming
SSL connection is accepted. Only the connection is allocated, not the
buffers nor stream interface, etc... The connection handles the SSL
handshake by itself. Once this handshake is complete, we can't use the
data functions because the buffers and stream interface are not there
yet. Hence we have to first call a specific function to complete the
session initialization, after which we'll be able to use the data
functions. This clearly proves that SSL here is only a transport layer
and that the stream interface constitutes the data layer.

A similar change will be performed to rename app_cb => data, but the
two could not be in the same commit for obvious reasons.
2012-10-04 22:26:09 +02:00
Willy Tarreau
e603e69d18 MEDIUM: connection: make use of the owner instead of container_of
This way the connection can become independant on the stream interface.
2012-09-28 00:01:23 +02:00
Willy Tarreau
82569f9158 MEDIUM: monitor: simplify handling of monitor-net and mode health
We were having several different behaviours with monitor-net and
"mode health" :
  - monitor-net on TCP connections was evaluated just after accept(),
    did not count a connection on the frontend and were not subject
    to tcp-request connection rules, and caused an immediate close().

  - monitor-net in HTTP mode was evaluated once the session was
    accepted (eg: on top of SSL), returned "HTTP/1.0 200 OK\r\n\r\n"
    over the connection's data layer and instanciated a session which
    was responsible for closing this connection. A connection AND a
    session were counted for the frontend ;

  - "mode health" with "option httpchk" would do exactly the same as
    monitor-net in HTTP mode ;

  - "mode health" without "option httpchk" would do the same as above
    except that "OK" was returned instead of "HTTP/1.0 200 OK\r\n\r\n".

None of them took care of cleaning the input buffer, sometimes resulting
in a TCP reset to be emitted after the last packet if a request was received
over the connection.

Given the inconsistencies and the complexity in keeping all these features
handled at the right position, we now slightly changed the way they are
handled :

  - all of them are handled just after the "tcp-request connection" rules,
    so that all of them may be blocked using such rules, offering more
    flexibility and consistency ;

  - no connection handshake is performed anymore for non-TCP modes

  - all of them send the response as raw data over the socket, there is no
    more difference between TCP and HTTP mode for example (these rules were
    never meant to be served over SSL connections and were never documented
    as able to do that).

  - any possible pending data on the incoming socket is drained before the
    response is sent, in order to avoid the risk of a reset.

  - none of them exactly did what was documented !

This results in more consistent, more flexible and more accurate handling of
monitor rules, with smaller and more robust code.
2012-09-28 00:01:22 +02:00
Cyril Bonté
3aaba440a2 BUILD: fix compilation error with DEBUG_FULL
Recent changes in structures broke the compilation when using DEBUG_FULL.
Let's update apply the changes also to the variables used in DPRINTF calls.
2012-09-24 20:36:39 +02:00
Willy Tarreau
d1d5454180 REORG: split "protocols" files into protocol and listener
It was becoming confusing to have protocols and listeners in the same
files, split them.
2012-09-15 22:29:32 +02:00
Willy Tarreau
cbaaec475c MINOR: session: do not send an HTTP/500 error on SSL sockets
If a session fails its initialization, we don't want to send HTTP/500
over the socket if it's not a raw data layer.
2012-09-06 11:32:07 +02:00
Willy Tarreau
783f25800c BUILD: http: rename error_message http_error_message to fix conflicts on RHEL
Duncan Hall reported a build issue on CentOS where error_message conflicts
with another system declaration when SSL is enabled. Rename the function.
2012-09-04 12:19:04 +02:00
Willy Tarreau
dd2f85eb3b CLEANUP: includes: fix includes for a number of users of fd.h
It appears that fd.h includes a number of unneeded files and was
included from standard.h, and as such served as an intermediary
to provide almost everything to everyone.

By removing its useless includes, a long dependency chain broke
but could easily be fixed.
2012-09-03 20:49:14 +02:00
Willy Tarreau
40ff59d820 CLEANUP: fd: remove fdtab->flags
These flags were added for TCP_CORK. They were only set at various places
but never checked by any user since TCP_CORK was replaced with MSG_MORE.
Simply get rid of this now.
2012-09-03 20:49:14 +02:00
Willy Tarreau
74172ff9c3 CLEANUP: frontend: remove the old proxy protocol decoder
This one used to rely on a stream analyser which was inappropriate.
It's not used anymore.
2012-09-03 20:47:35 +02:00
Willy Tarreau
22cda21ad5 MAJOR: connection: make the PROXY decoder a handshake handler
The PROXY protocol is now decoded in the connection before other
handshakes. This means that it may be extracted from a TCP stream
before SSL is decoded from this stream.
2012-09-03 20:47:35 +02:00
Willy Tarreau
2542b53b19 MAJOR: session: introduce embryonic sessions
When an incoming connection request is accepted, a connection
structure is needed to store its state. However we don't want to
fully initialize a session until the data layer is about to be
ready.

As long as the connection is physically stored into the session,
it's not easy to split both allocations.

As such, we only initialize the minimum requirements of a session,
which results in what we call an embryonic session. Then once the
data layer is ready, we can complete the function's initialization.

Doing so avoids buffers allocation and ensures that a session only
sees ready connections.

The frontend's client timeout is used as the handshake timeout. It
is likely that another timeout will be used in the future.
2012-09-03 20:47:35 +02:00
Willy Tarreau
15678efc45 MEDIUM: connection: add an ->init function to data layer
SSL need to initialize the data layer before proceeding with data. At
the moment, this data layer is automatically initialized from itself,
which will not be possible once we extract connection from sessions
since we'll only create the data layer once the handshake is finished.

So let's have the application layer initialize the data layer before
using it.
2012-09-03 20:47:34 +02:00
Willy Tarreau
64ee491309 MINOR: tcp: replace tcp_src_to_stktable_key with addr_to_stktable_key
Make it more obvious that this function does not depend on any knowledge
of the session. This is important to plan for TCP rules that can run on
connection without any initialized session yet.
2012-09-03 20:47:34 +02:00
Willy Tarreau
93b0f4f6c6 MEDIUM: stream_interface: remove CAP_SPLTCP/CAP_SPLICE flags
These ones are implicitly handled by the connection's data layer, no need
to rely on them anymore and reaching them maintains undesired dependences
on stream-interface.
2012-09-03 20:47:34 +02:00
Willy Tarreau
986a9d2d12 MAJOR: connection: move the addr field from the stream_interface
We need to have the source and destination addresses in the connection.
They were lying in the stream interface so let's move them. The flags
SI_FL_FROM_SET and SI_FL_TO_SET have been moved as well.

It's worth noting that tcp_connect_server() almost does not use the
stream interface anymore except for a few flags.

It has been identified that once we detach the connection from the SI,
it will probably be needed to keep a copy of the server-side addresses
in the SI just for logging purposes. This has not been implemented right
now though.
2012-09-03 20:47:34 +02:00
Willy Tarreau
3cefd521fa REORG: connection: move the target pointer from si to connection
The target is per connection and is directly used by the connection, so
we need it there. It's not needed anymore in the SI however.
2012-09-03 20:47:34 +02:00
Willy Tarreau
8263d2b259 CLEANUP: channel: use "channel" instead of "buffer" in function names
This is a massive rename of most functions which should make use of the
word "channel" instead of the word "buffer" in their names.

In concerns the following ones (new names) :

unsigned long long channel_forward(struct channel *buf, unsigned long long bytes);
static inline void channel_init(struct channel *buf)
static inline int channel_input_closed(struct channel *buf)
static inline int channel_output_closed(struct channel *buf)
static inline void channel_check_timeouts(struct channel *b)
static inline void channel_erase(struct channel *buf)
static inline void channel_shutr_now(struct channel *buf)
static inline void channel_shutw_now(struct channel *buf)
static inline void channel_abort(struct channel *buf)
static inline void channel_stop_hijacker(struct channel *buf)
static inline void channel_auto_connect(struct channel *buf)
static inline void channel_dont_connect(struct channel *buf)
static inline void channel_auto_close(struct channel *buf)
static inline void channel_dont_close(struct channel *buf)
static inline void channel_auto_read(struct channel *buf)
static inline void channel_dont_read(struct channel *buf)
unsigned long long channel_forward(struct channel *buf, unsigned long long bytes)

Some functions provided by channel.[ch] have kept their "buffer" name because
they are really designed to act on the buffer according to some information
gathered from the channel. They have been moved together to the same place in
the file for better readability but they were not changed at all.

The "buffer" memory pool was also renamed "channel".
2012-09-03 20:47:33 +02:00
Willy Tarreau
03cdb7c678 CLEANUP: channel: usr CF_/CHN_ prefixes instead of BF_/BUF_
Get rid of these confusing BF_* flags. Now channel naming should clearly
be used everywhere appropriate.

No code was changed, only a renaming was performed. The comments about
channel operations was updated.
2012-09-03 20:47:33 +02:00
Willy Tarreau
3bf1b2b816 MAJOR: channel: stop relying on BF_FULL to take action
This flag is quite complex to get right and updating it everywhere is a
major pain, especially since the buffer/channel split. This is the first
step of getting rid of it. Instead now it's dynamically computed whenever
needed.
2012-09-03 20:47:33 +02:00
Willy Tarreau
a75bcef867 REORG: buffer: move buffer_flush, b_adv and b_rew to buffer.h
These one now operate over real buffers, not channels anymore.
2012-09-03 20:47:32 +02:00
Willy Tarreau
8e21bb9e52 MAJOR: channel: remove the BF_OUT_EMPTY flag
This flag was very problematic because it was composite in that both changes
to the pipe or to the buffer had to cause this flag to be updated, which is
not always simple (eg: there may not even be a channel attached to a buffer
at all).

There were not that many users of this flags, mostly setters. So the flag got
replaced with a macro which reports whether the channel is empty or not, by
checking both the pipe and the buffer.

One part of the change is sensible : the flag was also part of BF_MASK_STATIC,
which is used by process_session() to rescan all analysers in case the flag's
status changes. At first glance, none of the analysers seems to change its
mind base on this flag when it is subject to change, so it seems fine not to
add variation checks here. Otherwise it's possible that checking the buffer's
output size is more useful than checking the flag's replacement.
2012-09-03 20:47:32 +02:00
Willy Tarreau
c7e4238df0 REORG: buffers: split buffers into chunk,buffer,channel
Many parts of the channel definition still make use of the "buffer" word.
2012-09-03 20:47:32 +02:00
Willy Tarreau
c578891112 CLEANUP: connection: split sock_ops into data_ops, app_cp and si_ops
Some parts of the sock_ops structure were only used by the stream
interface and have been moved into si_ops. Some of them were callbacks
to the stream interface from the connection and have been moved into
app_cp as they're the application seen from the connection (later,
health-checks will need to use them). The rest has moved to data_ops.

Normally at this point the connection could live without knowing about
stream interfaces at all.
2012-09-03 20:47:31 +02:00
Willy Tarreau
96199b1016 MAJOR: stream-interface: restore splicing mechanism
The splicing is now provided by the data-layer rcv_pipe/snd_pipe functions
which in turn are called by the stream interface's recv and send callbacks.

The presence of the rcv_pipe/snd_pipe functions is used to attest support
for splicing at the data layer. It looks like the stream-interface's
SI_FL_CAP_SPLICE flag does not make sense anymore as it's used as a proxy
for the pointers above.

It also appears that we call chk_snd() from the recv callback and then
try to call it again in update_conn(). It is very likely that this last
function will progressively slip into the recv/send callbacks in order
to avoid duplicate check code.

The code works right now with and without splicing. Only raw_sock provides
support for it and it is automatically selected when the various splice
options are set. However it looks like splice-auto doesn't enable it, which
possibly means that the streamer detection code does not work anymore, or
that it's only called at a time where it's too late to enable splicing (in
process_session).
2012-09-03 20:47:31 +02:00
Willy Tarreau
75bf2c925f REORG: sock_raw: rename the files raw_sock*
The "raw_sock" prefix will be more convenient for naming functions as
it will be prefixed with the data layer and suffixed with the data
direction. So let's rename the files now to avoid any further confusion.

The #include directive was also removed from a number of files which do
not need it anymore.
2012-09-02 21:54:56 +02:00
Willy Tarreau
572bf9095d REORG/MAJOR: extract "struct buffer" from "struct channel"
At the moment, the struct is still embedded into the struct channel, but
all the functions have been updated to use struct buffer only when possible,
otherwise struct channel. Some functions would likely need to be splitted
between a buffer-layer primitive and a channel-layer function.

Later the buffer should become a pointer in the struct buffer, but doing so
requires a few changes to the buffer allocation calls.
2012-09-02 21:54:56 +02:00
Willy Tarreau
7421efb85f REORG/MAJOR: use "struct channel" instead of "struct buffer"
This is a massive rename. We'll then split channel and buffer.

This change needs a lot of cleanups. At many locations, the parameter
or variable is still called "buf" which will become ambiguous. Also,
the "struct channel" is still defined in buffers.h.
2012-09-02 21:54:55 +02:00
Willy Tarreau
f9dabecd03 MEDIUM: connection: make use of the new polling functions
Now the connection handler, the handshake callbacks and the I/O callbacks
make use of the connection-layer polling functions to enable or disable
polling on a file descriptor.

Some changes still need to be done to avoid using the FD_WAIT_* constants.
2012-09-02 21:53:11 +02:00
Willy Tarreau
49b046dddf MAJOR: fd: replace all EV_FD_* macros with new fd_*_* inline calls
These functions have a more explicity meaning and will offer provisions
for explicit polling.

EV_FD_ISSET() has been left for now as it is still in use in checks.
2012-09-02 21:53:11 +02:00
Willy Tarreau
8b117082bc REORG: connection: replace si_data_close() with conn_data_close()
This close function only applies to connection-specific parts and
the stream-interface entry may soon disappear. Move this to the
connection instead.
2012-09-02 21:53:10 +02:00
Willy Tarreau
fd31e53139 MAJOR: remove the stream interface and task management code from sock_*
The socket data layer code must only focus on moving data between a
socket and a buffer. We need a special stream interface handler to
update the stream interface and the file descriptor status.

At the moment the code works but suffers from a race condition caused
by its API : the read/write callbacks still make use of the fd instead
of using the connection. And when a double shutdown is performed, a call
to ->write() after ->read() processed an error results in dereferencing
a NULL fdtab[]->owner. This is only a temporary issue which doesn't need
to be fixed now since this will automatically go away when the functions
change to use the connection instead.
2012-09-02 21:53:08 +02:00
Willy Tarreau
076be25ab8 CLEANUP: remove the now unused fdtab direct I/O callbacks
They were all left to NULL since last commit so we can safely remove them
all now and remove the temporary dual polling logic in pollers.
2012-09-02 21:51:29 +02:00
Willy Tarreau
8018471f44 MINOR: fd: make fdtab->owner a connection and not a stream_interface anymore
It is more convenient with a connection here and will abstract stream_interface
more easily.
2012-09-02 21:51:28 +02:00
Willy Tarreau
d2274c6536 MAJOR: connection: replace direct I/O callbacks with the connection callback
Almost all direct I/O callbacks have been changed to use the connection
callback instead. Only the TCP connection validation remains.
2012-09-02 21:51:28 +02:00
Willy Tarreau
4e6049e553 MINOR: fd: add a new I/O handler to fdtab
This one will eventually replace both cb[] handlers. At the moment it
is not used yet.
2012-09-02 21:51:27 +02:00
Willy Tarreau
505e34a36d MAJOR: get rid of fdtab[].state and use connection->flags instead
fdtab[].state was only used to know whether a connection was in progress
or an error was encountered. Instead we now use connection->flags to store
a flag for both. This way, connection management will be able to update the
connection status on I/O.
2012-09-02 21:51:26 +02:00
Willy Tarreau
db3b32610f REORG/MEDIUM: fd: remove FD_STCLOSE from struct fdtab
In an attempt to get rid of fdtab[].state, and to move the relevant
parts to the connection struct, we remove the FD_STCLOSE state which
can easily be deduced from the <owner> pointer as there is a 1:1 match.
2012-09-02 21:51:25 +02:00
Willy Tarreau
96596aeead MEDIUM: fd/si: move peeraddr from struct fdinfo to struct connection
The destination address is purely a connection thing and not an fd thing.
It's also likely that later the address will be stored into the connection
and linked to by the SI.

struct fdinfo only keeps the pointer to the port range and the local port
for now. All of this also needs to move to the connection but before this
the release of the port range must move from fd_delete() to a new function
dedicated to the connection.
2012-06-08 22:59:52 +02:00
Emeric Brun
d88fd824b7 MEDIUM: protocol: add a pointer to struct sock_ops to the listener struct
The listener struct is now aware of the socket layer to use upon accept().
At the moment, only sock_raw is supported so this patch should not change
anything.
2012-05-21 22:22:39 +02:00
Emeric Brun
21adb02d19 MINOR: stream_interface: add a pointer to the listener for TARG_TYPE_CLIENT
When the target is a client, it will be convenient to have a pointer to the
original listener so that we can retrieve some configuration information at
the stream interface level.
2012-05-21 22:22:39 +02:00
Willy Tarreau
4da69a91a0 MEDIUM: stream_interface: call si_data_close() before releasing the si
This will ensure that the data layer releases anything previously allocated.
2012-05-21 18:07:11 +02:00
Willy Tarreau
fb7508aefb REORG/MINOR: stream_interface: move si->fd to struct connection
The socket fd is used only when in socket mode and with a connection.
2012-05-21 16:47:54 +02:00
Willy Tarreau
73b013b070 MINOR: stream_interface: introduce a new "struct connection" type
We start to move everything needed to manage a connection to a special
entity "struct connection". We have the data layer operations and the
control operations there. We'll also have more info in the future such
as file descriptors and applet contexts, so that in the end it becomes
detachable from the stream interface, which will allow connections to
be reused between sessions.

For now on, we start with minimal changes.
2012-05-21 16:31:45 +02:00
Willy Tarreau
fe7f1ea68e REORG/MINOR: session: detect the TCP monitor checks at the protocol accept
It does not make sense anymore to wait for a session creation to process
a TCP monitor check which only closes the connection and returns. Better
to process this immediately after the accept() return. It also saves us
from counting a connection for monitor checks, which is much more logical.
2012-05-20 19:22:25 +02:00
Willy Tarreau
be0688c64d MEDIUM: stream_interface: remove the si->init
Calling the init() function in sess_establish was a bad idea, it is
too late to allow it to fail on lack of resource and does not help at
all. Remove it for now before it's used.
2012-05-18 15:15:26 +02:00
Willy Tarreau
7bb68abb9f OPTIM/MEDIUM: stream_interface: add a new SI_FL_NOHALF flag
This flag indicates that we're not interested in keeping half-open
connections on a stream interface. It has the benefit of allowing
the socket layer to cause an immediate write close when detecting
an incoming read close. This releases resources much faster and
saves one syscall (either a shutdown or setsockopt).

This flag is only set by HTTP on the interface going to the server
since we don't want to continue pushing data there when it has
closed.

Another benefit is that it responds with a FIN to a server's FIN
instead of responding with an RST as it used to, which is much
cleaner.

Performance gains of 7.5% have been measured on HTTP connection
rate on empty objects.
2012-05-13 14:52:22 +02:00
Willy Tarreau
b147a8382a CLEANUP: fd: remove unused cb->b pointers in the struct fdtab
These pointers were used to hold pointers to buffers in the past, but
since we introduced the stream interface, they're no longer used but
they were still sometimes set.

Removing them shrink the struct fdtab from 32 to 24 bytes on 32-bit machines,
and from 52 to 36 bytes on 64-bit machines, which is a significant saving. A
quick tests shows a steady 0.5% performance gain, probably due to the better
cache efficiency.
2012-05-13 00:35:44 +02:00
Willy Tarreau
ce887fd3b2 MEDIUM: session: add support for tunnel timeouts
Tunnel timeouts are used when TCP connections are forwarded, or
when forwarding upgraded HTTP connections (WebSocket) as well as
CONNECT requests to proxies.

This timeout allows long-lived sessions to be supported without
having to set large timeouts to normal requests.
2012-05-12 12:50:00 +02:00
Willy Tarreau
2f5b6fc090 MINOR: session: call the socket layer init function when a session establishes
In sess_establish, once we've prepared everythin, we can call the socket layer
init function. We pass an argument for targets which have one (eg: servers). At
the moment, the existing socket layers don't have init functions, but SSL will
need one.
2012-05-12 08:09:27 +02:00
Willy Tarreau
f873d754f8 CLEANUP: stream_interface: stop exporting socket layer functions
Similarly to the previous patch, we don't need the socket-layer functions
outside of stream_interface. They could even move to a file dedicated to
applets, though that does not seem particularly useful at the moment.
2012-05-11 17:47:17 +02:00
Willy Tarreau
b277d6e568 CLEANUP: sock_raw: remove last references to stream_sock
We also stop exporting all functions since they're not needed anymore outside
of sock_raw.c.
2012-05-11 17:03:42 +02:00
Willy Tarreau
1539a01645 MINOR: stream_interface: add a client target : TARG_TYPE_CLIENT
This one will be used to identify the direction the SI is being used. All
incoming connections have a target of type TARG_TYPE_CLIENT.
2012-05-11 14:47:34 +02:00