Commit Graph

1001 Commits

Author SHA1 Message Date
Willy Tarreau
5d707e1aaa [MEDIUM] stream_sock: don't close prematurely when nolinger is set
When the nolinger option is used, we must not close too fast because
some data might be left unsent. Instead we must proceed with a normal
shutdown first, then a close. Also, we want to avoid merging FIN with
the last segment if nolinger is set, because if that one gets lost,
there is no chance for it to be retransmitted.
2009-06-28 11:09:07 +02:00
Willy Tarreau
5d01a63b78 [MEDIUM] config: support loading multiple configuration files
We now support up to 10 distinct configuration files. They are
all loaded in the order defined by -f <file1> -f <file2> ...

This can be useful in order to store global, private, public,
etc... configurations in distinct files.
2009-06-23 08:17:17 +02:00
Willy Tarreau
915e1ebe63 [MEDIUM] config: split parser and checker in two functions
This is a first step towards support of multiple configuration files.
Now readcfgfile() only reads a file in memory and performs very minimal
parsing. The checks are performed afterwards.
2009-06-23 08:17:17 +02:00
Willy Tarreau
3deb3d0418 [MAJOR] session: simplify buffer error handling
Buffer errors (timeouts and I/O errors) were handled at two places,
just after the analysers and after again.

Now that the timeout detection has moved, it has become easier to
handle those errors.

This has also made it possible for the request and response analysers
to be processed together as a down-up event, and all the up-down I/O
updates to be processed afterwards, which is exactly what we're looking
for. Interestingly this has reduced the number of iterations of
(stream_int, req_resp) from (5,6,5) to (5,5,4).

Several tests have been run without any issue found.
2009-06-21 23:40:24 +02:00
Willy Tarreau
b67a9b8ca8 [MAJOR] session: only check for timeouts when they have just occurred.
It's useless to check for buffer timeouts every time we call
process_session() because we already control when we set the flag. So
let's check them at the precise moment where the flag is set.
2009-06-21 22:12:49 +02:00
Willy Tarreau
ea38854d34 [MAJOR] session: don't clear buffer status flags anymore
We want to be able to keep information about errors and timeouts
as long as possible in the buffer. Let's not clear these flags
anymore and keep them static. This does not seem to cause any
trouble, though a finer review might be wise.
2009-06-21 21:45:58 +02:00
Willy Tarreau
6704d67d65 [MEDIUM] support setting a server weight to zero
Sometimes it is useful to be able to set a server's weight to zero.
It allows the server to receive only persistent traffic but never
normal traffic.
2009-06-15 10:56:05 +02:00
Willy Tarreau
be1b91842a [MEDIUM] add support for TCP MSS adjustment for listeners
Sometimes it can be useful to limit the advertised TCP MSS on
incoming connections, for instance when requests come through
a VPN or when the system is running with jumbo frames enabled.

Passing the "mss <value>" arguments to a "bind" line will set
the value. This works under Linux >= 2.6.28, and maybe a few
earlier ones, though due to an old kernel bug most of earlier
versions will probably ignore it. It is also possible that some
other OSes will support this.
2009-06-14 18:48:19 +02:00
Willy Tarreau
d88edf2e52 [MEDIUM] implement tcp-smart-connect option at the backend
This new option enables combining of request buffer data with
the initial ACK of an outgoing TCP connection. Doing so saves
one packet per connection which is quite noticeable on workloads
mostly consisting in small objects. The option is not enabled by
default.
2009-06-14 15:48:17 +02:00
Willy Tarreau
fb14edc215 [MEDIUM] stream_sock: implement tcp-cork for use during shutdowns on Linux
Setting TCP_CORK on a socket before sending the last segment enables
automatic merging of this segment with the FIN from the shutdown()
call. Playing with TCP_CORK is not easy though as we have to track
the status of the TCP_NODELAY flag since both are mutually exclusive.
Doing so saves one more packet per session and offers about 5% more
performance.

There is no reason not to do it, so there is no associated option.
2009-06-14 15:24:37 +02:00
Willy Tarreau
9ea05a790f [MEDIUM] implement option tcp-smart-accept at the frontend
This option disables TCP quick ack upon accept. It is also
automatically enabled in HTTP mode, unless the option is
explicitly disabled with "no option tcp-smart-accept".

This saves one packet per connection which can bring reasonable
amounts of bandwidth for servers processing small requests.
2009-06-14 12:07:01 +02:00
Willy Tarreau
3842f00a19 [MINOR] config: support resetting options do default values
A new keyword prefix "default" has been introduced in order to
reset some options to their default values. This can be needed
for instance when an option is forced disabled or enabled in a
defaults section and when later sections want to use automatic
settings regardless of what was specified there. Right now it
is only supported by options, just like the "no" prefix.
2009-06-14 11:39:52 +02:00
Willy Tarreau
84b57dae4a [MINOR] config: track "no option"/"option" changes
Sometimes we would want to implement implicit default options,
but for this we need to be able to disable them, which requires
to keep track of "no option" settings. With this change, an option
explicitly disabled in a defaults section will still be seen as
explicitly disabled. There should be no regression as nothing makes
use of this yet.
2009-06-14 11:10:45 +02:00
Willy Tarreau
c6f4ce8fc4 [MEDIUM] add support for binding to source port ranges during connect
Some users are already hitting the 64k source port limit when
connecting to servers. The system usually maintains a list of
unused source ports, regardless of the source IP they're bound
to. So in order to go beyond the 64k concurrent connections, we
have to manage the source ip:port lists ourselves.

The solution consists in assigning a source port range to each
server and use a free port in that range when connecting to that
server, either for a proxied connection or for a health check.
The port must then be put back into the server's range when the
connection is closed.

This mechanism is used only when a port range is specified on
a server. It makes it possible to reach 64k connections per
server, possibly all from the same IP address. Right now it
should be more than enough even for huge deployments.
2009-06-10 12:23:32 +02:00
Willy Tarreau
f68da4603a [BUG] ensure that we correctly re-start old process in case of error
When a new process fails to grab some ports, it sends a signal to
the old process in order to release them. Then it tries to bind
again. If it still fails (eg: one of the ports is bound to a
completely different process), it must send the continue signal
to the old process so that this one re-binds to the ports. This
is correctly done, but the newly bound ports are not released
first, which sometimes causes the old process to remain running
with no port bound. The fix simply consists in unbinding all
ports before sending the signal to the old process.
2009-06-09 14:36:00 +02:00
Willy Tarreau
6bde87b757 [MINOR] startup: don't imply -q with -D
It is recommended to have -D in init scripts, but -D also implies
quiet mode, which hides warning messages, and both options are now
completely unrelated. Remove the implication to get warnings with
-D.
2009-06-09 12:09:15 +02:00
Willy Tarreau
a94f2d2ef9 [MINOR] stats/html: use the arial font before helvetica
The stats HTML output were barely readable on some browsers such as
firefox on Linux, due to the selected helvetica font which is too
small. Specifying "arial" first fixes the issue without changing the
table size. Also, the default size of 0.8em choosen to get 10px out
of 12px is wrong because it gets 9px when rounded down.
2009-05-10 20:13:32 +02:00
Willy Tarreau
a3e4942ccb [MINOR] stats: report max sessions/s and limit in HTML stats 2009-05-10 19:20:15 +02:00
Willy Tarreau
8f208ecc7b [MINOR] stats: report max sessions/s and limit in CSV export 2009-05-10 19:01:49 +02:00
Willy Tarreau
13a34bd110 [MINOR] compute the max of sessions/s on fe/be/srv
Some users want to keep the max sessions/s seen on servers, frontends
and backends for capacity planning. It's easy to grab it while the
session count is updated, so let's keep it.
2009-05-10 18:52:49 +02:00
Willy Tarreau
f7edefa413 [MINOR] implement per-logger log level limitation
Some people are using haproxy in a shared environment where the
system logger by default sends alert and emerg messages to all
consoles, which happens when all servers go down on a backend for
instance. These people can not always change the system configuration
and would like to limit the outgoing messages level in order not to
disturb the local users.

The addition of an optional 4th field on the "log" line permits
exactly this. The minimal log level ensures that all outgoing logs
will have at least this level. So the logs are not filtered out,
just set to this level.
2009-05-10 17:20:05 +02:00
Benoit
affb481f1a [MEDIUM] add support for "balance hdr(name)"
There is a patch made by me that allow for balancing on any http header
field.

[WT:
  made minor changes:
  - turned 'balance header name' into 'balance hdr(name)' to match more
    closely the ACL syntax for easier future convergence
  - renamed the proxy structure fields header_* => hh_*
  - made it possible to use the domain name reduction to any header, not
    only "host" since it makes sense to do it with other ones.
  Otherwise patch looks good.
/WT]
2009-05-10 15:50:15 +02:00
Willy Tarreau
946ba59190 [MINOR] standard: provide a new 'my_strndup' function
This function is only offered by GNU extensions and is sometimes
useful during configuration parsing.
2009-05-10 15:41:18 +02:00
Willy Tarreau
8e80e0bc4c [BUG] fix parser crash on unconditional tcp content rules
Since 1.3.17, a config containing one of the following lines would
crash the parser :

    tcp content reject
    tcp content accept

This is because a check is performed on the condition which is not
specified. The obvious fix consists in checkinf for a condition
first.
2009-05-10 12:22:39 +02:00
Willy Tarreau
c9bd0cc224 [MINOR] add options dontlog-normal and log-separate-errors
Some big traffic sites have trouble dealing with logs and tend to
disable them. Here are two new options to help cope with massive
logs.

  - dontlog-normal only disables logging for 100% successful
    connections, other ones will still be logged

  - log-separate-errors will cause non-100% successful connections
    to be logged at level "err" instead of level "info" so that a
    properly configured syslog daemon can send them to a different
    file for longer conservation.
2009-05-10 11:57:02 +02:00
Willy Tarreau
1700b9c3d5 [MINOR] don't close stdio fds twice
This minor harmless bug dates back to v1.1. When fclose() was added,
the close() calls were not removed, so the fds 0,1,2 are closed twice.
2009-05-10 10:33:29 +02:00
Willy Tarreau
d79e79b436 [BUG] O(1) pollers should check their FD before closing it
epoll, sepoll and kqueue pollers should check that their fd is not
closed before attempting to close it, otherwise we can end up with
multiple closes of fd #0 upon exit, which is harmless but dirty.
2009-05-10 10:18:54 +02:00
Willy Tarreau
01b3a53f49 [MEDIUM] convert all signals to asynchronous signals
The small list of signals currently handled by haproxy were processed
as soon as they were received. This has caused trouble with calls to
pool_gc2() occuring in the middle of libc's memory management functions
seldom causing deadlocks preventing the old process from leaving.

Now these signals use the new async signal framework and are called
asynchronously, when there is no risk of recursion. This ensures more
reliable operation, especially for sensible processing such as memory
management.
2009-05-10 09:59:50 +02:00
Willy Tarreau
332740dab2 [MEDIUM] pollers: don't wait if a signal is pending
If an asynchronous signal is received outside of the poller, we don't
want the poller to wait for a timeout to occur before processing it,
so we set its timeout to zero, just like we do with pending tasks in
the run queue.
2009-05-10 09:57:21 +02:00
Willy Tarreau
2985794ca9 [MEDIUM] call signal_process_queue from run_poll_loop
Now we check for asynchronous pending signals. There's no user yet
so this cannot cause any trouble.
2009-05-10 09:24:56 +02:00
Willy Tarreau
8f38bd0497 [MINOR] add basic signal handling functions
These functions will be used to deliver asynchronous signals in order
to make the signal handling functions more robust. The goal is to keep
the same interface to signal handlers.
2009-05-10 09:24:23 +02:00
Maik Broemme
36db02e6b1 [BUG] x-original-to: fix missing initialization to default value 2009-05-08 17:02:07 +02:00
Maik Broemme
2850cb42b6 [MINOR] add X-Original-To: header
I have attached a patch which will add on every http request a new
header 'X-Original-To'. If you have HAProxy running in transparent mode
with a big number of SQUID servers behind it, it is very nice to have
the original destination ip as a common header to make decisions based
on it.

The whole thing is configurable with a new option 'originalto'. I have
updated the sourcecode as well as the documentation. The 'haproxy-en.txt'
and 'haproxy-fr.txt' files are untouched, due to lack of my french
language knowledge. ;)

Also the patch adds this header for IPv4 only. I haven't any IPv6 test
environment running here and don't know if getsockopt() with SO_ORIGINAL_DST
will work on IPv6. If someone knows it and wants to test it I can modify
the diff. Feel free to ask me questions or things which should be changed. :)

--Maik
2009-05-01 16:22:33 +02:00
Willy Tarreau
2df8d713b3 [BUG] fix wrong pointer arithmetics in HTTP message captures
The pointer arithmetics was wrong in http_capture_bad_message().
This has no impact right now because the error only msg->som was
affected and right now it's always 0. But this was a bug waiting
for keepalive support to strike.
2009-05-01 11:33:17 +02:00
Willy Tarreau
79e9989196 [CRITICAL] uninitialized response field can sometimes cause crashes
The response message in the transaction structure was not properly
initialised at session initialisation. In theory it cannot cause any
trouble since the affected field os expected to always remain NULL.
However, in some circumstances, such as building on 64-bit platforms
with certain options, the struct session can be exactly 1024 bytes,
the same size of the requri field, so the pools are merged and the
uninitialised field may contain non-null data, causing crashes if
an invalid response is encountered and archived.

The fix simply consists in correctly initialising the missing fields.
This bug cannot affect architectures where the session pool is not
shared (32-bit architectures), but this is only by pure luck.
2009-04-27 08:11:33 +02:00
Willy Tarreau
b7f9d126e2 [MEDIUM] ensure we don't recursively call pool_gc2()
A race condition exists in the hot reconfiguration code. It is
theorically possible that the second signal is sent during a free()
in the first list, which can cause crashes or freezes (the later
have been observed). Just set up a counter to ensure we do not
recurse.
2009-04-21 02:17:45 +02:00
Willy Tarreau
3b88d441e9 [MINOR] switch all stat counters to 64-bit
The byte counters have long been 64-bit to avoid overflows. But with
several sites nowadays, we see session counters wrap around every 10-days
or so. So it was the moment to switch counters to 64-bit, including
error and warning counters which can theorically rise as fast as session
counters even if in practice there is very low risk.

The performance impact should not be noticeable since those counters are
only updated once per session. The stats output have been carefully checked
for proper types on both 32- and 64-bit platforms.
2009-04-11 20:44:08 +02:00
Willy Tarreau
5e4a6f13f4 [MINOR] fix a few remaining printf-like formats on 64-bit platforms
Mainly two sizeof() returning size_t which is not the size of an int
on 64-bit platforms.
2009-04-11 19:42:49 +02:00
Willy Tarreau
0bba5a8f6c [BUG] stats: total and lbtot are unsigned
Some big users are seeing negative numbers in the CSV stats. This patch
needs to be backported to 1.3.15 and extended to the HTML part.
2009-04-07 13:27:40 +02:00
Jeffrey 'jf' Lim
af403fc59d [CLEANUP] give a little bit more information in error message
Indicate the error is about redirection.
2009-04-03 15:01:48 +02:00
Willy Tarreau
1772ece025 [MINOR] fix several printf formats and missing arguments
Last patch revealed a number of mistakes in printf-like calls, mostly int/long
mismatches, and a few missing arguments.
2009-04-03 14:49:12 +02:00
Willy Tarreau
4076a15255 [MEDIUM] http: capture invalid requests/responses even if accepted
It's useful to be able to accept an invalid header name in a request
or response but still be able to monitor further such errors. Now,
when an invalid request/response is received and accepted due to
an "accept-invalid-http-{request|response}" option, the invalid
request will be captured for later analysis with "show errors" on
the stats socket.
2009-04-02 21:36:37 +02:00
Willy Tarreau
32a4ec0ed7 [MEDIUM] http: add options to ignore invalid header names
Sometimes it is required to let invalid requests pass because
applications sometimes take time to be fixed and other servers
do not care. Thus we provide two new options :

     option accept-invalid-http-request  (for the frontend)
     option accept-invalid-http-response (for the backend)

When those options are set, invalid requests or responses do
not cause a 403/502 error to be generated.
2009-04-02 21:36:34 +02:00
Willy Tarreau
61d188920e [MINOR] improve reporting of misplaced acl/reqxxx rules
Now we can detect improper ordering of "block", "reqxxx", "reqadd",
"redirect" and "use_backend", and warn the user accordingly.
2009-03-31 10:49:21 +02:00
Willy Tarreau
0a6d2efc45 [MINOR] stats/html: group digits by 3 to clarify numbers
Large stats numbers are more readable if there is a small space
between groups of 3 digits.
2009-03-29 14:46:01 +02:00
Willy Tarreau
e7239b5152 [MINOR] implement ulltoh() to write HTML-formatted numbers
This function sets CSS letter spacing after each 3rd digit. The page must
create a class "rls" (right letter spacing) with style "letter-spacing: 0.3em"
in order to use it.
2009-03-29 13:41:58 +02:00
Willy Tarreau
2ab85e6fee [BUG] don't set an expiration date directly from now_ms
now_ms can be zero, don't set ->analyse_exp directly from it, we
must use tick_add() instead.
2009-03-29 10:24:15 +02:00
Willy Tarreau
d06e71179a [BUG] stream_sock: check for shut{r,w} before refreshing some timeouts
Under some circumstances, it appears possible to refresh a timeout
just after a side has been shut. For instance, if poll() plans to
call both read and write, and the read side calls chk_snd() which
in turn causes a shutw to occur, then stream_sock_write could update
its write timeout. The same problem happens the other way.

The timeout checks will then not catch these cases because they
ignore timeouts in case of shut{r,w}.

This is very likely to be the major cause of the 100% CPU usages
reported by Bart Bobrowski.

The fix consists in always ensuring that a side is not shut before
updating its timeout.
2009-03-29 10:18:41 +02:00
Willy Tarreau
c6dcad6e74 [MINOR] show sess: report a lot more information about sessions
For complex troubleshooting, it's sometimes useful to be able to
completely dump all the states and flags related to a session.
Now "show sess" will report the stream interfaces and buffers
status for each session.
2009-03-29 10:09:16 +02:00
Willy Tarreau
6574519c23 [MINOR] sepoll: don't count two events on the same FD.
sepoll counts the number of speculative events it has processed in
order to remain fair with epoll_wait(). If a same FD is processed
both for read and for write, it is counted twice. Fix this.
2009-03-28 23:42:55 +01:00
Willy Tarreau
1714e0ffda [BUG] stream_sock: disable I/O on fds reporting an error
Upon read or write error, we cannot immediately close the FD because
we want to first report the error to the upper layer which will do it
itself. However, we want to prevent any further I/O from being performed
on the FD. This is especially important in case of speculative I/O where
nothing else could stop the FD from still being polled until the upper
layer takes care of the condition.
2009-03-28 23:42:30 +01:00
Willy Tarreau
1eead503da [BUG] don't call epoll_ctl() on closed sockets
Some I/O callbacks are able to close their socket themselves. We
want to check this before calling epoll_ctl(EPOLL_CTL_DEL), otherwise
we get a -1 EBADF. Right now is looks like this could not cause any
trouble but the case is racy enough to fix it.
2009-03-28 19:43:06 +01:00
Willy Tarreau
3884cbaae6 [MINOR] show sess: report number of calls to each task
For debugging purposes, it can be useful to know how many times each
task has been called.
2009-03-28 17:54:35 +01:00
Willy Tarreau
2d045597f7 [BUG] reject unix accepts when connection limit is reached
unix sockets are not attached to a real frontend, so there is
no way to disable/enable the listener depending on the global
session count. For this reason, if the global maxconn is reached
and a unix socket comes in, it will just be ignored and remain
in the poll list, which will call again indefinitely.

So we need to accept then drop incoming unix connections when
the table is full.

This should not happen with clean configurations since the global
maxconn should provide enough room for unix sockets.
2009-03-28 11:02:18 +01:00
Willy Tarreau
127334e89b [BUG] reset the stream_interface connect timeout upon connect or error
The stream_interface timeout was not reset upon a connect success or
error, leading to busy loops when requeuing tasks in the past.

Thanks to Bart Bobrowski for reporting the issue.
2009-03-28 11:01:20 +01:00
Willy Tarreau
573fd806ed [OPTIM] sepoll: do not re-check whole list upon accepts
There is already an optimisation in the speculative poller which
causes newly created FDs to be checked immediately after being
created. Unfortunately, this optimisation causes the whole spec
list to be re-checked while we're only interested in the new FDs.

Doing this minor change causes performance gains of up to 6% on
medium-sized objects with a few hundreds concurrent connections.
2009-03-22 19:25:46 +01:00
Willy Tarreau
b00f9c456c [BUG] check for global.maxconn before doing accept()
If the accept() is done before checking for global.maxconn, we can
accept too many connections and encounter a lack of file descriptors
when trying to connect to the server. This is the cause of the
"cannot get a server socket" message  encountered in debug mode
during injections with low timeouts.
2009-03-21 22:43:12 +01:00
Willy Tarreau
06bea94266 [MEDIUM] session: don't resync FSMs on non-interesting changes
While processing the session, we used to resync the FSMs when buffer
flags changed. But since BF_KERN_SPLICING and BF_READ_DONTWAIT were
introduced, sometimes we could resync after they were set, which is
not what we want. This was because there were some old checks left
which did not mask changes with BF_MASK_STATIC before checking.
2009-03-21 22:09:29 +01:00
Willy Tarreau
1b194fe03e [OPTIM] buffer: new BF_READ_DONTWAIT flag reduces EAGAIN rates
When the reader does not expect to read lots of data, it can
set BF_READ_DONTWAIT on the request buffer. When it is set,
the stream_sock_read callback will not try to perform multiple
reads, it will return after only one, and clear the flag.
That way, we can immediately return when waiting for an HTTP
request without trying to read again.

On pure request/responses schemes such as monitor-uri or
redirects, this has completely eliminated the EAGAIN occurrences
and the epoll_ctl() calls, resulting in a performance increase of
about 10%. Similar effects should be observed once we support
HTTP keep-alive since we'll immediately disable reads once we
get a full request.
2009-03-21 21:57:30 +01:00
Willy Tarreau
6f4a82c7af [OPTIM] stream_sock: don't retry to read after a large read
If we get very large data at once, it's almost certain that it's
worthless trying to read again, because we got everything we could
get.

Doing this has made all -EAGAIN disappear from splice reads. The
threshold has been put in the global tunable structures so that if
we one day want to make it accessible from user config, it will be
easy to do so.
2009-03-21 20:43:57 +01:00
Willy Tarreau
e38388033f [BUG] server check intervals must not be null
If server check interval is null, we might end up looping in
process_srv_chk().

Prevent those values from being zero and add some control in
process_srv_chk() against infinite loops.
2009-03-21 18:58:32 +01:00
Willy Tarreau
c7bdf09f9f [MINOR] stats: report number of tasks (active and running)
It may be useful for statistics purposes to report the number of
tasks.
2009-03-21 18:33:52 +01:00
Willy Tarreau
a461318f97 [MINOR] task: keep a task count and clean up task creators
It's sometimes useful at least for statistics to keep a task count.
It's easy to do by forcing the rare task creators to always use the
same functions to create/destroy a task.
2009-03-21 18:13:21 +01:00
Willy Tarreau
135a113e36 [MINOR] sched: permit a task to stay up between calls
If a task wants to stay in the run queue, it is possible. It just
needs to wake itself up. We just want to ensure that a reniced
task will be processed at the right instant.
2009-03-21 13:26:05 +01:00
Willy Tarreau
26ca34e66e [BUG] scheduler: fix improper handling of duplicates __task_queue()
The top of a duplicate tree is not where bit == -1 but at the most
negative bit. This was causing tasks to be queued in reverse order
within duplicates. While this is not dramatic, it's incorrect and
might lead to longer than expected duplicate depths under some
circumstances.
2009-03-21 12:57:06 +01:00
Willy Tarreau
218859ad6c [BUG] sched: don't leave 3 lasts tasks unprocessed when niced tasks are present
When there are niced tasks, we would only process #tasks/4 per
turn, without taking care of running #tasks when #tasks was below
4, leaving those tasks waiting for a few other tasks to push them.

The fix simply consists in checking (#tasks+3)/4.
2009-03-21 11:53:09 +01:00
Willy Tarreau
e35c94a748 [MEDIUM] scheduler: get rid of the 4 trees thanks and use ebtree v4.1
Since we're now able to search from a precise expiration date in
the timer tree using ebtree 4.1, we don't need to maintain 4 trees
anymore. Not only does this simplify the code a lot, but it also
ensures that we can always look 24 days back and ahead, which
doubles the ability of the previous scheduler. Indeed, while based
on absolute values, the timer tree is now relative to <now> as we
can always search from <now>-31 bits.

The run queue uses the exact same principle now, and is now simpler
and a bit faster to process. With these changes alone, an overall
0.5% performance gain was observed.

Tests were performed on the few wrapping cases and everything works
as expected.
2009-03-21 10:25:14 +01:00
Willy Tarreau
5804434a0f [MINOR] update ebtree to version 4.1
Ebtree version 4.1 brings lookup by ranges. This will be useful for
the scheduler.
2009-03-21 10:23:36 +01:00
Willy Tarreau
8365f9335d [CLEANUP] http: remove some commented out obsolete code in process_response 2009-03-15 23:11:49 +01:00
Willy Tarreau
86ef7dc98d [MINOR] tcp_request: let the caller take care of errors and timeouts
tcp_request is not meant to decide how an error or a timeout has to
be handled. It must just apply it rules. Now that the error checks
have been added to the session, we don't need to check them anymore
in tcp_request_inspect(), which will only consider the shutdown which
may be the result of such an error.

That makes a lot more sense since tcp_request is not really waiting
for a request.
2009-03-15 22:55:47 +01:00
Willy Tarreau
844553303d [BUG] session: errors were not reported in termination flags in TCP mode
In order to get termination flags properly updated, the session was
relying a bit too much on http_return_srv_error() which is http-centric.

A generic srv_error function was implemented in the session in order to
catch all connection abort situations. It was then noticed that a request
abort during a connection attempt was not reported, which is now fixed.

Read and write errors/timeouts were not logged either. It was necessary
to add those tests at 4 new locations.

Now it looks like everything is correctly logged. Most likely some error
checking code could now be removed from some analysers.
2009-03-15 22:34:05 +01:00
Willy Tarreau
a3780f2db8 [BUG] connect timeout is in the stream interface, not the buffer
The connect timeout was not properly detected due to the fact that
it was not correctly initialized. It must be set as the stream interface
timeout, not the buffer's write timeout.
2009-03-15 21:49:00 +01:00
Willy Tarreau
5af24efee9 [CLEANUP] config: catch and report some possibly wrong rule ordering
There are some configurations in which redirect rules are declared
after use_backend rules. We can also find "block" rules after any
of these ones. The processing sequence is :
  - block
  - redirect
  - use_backend

So as of now we try to detect wrong ordering to warn the user about
a possibly undesired behaviour.
2009-03-15 15:23:16 +01:00
Willy Tarreau
55bc0f8eb7 [MEDIUM] reverse internal proxy declaration order to match configuration
People are regularly complaining that proxies are linked in reverse
order when reading the stats. This is now definitely fixed because
the proxy order is now fixed to match configuration order.
2009-03-15 14:51:53 +01:00
Willy Tarreau
d869b24119 [MINOR] tcp-inspect: permit the use of no-delay inspection
Sometimes it may make sense to be able to immediately apply a verdict
without waiting at all. It was not possible because no inspect-delay
meant no inspection at all. This is now fixed.
2009-03-15 14:43:58 +01:00
Willy Tarreau
3cd9af228f [MINOR] cfgparse: set backends to "balance roundrobin" by default
When a backend has no LB algo specified and is not in dispatch, proxy
nor transparent mode, use "balance roundrobin" by default instead of
complaining. This will be particularly useful with stats and redirects.
2009-03-15 14:11:27 +01:00
Willy Tarreau
ff01a21ebe [MINOR] cfgparse: some cleanups in the consistency checks
Check for servers in health mode, for health mode in pure-backends.
Some code have been refactored for better organization.
2009-03-15 13:46:16 +01:00
Willy Tarreau
787bbd9b7a [MINOR] show errors: encode backslash as well as non-ascii characters
These ones were not properly encoded, causing confusion on the output.
2009-03-12 08:18:33 +01:00
Willy Tarreau
c9619468ea [BUG] stream_sock: write timeout must be updated when forwarding !
When data are forwarded between socket, we must update the output
socket's write timeout. This was forgotten, causing sessions to
unexpectedly expire during long posts.
2009-03-09 22:40:57 +01:00
Willy Tarreau
6bf1736fb1 [BUILD] proto_http did not build on gcc-2.95 (again)
move the DPRINTF below the local variable declarations.
(cherry picked from commit 7b92db4cd5)

The patch accidently got reverted.
2009-03-08 23:10:34 +01:00
Willy Tarreau
87bed62a92 [BUILD] build fixes for Solaris
One build error in stream_sock.c when MSG_NOSIGNAL is not defined,
and a warning in task.c.
2009-03-08 22:25:28 +01:00
Willy Tarreau
7c84bab879 [MEDIUM] rearrange forwarding condition to enable splice during analysis
The forwarding condition was not very clear. We would only enable
forwarding when send_max is zero, and we would only splice when no
analyser is installed. In fact we want to enable forward when there
is no analyser and we want to splice at soon as there is data to
forward, regardless of the analysers.
2009-03-08 21:38:23 +01:00
Willy Tarreau
6f0aa476bd [CLEANUP] buffer_flush() was misleading, rename it as buffer_erase 2009-03-08 20:33:29 +01:00
Willy Tarreau
ed066fae25 [CLEANUP] don't enable kernel splicing when socket is closed
Splicing will not be used when the source socket is closed. Don't
enable it uselessly.
2009-03-08 19:44:29 +01:00
Willy Tarreau
0be0ef9604 [OPTIM] do not re-check req buffer when only response has changed
In process_session(), we used to re-run through all the evaluation
loop when only the response had changed. Now we carefully check in
this order :
  - changes to the stream interfaces (only SI_ST_DIS)
  - changes to the request buffer flags
  - changes to the response buffer flags

And we branch to the appropriate section. This saves significant
CPU cycles, which is important since process_session() is one of
the major CPU eaters.

The same changes have been applied to uxst_process_session().
2009-03-08 19:20:25 +01:00
Willy Tarreau
531cf0cf8d [OPTIM] task: reduce the number of calls to task_queue()
Most of the time, task_queue() will immediately return. By extracting
the preliminary checks and putting them in an inline function, we can
significantly reduce the number of calls to the function itself, and
most of the tests can be optimized away due to the caller's context.

Another minor improvement in process_runnable_tasks() consisted in
taking benefit from the processor's branch prediction unit by making
a special case of the process_session() callback which is by far the
most common one.

All this improved performance by about 1%, mainly during the call
from process_runnable_tasks().
2009-03-08 16:35:27 +01:00
Willy Tarreau
d0a201b35c [CLEANUP] task: distinguish between clock ticks and timers
Timers are unsigned and used as tree positions. Ticks are signed and
used as absolute date within current time frame. While the two are
normally equal (except zero), it's important not to confuse them in
the code as they are not interchangeable.

We add two inline functions to turn each one into the other.

The comments have also been moved to the proper location, as it was
not easy to understand what was a tick and what was a timer unit.
2009-03-08 15:58:07 +01:00
Willy Tarreau
721fdbc381 [BUG] event_accept() must always wake the task up, even in health mode
event_accept() did not wake the task up in health mode, so that mode was
not working anymore.
2009-03-08 12:25:07 +01:00
Willy Tarreau
26c250683f [MEDIUM] minor update to the task api: let the scheduler queue itself
All the tasks callbacks had to requeue the task themselves, and update
a global timeout. This was not convenient at all. Now the API has been
simplified. The tasks callbacks only have to update their expire timer,
and return either a pointer to the task or NULL if the task has been
deleted. The scheduler will take care of requeuing the task at the
proper place in the wait queue.
2009-03-08 09:38:41 +01:00
Willy Tarreau
4136522527 [OPTIM] displace tasks in the wait queue only if absolutely needed
We don't need to remove then add tasks in the wait queue every time we
update a timeout. We only need to do that when the new timeout is earlier
than previous one. We can rely on wake_expired_tasks() to perform the
proper checks and bounce the misplaced tasks in the rare case where this
happens. The motivation behind this is that we very rarely hit timeouts,
so we save a lot of CPU cycles by moving the tasks very rarely. This now
means we can also find tasks with expiration date set to eternity in the
queue, and that is not a problem.
2009-03-08 07:59:27 +01:00
Willy Tarreau
4726f53794 [OPTIM] task: don't unlink a task from a wait queue when waking it up
In many situations, we wake a task on an I/O event, then queue it
exactly where it was. This is a real waste because we delete/insert
tasks into the wait queue for nothing. The only reason for this is
that there was only one tree node in the task struct.

By adding another tree node, we can have one tree for the timers
(wait queue) and one tree for the priority (run queue). That way,
we can have a task both in the run queue and wait queue at the
same time. The wait queue now really holds timers, which is what
it was designed for.

The net gain is at least 1 delete/insert cycle per session, and up
to 2-3 depending on the workload, since we save one cycle each time
the expiration date is not changed during a wake up.
2009-03-08 07:59:18 +01:00
Willy Tarreau
1b8ca663a4 [BUG] task: fix handling of duplicate keys
A bug was introduced with the ebtree-based scheduler. It seldom causes
some timeouts to last longer than required if they hit an expiration
date which is the same as the last queued date, is also part of a
duplicate tree without being the top of the tree. In this case, the
task will not be expired until after the duplicate tree has been
flushed.

It is easier to reproduce by setting a very short client timeout (1s)
and sending connections and waiting for them to expire with the 408
status. Then in parallel, inject at about 1kh/s. The bug causes the
connections to sometimes wait longer than 1s before timing out.

The cause was the use of eb_insert_dup() on wrong nodes, as this
function is designed to work only on the top of the dup tree. The
solution consists in updating last_timer only when its bit is -1,
and using it only if its bit is still -1 (top of a dup tree).

The fix has not reduced performance because it only fixes the case
where this bug could fire, which is extremely rare.
2009-03-08 07:57:47 +01:00
Willy Tarreau
39af0f663d [BUG] rate-limit in defaults section was ignored
Just a missing initialisation of the field when creating a proxy.
2009-03-07 11:53:44 +01:00
Willy Tarreau
2ade301505 [BUG] disable any analysers for monitoring requests
We must not parse an HTTP request on a monitoring request. In fact,
we should even create a dedicated monitoring analyser.
2009-03-06 19:16:39 +01:00
Willy Tarreau
3d8c5531d8 [OPTIM] freq_ctr: do not rotate the counters when reading
It's easier to take the counter's age into account when consulting it
than to rotate it first. It also saves some CPU cycles and avoids the
multiply for outdated counters, finally saving CPU cycles here too
when multiple operations need to read the same counter.

The freq_ctr code has also shrinked by one third consecutively to these
optimizations.
2009-03-06 14:29:25 +01:00
Willy Tarreau
ec22b2c27a [CLEANUP] remove last references to term_trace
term_trace was very useful while reworking the lower layers but has almost
completely been removed from every place it was referenced. Even the few
remaining ones were not accurate, so it's better to completely remove those
references and re-add them from scratch later if needed.
2009-03-06 13:07:40 +01:00
Willy Tarreau
9279562e2a [BUG] switch server-side stream interface to close in case of abort
In pure TCP mode, there is no response analyser to switch the server-side
stream interface from INI to CLO when the output has been closed after an
abort. This caused sessions to remain indefinitely active when they were
aborted by the client during a TCP content analysis.

The proper action is to switch the stream interface to the CLO state from
INI when we have write enable and shutdown write.
2009-03-06 12:51:23 +01:00
Willy Tarreau
79584225e5 [OPTIM] rate-limit: cleaner behaviour on low rates and reduce consumption
The rate-limit was applied to the smoothed value which does a special
case for frequencies below 2 events per period. This caused irregular
limitations when set to 1 session per second.

The proper way to handle this is to compute the number of remaining
events that can occur without reaching the limit. This is what has
been added. It also has the benefit that the frequency calculation
is now done once when entering event_accept(), before the accept()
loop, and not once per accept() loop anymore, thus saving a few CPU
cycles during very high loads.

With this fix, rate limits of 1/s are perfectly respected.
2009-03-06 09:18:27 +01:00
Willy Tarreau
efcbc6e66d [OPTIM] maintain_proxies: only wake up when the frontend will be ready
It's not needed to try to check the frontend's freq counter every
millisecond, we can precisely compute when to wake up.
2009-03-06 08:27:10 +01:00
Willy Tarreau
bb9251ed8f [BUG] typo in timeout error reporting : report *res and not *err 2009-03-06 08:05:40 +01:00
Willy Tarreau
604e83097f [BUG] interface binding: length must include the trailing zero
The interface length passed to the setsockopt(SO_BINDTODEVICE) must
include the trailing \0. Otherwise it will randomly fail.
2009-03-06 00:48:23 +01:00
Willy Tarreau
3a7d20781d [MEDIUM] implement "rate-limit sessions" for the frontend
The new "rate-limit sessions" statement sets a limit on the number of
new connections per second on the frontend. As it is extremely accurate
(about 0.1%), it is efficient at limiting resource abuse or DoS.
2009-03-05 23:48:25 +01:00
Willy Tarreau
079ff0a207 [MINOR] acl: add 2 new verbs: fe_sess_rate and be_sess_rate
These new ACLs match frontend session rate and backend session rate.
Examples are provided in the doc to explain how to use that in order
to limit abuse of service.
2009-03-05 21:34:28 +01:00
Willy Tarreau
3a8efeb46d [BUG] the "connslots" keyword was matched as "connlots"
This bug has been lying there since the patch got merged.
2009-03-05 21:31:36 +01:00
Willy Tarreau
7f062c4193 [MEDIUM] measure and report session rate on frontend, backends and servers
With this change, all frontends, backends, and servers maintain a session
counter and a timer to compute a session rate over the last second. This
value will be very useful because it varies instantly and can be used to
check thresholds. This value is also reported in the stats in a new "rate"
column.
2009-03-05 18:43:00 +01:00
Willy Tarreau
755905857a [MINOR] add curr_sec_ms and curr_sec_ms_scaled for current second.
Several algorithms will need to know the millisecond value within
the current second. Instead of doing a divide every time it is needed,
it's better to compute it when it changes, which is when now and now_ms
are recomputed.

curr_sec_ms_scaled is the same multiplied by 2^32/1000, which will be
useful to compute some ratios based on the position within last second.
2009-03-05 16:56:16 +01:00
Willy Tarreau
defc52da95 [MINOR] errors dump must use user-visible date, not internal date. 2009-03-04 20:53:44 +01:00
Willy Tarreau
74808cb907 [MEDIUM] implement error dump on unix socket with "show errors"
The new "show errors" command sent on a unix socket will dump
all captured request and response errors for all proxies. It is
also possible to bound the log to frontends and backends whose
ID is passed as an optional parameter.

The output provides information about frontend, backend, server,
session ID, source address, error type, and error position along
with a complete dump of the request or response which has caused
the error.

If a new error scratches the one currently being reported, then
the dump is aborted with a warning message, and processing goes
on to next error.
2009-03-04 15:53:18 +01:00
Willy Tarreau
f073a83b1d [MEDIUM] store a complete dump of request and response errors in proxies
Each proxy instance, either frontend or backend, now has some room
dedicated to storing a complete dated request or response in case
of parsing error. This will make it possible to consult errors in
order to find the exact cause, which is particularly important for
troubleshooting faulty applications.
2009-03-04 10:26:38 +01:00
Willy Tarreau
7552c031c0 [MINOR] ensure that http_msg_analyzer updates pointer to invalid char
If an invalid character is encountered while parsing an HTTP message, we
want to get buf->lr updated to reflect it.

Along this change, a few useless __label__ declarations have been removed
because they caused gcc to consume stack space without putting anything
there.
2009-03-01 11:10:40 +01:00
Willy Tarreau
f49d1df25c [BUG] global.tune.maxaccept must be limited even in mono-process mode
On overloaded systems, it sometimes happens that hundreds or thousands
of incoming connections are queued in the system's backlog, and all get
dequeued at once. The problem is that when haproxy processes them and
does not apply any limit, this can take some time and the internal date
does not progress, resulting in wrong timer measures for all sessions.

The most common effect of this is that all of these sessions report a
large request time (around several hundreds of ms) which is in fact
caused by the time spent accepting other connections. This might happen
on shared systems when the machine swaps.

For this reason, we finally apply a reasonable limit even in mono-process
mode. Accepting 100 connections at once is fast enough for extreme cases
and will not cause that much of a trouble when the system is saturated.
2009-03-01 08:35:41 +01:00
Willy Tarreau
368480cf45 [BUG] the "source" keyword must first clear optional settings
Problem reported by John Lauro. When "source ... usesrc ..." is
set in the defaults section, it is not possible anymore to remove
the "usesrc" part when declaring a more precise "source" in a
backend. The only workaround was to declare it by server.

We need to clear optional settings when declaring a new "source".
The problem was the same with the "interface" declaration.
2009-03-01 08:27:21 +01:00
Willy Tarreau
7b92db4cd5 [BUILD] proto_http did not build on gcc-2.95
move the DPRINTF below the local variable declarations.
2009-02-24 10:48:35 +01:00
Willy Tarreau
38c99bcb98 [BUG] fix unix socket processing of interrupted output
Unix socket processing was still quite buggy. It did not properly
handle interrupted output due to a full response buffer. The fix
mainly consists in not trying to prematurely enable write on the
response buffer, just like the standard session works. This also
gets the unix socket code closer to the standard session code
handling.
2009-02-22 15:58:45 +01:00
Willy Tarreau
fd3828e263 [BUG] fix random memory corruption using "show sess"
Commit 8a5c626e73 introduced the sessions
dump on the unix socket. This implementation is buggy because it may try
to link to the sessions list's head after the last session is removed
with a backref. Also, for the LIST_ISEMPTY test to succeed, we have to
proceed with LIST_INIT after LIST_DEL.
2009-02-22 15:17:24 +01:00
Vincenzo Farruggia
9b97cff1c2 [BUILD] Haproxy won't compile if DEBUG_FULL is defined
As subject when i try to compile haproxy with -DDEBUG_FULL it stop at
stream_sock.c file with:
gcc -Iinclude -Wall -O2 -g     -DDEBUG_FULL  -DTPROXY -DENABLE_POLL
-DENABLE_EPOLL -DENABLE_SEPOLL -DNETFILTER -DUSE_GETSOCKNAME
-DCONFIG_HAPROXY_VERSION=\"1.3.15\"
-DCONFIG_HAPROXY_DATE=\"2008/04/19\" -c -o src/stream_sock.o
src/stream_sock.c
src/stream_sock.c: In function 'stream_sock_chk_rcv':
src/stream_sock.c:905: error: 'fd' undeclared (first use in this function)
src/stream_sock.c:905: error: (Each undeclared identifier is reported only once
src/stream_sock.c:905: error: for each function it appears in.)
src/stream_sock.c:905: error: 'ob' undeclared (first use in this function)
src/stream_sock.c: In function 'stream_sock_chk_snd':
src/stream_sock.c:940: error: 'fd' undeclared (first use in this function)
src/stream_sock.c:940: error: 'ib' undeclared (first use in this function)
make: *** [src/stream_sock.o] Error 1

With this patch all build fine:
2009-02-04 22:46:19 +01:00
Krzysztof Piotr Oledzki
f39c71c981 [CRITICAL] fix server state tracking: it was O(n!) instead of O(n)
Using the wrong operator (&& instead of &) causes DOWN->UP
transition to take longer than it should and to produce a lot of
redundant logs. With typical "track" usage (1-6 tracking servers) it
shouldn't make a big difference but for heavily tracked servers
this bug leads to hang with 100% CPU usage and extremely big
log spam.
2009-02-04 22:39:03 +01:00
Willy Tarreau
0b9c02c861 [MEDIUM] implement bind-process to limit service presence by process
The "bind-process" keyword lets the admin select which instances may
run on which process (in multi-process mode). It makes it easier to
more evenly distribute the load across multiple processes by avoiding
having too many listen to the same IP:ports.
2009-02-04 22:05:05 +01:00
Willy Tarreau
c76721da57 [MEDIUM] add support for source interface binding at the server level
Add support for "interface <name>" after the "source" statement on
the server line.
2009-02-04 20:20:58 +01:00
Willy Tarreau
d53f96b3f0 [MEDIUM] add support for source interface binding
Specifying "interface <name>" after the "source" statement allows
one to bind to a specific interface for proxy<->server traffic.

This makes it possible to use multiple links to reach multiple
servers, and to force traffic to pass via an interface different
from the one the system would have chosen based on the routing
table.
2009-02-04 18:46:54 +01:00
Willy Tarreau
4e30ed73f4 [BUG] inform the user when root is expected but not set
When a plain user runs haproxy as non-root but some options require
root, let's inform him.
2009-02-04 18:02:48 +01:00
Willy Tarreau
5e6e204d1c [MINOR] add support for bind interface name
By appending "interface <name>" to a "bind" line, it is now possible
to specifically bind to a physical interface name. Note that this
currently only works on Linux and requires root privileges.
2009-02-04 17:19:29 +01:00
Willy Tarreau
0a3b9d90d3 [BUG] we must not exit if protocol binding only returns a warning
Right now, protocol binding cannot return a warning, but when this
will happen, we must not exit but just print the warning.
2009-02-04 17:05:23 +01:00
Krzysztof Piotr Oledzki
7b723efca3 [DOC] remove buggy comment for use_backend
"early blocking based on ACLs" is definitely wrong here
2009-01-27 21:30:31 +01:00
Krzysztof Piotr Oledzki
52d522b566 [BUG] Fix listen & more of 2 couples <ip>:<port>
Fix "listen www-mutualise 80.248.x.y1:80,80.248.x.y2:80,80.248.x.y3:80":

[ALERT] 309/161509 (15450) : Invalid server address: '80.248.x.y1:80,80.248.x.y2'
[ALERT] 309/161509 (15450) : Error reading configuration file : /etc/haproxy/haproxy.cfg

Bug reported by Laurent Dolosor.
2009-01-27 21:00:18 +01:00
Willy Tarreau
3ab68cf0ae [MEDIUM] splice: add the global "nosplice" option
Setting "nosplice" in the global section will disable the use of TCP
splicing (both tcpsplice and linux 2.6 splice). The same will be
achieved using the "-dS" parameter on the command line.
2009-01-25 16:03:28 +01:00
Willy Tarreau
43b78999ec [MEDIUM] move global tuning options to the global structure
The global tuning options right now only concern the polling mechanisms,
and they are not in the global struct itself. It's not very practical to
add other options so let's move them to the global struct and remove
types/polling.h which was not used for anything else.
2009-01-25 15:42:27 +01:00
Willy Tarreau
686ac828fa [OPTIM] make global.maxpipes default to global.maxconn/4 when not specified
global.maxconn/4 seems to be a good hint for global.maxpipes when that
one must be guessed. If the limit is reached, it's still possible to
set it manually in the configuration.
2009-01-25 14:06:58 +01:00
Willy Tarreau
a206fa9d5d [STATS] report pipe usage in the statistics
Pipe usage is reported in info and web stats including maxpipes, pipes_free,
and pipes_used.
2009-01-25 14:02:00 +01:00
Willy Tarreau
3eba98aa57 [MEDIUM] splice: make use of pipe pools
Using pipe pools makes pipe management a lot easier. It also allows to
remove quite a bunch of #ifdefs in areas which depended on the presence
or not of support for kernel splicing.

The buffer now holds a pointer to a pipe structure which is always NULL
except if there are still data in the pipe. When it needs to use that
pipe, it dynamically allocates it from the pipe pool. When the data is
consumed, the pipe is immediately released.

That way, there is no need anymore to care about pipe closure upon
session termination, nor about pipe creation when trying to use
splice().

Another immediate advantage of this method is that it considerably
reduces the number of pipes needed to use splice(). Tests have shown
that even with 0.2 pipe per connection, almost all sessions can use
splice(), because the same pipe may be used by several consecutive
calls to splice().
2009-01-25 13:56:13 +01:00
Willy Tarreau
982b6e37e4 [MEDIUM] introduce pipe pools
A new data type has been added : pipes. Some pre-allocated empty pipes
are maintained in a pool for users such as splice which use them a lot
for very short times.

Pipes are allocated using get_pipe() and released using put_pipe().
Pipes which are released with pending data are immediately killed.
The struct pipe is small (16 to 20 bytes) and may even be further
reduced by unifying ->data and ->next.

It would be nice to have a dedicated cleanup task which would watch
for the pipes usage and destroy a few of them from time to time.
2009-01-25 13:49:53 +01:00
Willy Tarreau
98b306be65 [MEDIUM] splice: add hints to support older buggy kernels
Kernels before 2.6.27.13 would have splice() return EAGAIN on shutdown.
By adding a few tricks, we can deal with the situation. If splice()
returns EAGAIN and the pipe is empty, then fallback to recv() which
will be able to check if it's an end of connection or not.

The advantage of this method is that it remains transparent for good
kernels since there is no reason that epoll() will return EPOLLIN
without anything to read, and even if it would happen, the recv()
overhead on this check is minimal.
2009-01-25 11:11:32 +01:00
Willy Tarreau
afb4876778 [BUG] reserve some pipes for backends with splice enabled
If splicing is enabled in a backend, we need to guess how many
pipes will be needed. We used to rely on fullconn, but this leads
to non-working splicing when fullconn is not specified. So we now
fallback to global.maxconn.
2009-01-25 10:42:05 +01:00
Willy Tarreau
5bd8c376ad [MAJOR] complete support for linux 2.6 kernel splicing
This code provides support for linux 2.6 kernel splicing. This feature
appeared in kernel 2.6.25, but initial implementations were awkward and
buggy. A kernel >= 2.6.29-rc1 is recommended, as well as some optimization
patches.

Using pipes, this code is able to pass network data directly between
sockets. The pipes are a bit annoying to manage (fd creation, release,
...) but finally work quite well.

Preliminary tests show that on high bandwidths, there's a substantial
gain (approx +50%, only +20% with kernel workarounds for corruption
bugs). With 2000 concurrent connections, with Myricom NICs, haproxy
now more easily achieves 4.5 Gbps for 1 process and 6 Gbps for two
processes buffers. 8-9 Gbps are easily reached with smaller numbers
of connections.

We also try to splice out immediately after a splice in by making
profit from the new ability for a data producer to notify the
consumer that data are available. Doing this ensures that the
data are immediately transferred between sockets without latency,
and without having to re-poll. Performance on small packets has
considerably increased due to this method.

Earlier kernels return only one TCP segment at a time in non-blocking
splice-in mode, while newer return as many segments as may fit in the
pipe. To work around this limitation without hurting more recent kernels,
we try to collect as much data as possible, but we stop when we believe
we have read 16 segments, then we forward everything at once. It also
ensures that even upon shutdown or EAGAIN the data will be forwarded.

Some tricks were necessary because the splice() syscall does not make
a difference between missing data and a pipe full, it always returns
EAGAIN. The trick consists in stop polling in case of EAGAIN and a non
empty pipe.

The receiver waits for the buffer to be empty before using the pipe.
This is in order to avoid confusion between buffer data and pipe data.
The BF_EMPTY flag now covers the pipe too.

Right now the code is disabled by default. It needs to be built with
CONFIG_HAP_LINUX_SPLICE, and the instances intented to use splice()
must have "option splice-response" (or option splice-request) enabled.

It is probably desirable to keep a pool of pre-allocated pipes to
avoid having to create them for every session. This will be worked
on later.

Preliminary tests show very good results, even with the kernel
workaround causing one memcpy(). At 3000 connections, performance
has moved from 3.2 Gbps to 4.7 Gbps.
2009-01-19 00:32:22 +01:00
Willy Tarreau
6b4aad4c1b [MEDIUM] add definitions for Linux kernel splicing
Some older libc don't define the splice() syscall, and some even
define a wrong one. For this reason, we try our best to declare
it correctly. These definitions still work with recent glibc.
2009-01-18 21:59:13 +01:00
Willy Tarreau
259de1b702 [MINOR] introduce structures required to support Linux kernel splicing
When CONFIG_HAP_LINUX_SPLICE is defined, the buffer structure will be
slightly enlarged to support information needed for kernel splicing
on Linux.

A first attempt consisted in putting this information into the stream
interface, but in the long term, it appeared really awkward. This
version puts the information into the buffer. The platform-dependant
part is conditionally added and will only enlarge the buffers when
compiled in.

One new flag has also been added to the buffers: BF_KERN_SPLICING.
It indicates that the application considers it is appropriate to
use splicing to forward remaining data.
2009-01-18 21:56:21 +01:00
Willy Tarreau
66aa61f76b [MEDIUM] splice: add configuration options and set global.maxpipes
Three new options have been added when CONFIG_HAP_LINUX_SPLICE is
set :
  - splice-request
  - splice-response
  - splice-auto

They are used to enable splicing per frontend/backend. They are also
supported in defaults sections. The "splice-auto" option is meant to
automatically turn splice on for buffers marked as fast streamers.
This should save quite a bunch of file descriptors.

It was required to add a new "options2" field to the proxy structure
because the original "options" is full.

When global.maxpipes is not set, it is automatically adjusted to
the max of the sums of all frontend's and backend's maxconns for
those which have at least one splice option enabled.
2009-01-18 21:44:07 +01:00
Willy Tarreau
3ec79b9c42 [MINOR] global.maxpipes: add the ability to reserve file descriptors for pipes
This will be needed to use linux's splice() syscall.
2009-01-18 20:39:42 +01:00
Willy Tarreau
a456f2a059 [MEDIUM] stream_sock: try to send pending data on chk_snd()
When the producer calls stream_sock_chk_snd(), we now try to send
all pending data asynchronously. If it succeeds, we don't have to
enable polling on the FD which saves about half of the calls to
epoll_wait().

In stream_sock_read(), we finally set the WAIT_ROOM flag as soon as
possible, in preparation of the splice code. We reset it when we
detect that some room has been released either in the buffer or in
the splice.
2009-01-18 19:43:47 +01:00
Willy Tarreau
d2def0fd25 [MINOR] stream_sock: fix a few wrong empty calculations 2009-01-18 17:37:33 +01:00
Willy Tarreau
9c0fe59612 [MEDIUM] stream_sock_read: call ->chk_snd whenever there are data pending
The condition to cakk ->chk_snd() in stream_sock_read() was suboptimal
because we did not call it when the socket was shut down nor when there
was an error after data were added.

Now we ensure to call is whenever there are data pending.

Also, the "full" condition was handled before calling chk_snd(), which
could cause deadlock issues if chk_snd() did consume some data.
2009-01-18 16:25:31 +01:00
Willy Tarreau
0c2fc1f39d [MEDIUM] split stream_sock_write() into callback and core functions
stream_sock_write() has been split in two parts :
  - the poll callback, intented to be called when an I/O event has
    been detected
  - the write() core function, which ought to be usable from various
    other places, possibly not meant to wake the task up.

The code has also been slightly cleaned up in the process. It's more
readable now.
2009-01-18 15:48:52 +01:00
Willy Tarreau
ac128fef73 [CLEANUP] stream_sock: move the write-nothing condition out of the loop
Some tricks to handle situations where we write nothing were in the
middle of the main loop in stream_sock_write(). This cleanup provides
better source and object code, and slightly shrinks the output code.
2009-01-09 13:05:19 +01:00
Willy Tarreau
efc612c17b [CLEANUP] replace a few occurrences of (flags & X) && !(flags & Y)
This construct collapses into ((flags & (X|Y)) == X) when X is a
single-bit flag. This provides a noticeable code shrink and the
output code results in less conditional jumps.
2009-01-09 12:18:24 +01:00
Willy Tarreau
68eac13217 [OPTIM] stream_sock: factor out the buffer full handling out of the loop
Handling the buffer full condition is not trivial and this code was
duplicated inside the loop. Move it out of the loop at a single place.
2009-01-09 11:38:52 +01:00
Willy Tarreau
03d60bbaf9 [OPTIM] buffer: replace rlim by max_len
In the buffers, the read limit used to leave some place for header
rewriting was set by a pointer to the end of the buffer. Not only
this required subtracts at every place in the code, but this will
also soon not be usable anymore when we want to support keepalive.

Let's replace this with a length limit, comparable to the buffer's
length. This has also sightly reduced the code size.
2009-01-09 11:14:39 +01:00
Willy Tarreau
af78d0fdb6 [OPTIM] stream_sock: do not ask for polling on EAGAIN if we have read
It is not always wise to return 0 in stream_sock_read() upon EAGAIN,
because if we have read enough data, we should consider that enough
and try again later without polling in between.

We still make a difference between small reads and large reads though.
Small reads still lead to polling because we're sure that there's
nothing left in the system's buffers if we read less than one MSS.
2009-01-09 10:15:03 +01:00
Willy Tarreau
0abebcc0fb [MEDIUM] i/o: rework ->to_forward and ->send_max
The way the buffers and stream interfaces handled ->to_forward was
really not handy for multiple reasons. Now we've moved its control
to the receive-side of the buffer, which is also responsible for
keeping send_max up to date. This makes more sense as it now becomes
possible to send some pre-formatted data followed by forwarded data.

The following explanation has also been added to buffer.h to clarify
the situation. Right now, tests show that the I/O is behaving extremely
well. Some work will have to be done to adapt existing splice code
though.

/* Note about the buffer structure

   The buffer contains two length indicators, one to_forward counter and one
   send_max limit. First, it must be understood that the buffer is in fact
   split in two parts :
     - the visible data (->data, for ->l bytes)
     - the invisible data, typically in kernel buffers forwarded directly from
       the source stream sock to the destination stream sock (->splice_len
       bytes). Those are used only during forward.

   In order not to mix data streams, the producer may only feed the invisible
   data with data to forward, and only when the visible buffer is empty. The
   consumer may not always be able to feed the invisible buffer due to platform
   limitations (lack of kernel support).

   Conversely, the consumer must always take data from the invisible data first
   before ever considering visible data. There is no limit to the size of data
   to consume from the invisible buffer, as platform-specific implementations
   will rarely leave enough control on this. So any byte fed into the invisible
   buffer is expected to reach the destination file descriptor, by any means.
   However, it's the consumer's responsibility to ensure that the invisible
   data has been entirely consumed before consuming visible data. This must be
   reflected by ->splice_len. This is very important as this and only this can
   ensure strict ordering of data between buffers.

   The producer is responsible for decreasing ->to_forward and increasing
   ->send_max. The ->to_forward parameter indicates how many bytes may be fed
   into either data buffer without waking the parent up. The ->send_max
   parameter says how many bytes may be read from the visible buffer. Thus it
   may never exceed ->l. This parameter is updated by any buffer_write() as
   well as any data forwarded through the visible buffer.

   The consumer is responsible for decreasing ->send_max when it sends data
   from the visible buffer, and ->splice_len when it sends data from the
   invisible buffer.

   A real-world example consists in part in an HTTP response waiting in a
   buffer to be forwarded. We know the header length (300) and the amount of
   data to forward (content-length=9000). The buffer already contains 1000
   bytes of data after the 300 bytes of headers. Thus the caller will set
   ->send_max to 300 indicating that it explicitly wants to send those data,
   and set ->to_forward to 9000 (content-length). This value must be normalised
   immediately after updating ->to_forward : since there are already 1300 bytes
   in the buffer, 300 of which are already counted in ->send_max, and that size
   is smaller than ->to_forward, we must update ->send_max to 1300 to flush the
   whole buffer, and reduce ->to_forward to 8000. After that, the producer may
   try to feed the additional data through the invisible buffer using a
   platform-specific method such as splice().
 */
2009-01-09 10:15:03 +01:00
Willy Tarreau
4d9b1dee9f [MEDIUM] stream_sock: factor out the return path in case of no-writes
Previously, we wrote nothing only if the buffer was empty. Now with
send_max, we can also write nothing because we are not allowed to send
anything due to send_max.

The code starts to look like spaghetti. It needs to be rearranged a
lot before merging the splice patches.
2009-01-09 10:15:02 +01:00
Willy Tarreau
dcef33fa9b [MINOR] add the splice_len member to the buffer struct in preparation of splice support
In preparation of splice support, let's add the splice_len member
to the buffer struct. An earlier implementation made it conditional,
which made the whole logics very complex due to a large number of
ifdefs.

Now BF_EMPTY is only set once both buf->l and buf->splice_len are
null. Splice_len is initialized to zero during buffer creation and
is currently not changed, so the whole logics remains unaffected.

When splice gets merged, splice_len will reflect the number of bytes
in flight out of the buffer but not yet sent, typically in a pipe for
the Linux case.
2009-01-09 10:15:02 +01:00
Willy Tarreau
6b66f3e4f6 [MAJOR] implement autonomous inter-socket forwarding
If an analyser sets buf->to_forward to a given value, that many
data will be forwarded between the two stream interfaces attached
to a buffer without waking the task up. The same applies once all
analysers have been released. This saves a large amount of calls
to process_session() and a number of task_dequeue/queue.
2009-01-09 10:15:02 +01:00
Willy Tarreau
3ffeba1f67 [MEDIUM] enable inter-stream_interface wakeup calls
By letting the producer tell the consumer there is data to check,
and the consumer tell the producer there is some space left again,
we can cut in half the number of session wakeups.

This is also an important starting point for future splicing support.
2008-12-28 11:09:02 +01:00
Willy Tarreau
b0ef735c71 [MINOR] add flags to indicate when a stream interface is waiting for space/data
It will soon be required to know when a stream interface is waiting for
buffer data or buffer room. Let's add two flags for that.
2008-12-28 11:08:03 +01:00
Willy Tarreau
86491c3164 [MEDIUM] indicate when we don't care about read timeout
Sometimes we don't care about a read timeout, for instance, from the
client when waiting for the server, but we still want the client to
be able to read.

Till now it was done by articially forcing the read timeout to ETERNITY.
But this will cause trouble when we want the low level stream sock to
communicate without waking the session up. So we add a BF_READ_NOEXP
flag to indicate that when the read timeout is to be set, it might
have to be set to ETERNITY.

Since BF_READ_ENA was not used, we replaced this flag.
2008-12-28 11:06:40 +01:00
Willy Tarreau
f890dc9003 [MEDIUM] add a send limit to a buffer
For keep-alive, line-mode protocols and splicing, we will need to
limit the sender to process a certain amount of bytes. The limit
is automatically set to the buffer size when analysers are detached
from the buffer.
2008-12-28 10:58:52 +01:00
Willy Tarreau
05cb29bcd0 [MINOR] transfer errors were not reported anymore in data phase 2008-12-28 10:58:25 +01:00
Willy Tarreau
4b1f85912c [BUG] "option transparent" is for backend, not frontend !
"option transparent" was set and checked on frontends only while it
is purely a backend thing as it replaces the "balance" mode. For this
reason, it did only work in "listen" sections. This change will then
not affect the rare users of this option.
2008-12-23 23:13:55 +01:00
Willy Tarreau
7cd9d94360 [BUG] check timeout must not be changed if timeout.check is not set
This causes health checks to stop after some time since the new
ticks-based scheduler because a check timeout is set to eternity.
This fix must be merged into master but not in earlier versions
as it only affects the new scheduler.
(cherry picked from commit e349eb452b655dc1adc059f05ba8b36565753393)
2008-12-23 09:58:49 +01:00
Willy Tarreau
8a5c626e73 [MINOR] stats: indicate if a task is running in "show sess"
It's sometimes useful to know that a task is currently running.
2008-12-08 00:16:21 +01:00
Willy Tarreau
922a806075 [BUG] do not dequeue the backend's pending connections on a dead server
Kai Krueger found that previous patch was incomplete, because there is
an unconditionnal call to process_srv_queue() in session_free() which
still causes a dead server to consume pending connections from the
backend.

This call was made unconditionnal so that we don't leave unserved
connections in the server queue, for instance connections coming
in with "option persist" which can bypass the server status check.
However, the server must not touch the backend's queue if it is down.

Another fear was that some connections might remain unserved when
the server is using a dynamic maxconn if the number of connections
to the backend is too low. Right now, srv_dynamic_maxconn() ensures
this cannot happen, so the call can remain conditionnal.

The fix consists in allowing a server to process it own queue whatever
its state, but not to touch the backend's queue if it is down. Its
queue should normally be empty when the server is down because it is
redistributed when the server goes down. The only remaining cases are
precisely the persistent connections with "option persist" set, coming
in after the queue has been redispatched. Those ones must still be
processed when a connection terminates.
(cherry picked from commit cd485c4480)
2008-12-07 23:51:12 +01:00
Willy Tarreau
fe651a50d6 [MINOR] redirect: in prefix mode a "/" means not to change the URI
If the prefix is set to "/", it means the user does not want to alter
the original URI, so we don't want to insert a new slash before the
original URI.

(cherry-picked from commit 02a35c74942c1bce762e996698add1270e6a5030)
2008-12-07 23:48:39 +01:00
Willy Tarreau
0140f2553c [MINOR] redirect: add support for "set-cookie" and "clear-cookie"
It is now possible to set or clear a cookie during a redirection. This
is useful for logout pages, or for protecting against some DoSes. Check
the documentation for the options supported by the "redirect" keyword.

(cherry-picked from commit 4af993822e880d8c932f4ad6920db4c9242b0981)
2008-12-07 23:46:38 +01:00
Willy Tarreau
79da4697ca [MINOR] redirect: add support for the "drop-query" option
If "drop-query" is present on a "redirect" line using the "prefix" mode,
then the returned Location header will be the request URI without the
query-string. This may be used on some login/logout pages, or when it
must be decided to redirect the user to a non-secure server.

(cherry-picked from commit f2d361ccd73aa16538ce767c766362dd8f0a88fd)
2008-12-07 23:42:01 +01:00
Willy Tarreau
106cb76c4b [BUG] critical errors should be reported even in daemon mode
Josh Goebel reported that haproxy silently dies when it fails to
chroot. In fact, it does so when in daemon mode, because daemon
mode has been disabling output for ages.

Since the code has been reworked, this could have been changed
because there is no reason for this anymore, hence this patch.
(cherry picked from commit 304d6fb00f)
(cherry picked from commit 50b7f7f12c67322c793f50a6be009f0fd0eec1bb)
2008-12-07 23:37:28 +01:00
Jeffrey 'jf' Lim
65cb2f1c85 [MINOR] cfgparse: fix off-by 2 in error message size
was just looking through the source, and noticed this... :)
(cherry picked from commit 63b76be713)
(cherry picked from commit a801db6c5ea750f93a3795dbb2e70c03e05bbef4)
2008-12-07 23:37:15 +01:00
Willy Tarreau
fd39ddaa3d [BUG] cookie capture is declared in the frontend but checked on the backend
Cookie capture would only work by pure luck on the request but did
never work on responses since only the backend was checked. The fix
consists in always checking frontend for cookie captures.
(cherry picked from commit a83c5ba9315a7c47cda2698280b7e49a9d3eb374)
2008-12-07 23:36:52 +01:00
Willy Tarreau
b099aca91a [BUG] acl-related keywords are not allowed in defaults sections
Using an ACL-related keyword in the defaults section causes a
segfault during parsing because the list headers are not initialized.
We must initialize list headers for default instance and reject
keywords relying on ACLs.
(cherry picked from commit 1c90a6ec20)
(cherry picked from commit eb8131b4e418b838b2d62d991d91d94482ba49de)
2008-12-07 23:33:43 +01:00
Willy Tarreau
da250db376 [BUG] ensure that listeners from disabled proxies are correctly unbound.
There is a problem when an instance is marked "disabled". Its ports are
still bound but will not be unbound upon termination. This causes processes
to accumulate during soft restarts, and might even cause failures to restart
new ones due to the inability to bind to the same port.

The ideal solution would be to bind all ports at the end of the configuration
parsing. An acceptable workaround is to unbind all listeners of disabled
proxies. This is what the current patch does.
(cherry picked from commit a944218e9c)
(cherry picked from commit 8cfebbb82b87345bade831920177077e7d25840a)
2008-12-07 23:33:25 +01:00
Willy Tarreau
f8fbcef83c [BUG] do not try to pause backends during reload
During a configuration reload, haproxy tried to pause all proxies.
Unfortunately, it also tried to pause backends, which would fail
and cause trouble to the new process since the port was still bound.

(backported from commit eab5c70f93)
(cherry picked from commit ac1ca38e9b07422e21b5b4778918d243768e5498)
2008-12-07 23:32:54 +01:00
Willy Tarreau
28a9e529f8 [BUG] dynamic connection throttling could return a max of zero conns
srv_dynamic_maxconn() is clearly documented as returning at least 1
possible connection under throttling. But the computation was wrong,
the minimum 1 was divided and got lost in case of very low maxconns.

Apply the MAX(1, max) before returning the result in order to ensure
that a newly appeared server will get some traffic.
(cherry picked from commit 819970098f)
2008-12-07 23:30:38 +01:00
Willy Tarreau
43662ff35d [BUG] do not release the connection slot during a retry
(forward-port of commit 8262d8bd7f)

A bug was introduced during last queue management fix. If a server
connection fails, the allocated connection slot is released, but it
will be needed again after the turn-around. This also causes more
connections than expected to go to the server because it appears to
have less connections than real.

Many thanks to Rupert Fiasco, Mark Imbriaco, Cody Fauser, Brian
Gupta and Alexander Staubo for promptly providing configuration
and diagnosis elements to help reproduce this problem easily.
2008-12-07 23:27:58 +01:00
Jeffrey 'jf' Lim
5051d7bffc [MINOR] acl: add new keyword "connslots"
I'm in the process of setting up one haproxy instance now, and I find
the following acl option useful. I'm not too sure why this option has
not been available before, but I find this useful for my own usage, so
I'm submitting this patch in the hope that it will be useful as well.

The basic idea is to be able to measure the available connection slots
still available (connection, + queue) - anything beyond that can be
redirected to a different backend. 'connslots' = number of available
server connection slots, + number of available server queue slots. In
the case where we encounter srv maxconn = 0, or srv maxqueue = 0 (in
which case we dont need to care about connslots) the value you get is
-1. Note also that this code does not take care of dynamic connections
at this point in time.

The reason why I'm using this new acl (as opposed to 'nbsrv') is that
'nbsrv' only measures servers that are actually *down*. Whereas this
other acl is more fine-grained, and looks into the number of conn
slots available as well.
2008-12-07 23:14:01 +01:00
Willy Tarreau
3dfe6cd095 [MEDIUM] add support for "show sess" in unix stats socket
It is now possible to list all known sessions by issuing "show sess"
on the unix stats socket. The format is not much evolved but it is
very useful for debugging.

The doc has been updated to reflect the new keyword.
2008-12-07 22:41:17 +01:00
Willy Tarreau
62e4f1dedd [MINOR] add back-references to sessions for later use by a dumper.
This is the first step in implementing a session dump tool.
A session dump will need restart points. It will be necessary for
it to get references to sessions which can be moved when the session
dies.

The principle is not that complex : when a session ends, it looks for
any potential back-references. If it finds any, then it moves them to
the next session in the list. The dump function will of course have
to restart from that new point.
2008-12-07 21:57:02 +01:00
Willy Tarreau
0a46489228 [MINOR] slightly rebalance stats_dump_{raw,http}
Both should process the response buffer equally. They now both
clear the hijack bit once done, and both receive a pointer to
the response buffer in their arguments.
2008-12-07 18:30:00 +01:00
Willy Tarreau
01bf8675ed [MEDIUM] reference the current hijack function in the buffer itself
Instead of calling a hard-coded function to produce data, let's
reference this function into the buffer and call it from there
when BF_HIJACK is set. This goes in the direction of more generic
session management code.
2008-12-07 18:03:29 +01:00
Willy Tarreau
b5654f6ff4 [MINOR] move the listener reference from fd to session
The listener referenced in the fd was only used to check the
listener state upon session termination. There was no guarantee
that the FD had not been reassigned by the moment it was processed,
so this was a bit racy. Having it in the session is more robust.
2008-12-07 16:45:10 +01:00
Willy Tarreau
7e5067d459 [MEDIUM] remove cli_fd, srv_fd, cli_state and srv_state from the session
Those were previously used by the unix sockets only, and could be
removed.
2008-12-07 16:27:56 +01:00
Willy Tarreau
b1356cf4e4 [MAJOR] make unix sockets work again with stats
The unix protocol handler had not been updated during the last
stream_sock changes. This has been done now. There is still a
lot of duplicated code between session.c and proto_uxst.c due
to the way the session is handled. Session.c relies on the existence
of a frontend while it does not exist here.

It is easier to see the difference between the stats part (placed
in dumpstats.c) and the unix-stream part (in proto_uxst.c).

The hijacking function still needs to be dynamically set into the
response buffer, and some cleanup is still required, then all those
changes should be forward-ported to the HTTP part. Adding support
for new keywords should not cause trouble now.
2008-12-07 16:06:43 +01:00
Willy Tarreau
ff8d42ea68 [MINOR] add an analyser state in struct session
It will be very convenient to have an analyser state in the session.
It will always be initialized to zero. The analysers can make use of
it, but must reset it to zero when they leave.
2008-12-07 14:37:09 +01:00
Willy Tarreau
7f00651419 [MEDIUM] ensure that sock->shutw() also closes read for init states
Non-connected states will never have a chance to receive a shutr event,
so we need to propagate the shutw across the stream interface.
2008-12-07 14:04:04 +01:00
Willy Tarreau
3dbc69494a [BUG] do not forward close from cons to prod with analysers
We must not forward a close from consumer to producer as long as
an analyser is present.
2008-12-07 13:05:04 +01:00
Willy Tarreau
3bc13774e1 [MINOR] pre-set analyser flags on the listener at registration time
In order to achieve more generic accept() code, we can set the request
analysers at the listener registration time. It's better than doing it
during accept(), and allows more code reuse.
2008-12-07 11:50:35 +01:00
Willy Tarreau
a11e976163 [MEDIUM] first pass of lifting to proto_uxst.c:uxst_event_accept()
The accept function must be adapted to the new framework. It is
still broken, and calling it will still result in a segfault. But
this cleanup is needed anyway.
2008-12-01 01:44:25 +01:00
Willy Tarreau
8f6457c5bb [BUG] fix forgotten server session counter
The server session counter was forgotten when the session establishes.
2008-12-01 00:08:28 +01:00
Willy Tarreau
59234e91c2 [MEDIUM] rename process_request to http_process_request
Now the function only does HTTP request and nothing else. Also pass
the request buffer to it.
2008-11-30 23:51:27 +01:00
Willy Tarreau
d34af78a34 [MEDIUM] move the HTTP request body analyser out of process_request().
A new function http_process_request_body() has been created to process
the request body. Next step is now to clean up process_request().
2008-11-30 23:36:37 +01:00
Willy Tarreau
60b85b0694 [MEDIUM] extract the HTTP tarpit code from process_request().
The tarpit is now an autonomous independant analyser.
2008-11-30 23:28:40 +01:00
Willy Tarreau
edcf6687d6 [MEDIUM] extract TCP request processing from HTTP
The TCP analyser has moved to proto_tcp.c. Breaking the function
has required finer use of the return value and adding some tests
to process_session().
2008-11-30 23:15:34 +01:00
Willy Tarreau
b025325274 [MINOR] stream_sock_data_finish() should not expose fd
stream_sock_data_finish was still using a file descriptor as only
argument, while a stream interface is preferred. This is now fixed.
2008-11-30 21:37:12 +01:00
Willy Tarreau
42ffbf248b [CLEANUP] session.c: removed some migration left-overs in sess_establish()
A few obsolete fd manipulations were left in sess_establish. Obviously
they must go away.
2008-11-30 21:13:54 +01:00
Willy Tarreau
0cac36f415 [MEDIUM] make the http server error function a pointer in the session
It was a bit awkward to have session.c call return_srv_error() for
HTTP error messages related to servers. The function has been adapted
to be passed a pointer to the faulty stream interface, and is now a
pointer in the session. It is possible that in the future, it will
become a callback in the stream interface itself.
2008-11-30 20:44:17 +01:00
Willy Tarreau
2d3d94cf23 [MINOR] replace srv_close_with_err() with http_server_error()
The new function looks like the previous one except that it operates
at the stream interface level and assumes an already closed SI.

Also remove some old unused occurrences of srv_close_with_err().
2008-11-30 20:28:57 +01:00
Willy Tarreau
dded32defa [MINOR] replace client_retnclose() with stream_int_retnclose()
This makes more sense to return a message to a stream interface
than to a session.

senddata.{c,h} have been removed.
2008-11-30 19:48:07 +01:00
Willy Tarreau
81acfab4fd [MINOR] replace the ambiguous client_return function by stream_int_return
This one applies to a stream interface, which makes more sense.
2008-11-30 19:22:53 +01:00
Willy Tarreau
a5555ec68a [MINOR] call session->do_log() for logging
In order to avoid having to call per-protocol logging function directly
from session.c, it's better to assign the logging function when the session
is created. This also eliminates a test when the function is needed, and
opens the way to more complete logging functions.
2008-11-30 19:02:32 +01:00
Willy Tarreau
55a8d0e1bb [CLEANUP] move the session-related functions to session.c
proto_http.c was not suitable for session-related processing, it was
just convenient for the tranformation.

Some more splitting must occur: process_request/response in proto_http.c
must be split again per protocol, and the caller must run a list.

Some functions should be directly attached to the session or the buffer
(eg: perform_http_redirect, return_srv_error, http_sess_log).
2008-11-30 18:47:21 +01:00
Willy Tarreau
fe3718ab79 [MAJOR] complete layer4/7 separation
All the processing has now completely been split in layers. As of
now, everything is still in process_session() which is not the right
place, but the code sequence works. Timeouts, retries, errors, all
work.

The shutdown sequence has been strictly applied: BF_SHUTR/BF_SHUTW
are only assigned by lower layers. Upper layers can only indicate
their wish to close using BF_SHUTR_NOW and BF_SHUTW_NOW.

When a shutdown is performed on a stream interface, the buffer flags
are updated accordingly and re-checked by upper layers. A lot of care
has been taken to ensure that aborts during intermediate connection
setups are correctly handled and shutdowns correctly propagated to
both buffers.

A future evolution would consist in ensuring that BF_SHUT?_NOW may
be set at any time, and applies only when the buffer is empty. This
might help with error messages, but might complicate the processing
of data remaining in buffers.

Some useless buffer flag combinations have been removed.

Stat counters are still broken (eg: per-server total number of sessions).

Error messages should be delayed to the close instant and be produced by
protocol.

Many functions must now move to proper locations.
2008-11-30 18:14:12 +01:00
Willy Tarreau
99126c35c1 [MEDIUM] make the stream interface control the SHUT{R,W} bits
It's better that the stream interface controls the BF_SHUT* bits so
that they always reflect the real state of the interface.
2008-11-27 22:32:14 +01:00
Willy Tarreau
8bfa426cad [MEDIUM] process shutw during connection attempt
It sometimes happens that a connection is aborted at the exact same moment
it establishes. We have to close the socket and not only to shut it down
for writes.

Some corner cases remain. We have to handle the shutr/shutw at the stream
interface and only report the status to the buffer, not the opposite.
2008-11-27 09:25:45 +01:00
Willy Tarreau
b38903cf3c [BUG] shutw must imply close during a connect
The sessions which were remaining stuck were being connecting to the
server while they received a shutw which caused them to partially
stop. A shutw() during a connect() must imply a close().
2008-11-23 21:33:29 +01:00
Willy Tarreau
f54f8bdd8d [MINOR] maintain a global session list in order to ease debugging
Now the global variable 'sessions' will be a dual-linked list of all
known sessions. The list element is set at the beginning of the session
so that it's easier to follow them all with gdb.
2008-11-23 19:53:55 +01:00
Willy Tarreau
0a5d5ddeb9 [MEDIUM] remove stream_sock_update_data()
Two new functions are used instead : buffer_check_{shutr,shutw}.
It is indeed more adequate to check for new closures only when the
buffer reports them.

Several remaining unclosed connections were detected after a test,
even before this patch, so a bug remains. To reproduce, try the
following during 30 seconds :

  inject30l4 -n 20000 -l -t 1000 -P 10 -o 4 -u 100 -s 100 -G 127.0.0.1:8000/
2008-11-23 19:31:35 +01:00
Willy Tarreau
74ab2ac7b0 [MEDIUM] stream_interface: added a DISconnected state between CON/EST and CLO
There were rare situations where it was not easy to detect that a failed
session attempt had occurred and needed some server cleanup. In particular,
client aborts sometimes lead to session leaks on the server side.

A new state "SI_ST_DIS" (disconnected) has been introduced for this. When
a session has been closed at a stream interface but the server cleanup has
not occurred, this state is entered instead of CLO. The cleanup is then
performed there and the state goes to CLO.

A new diagram has been added to show possible stream_interface state
transitions that can occur in a stream-sock. It makes debugging easier.
2008-11-23 17:23:07 +01:00
Willy Tarreau
4351b3a4ca [MEDIUM] continue layering cleanups.
The server sessions are now only decremented when entering SI_ST_CER
and SI_ST_CLO states. A state is clearly missing between EST and CLO,
or after CLO (eg: END), because many cleanups are performed upon CLO
and must rely on tricks to ensure being done only once.

The goal of next changes will be to improve what has been started.
Ideally, the FD should only notify the SI about the change, which
should itself only notify the session when it has some news or when
it needs help (eg: redispatch). The buffer's error processing should
not change the FD's status immediately, otherwise we risk race conds
between a pending connect and a shutw (for instance). Also, the new
connect attempt should only be made after layer 7 and all the crap
above buffers.
2008-11-12 01:51:41 +01:00
Willy Tarreau
1e62de615b [MEDIUM] add the SN_CURR_SESS flag to the session to track open sessions
It is quite hard to track when the current session has already been counted
or discounted from the server's total number of established sessions. For
this reason, we introduce a new session flag, SN_CURR_SESS, which indicates
if the current session is one of those reported by the server or not. It
simplifies session accounting and makes it far more robust. It also makes
it possible to perform a last-minute cleanup during session_free().

Right now, with this fix and a few more buffer transitions fixes, no session
were found to remain after a test.
2008-11-11 20:26:58 +01:00
Willy Tarreau
cff6411f9a [MAJOR] add a connection error state to the stream_interface
Tracking connection status changes was hard, and some code was
redundant. A new SI_ST_CER state was added to the stream interface
to indicate a past connection error, and an SI_FL_ERR flag was
added to report past I/O error. The stream_sock code does not set
the connection to SI_ST_CLO anymore in case of I/O error, it's
the upper layer which does it. This makes it possible to know
exactly when the file descriptors are allocated.

The new SI_ST_CER state permitted to split tcp_connection_status()
in two parts, one processing SI_ST_CON and the other one SI_ST_CER.
Synchronous connection errors now make use of this last state, hence
eliminating duplicate code.

Some ib<->ob copy paste errors were found and fixed, and all entities
setting SI_ST_CLO also shut the buffers down.

Some of these stream_interface specific functions and structures
have migrated to a new stream_interface.c file.

Some types of errors are still not detected by the buffers. For
instance, let's assume the following scenario in one single pass
of process_session: a connection sits in SI_ST_TAR state during
a retry. At TAR expiration, a new connection attempt is made, the
connection is obtained and srv->cur_sess is increased. Then the
buffer timeout is fires and everything is cleared, the new state
becomes SI_ST_CLO. The cleaning code checks that previous state
was either SI_ST_CON or SI_ST_EST to release the connection. But
that's wrong because last state is still SI_ST_TAR. So the
server's connection count does not get decreased.

This means that prev_state must not be used, and must be replaced
by some transition detection instead of level detection.

The following debugging line was useful to track state changes :

  fprintf(stderr, "%s:%d: cs=%d ss=%d(%d) rqf=0x%08x rpf=0x%08x\n", __FUNCTION__, __LINE__,
          s->si[0].state, s->si[1].state, s->si[1].err_type, s->req->flags, s-> rep->flags);
2008-11-03 06:26:53 +01:00
Willy Tarreau
efb453c259 [MAJOR] migrate the connection logic to stream interface
The connection setup code has been refactored in order to
make it run only on low level (stream interface). Several
complicated functions have been removed from backend.c,
and we now have sess_update_stream_int() to manage
an assigned connection, sess_prepare_conn_req() to assign a
server to a connection request, perform_http_redirect() to
redirect instead of connecting to server, and return_srv_error()
to return connection error status messages.

The stream_interface status changes are checked before adjusting
buffer flags, so that the buffers can be informed about this lower
level update.

A new connection is initiated by changing si->state from SI_ST_INI
to SI_ST_REQ.

The code seems to work but is awfully dirty. Some functions need
to be moved, and the layering is not yet quite clear.

A lot of dead old code has simply been removed.
2008-11-02 10:19:10 +01:00
Willy Tarreau
d7704b5343 [MINOR] add an expiration flag to the stream_sock_interface
This expiration flag is used to indicate that the timer has
expired without having to check it everywhere.
2008-11-02 10:19:10 +01:00
Willy Tarreau
3c6ab2e28d [MEDIUM] use buffer_check_timeouts instead of stream_sock_check_timeouts()
It's more appropriate to use buffer_check_timeouts() to check for buffer
timeouts and si->shutw/shutr to shutdown the stream interfaces.
2008-11-02 10:19:10 +01:00
Willy Tarreau
3537467679 [MEDIUM] move QUEUE and TAR timers to stream interfaces
It was not practical to have QUEUE and TAR timers in buffers, as they caused
triggering of the timeout flags. Move them to the stream interface where they
belong.
2008-11-02 10:19:09 +01:00
Willy Tarreau
a37095b96f [CLEANUP] process_session: move debug outputs out of the critical loop
The if(debug&closed) printfs have moved outside of the loop. It also
permitted to merge several of them.
2008-11-02 10:19:09 +01:00
Willy Tarreau
4ffd51a848 [MEDIUM] process_session: make use of the new buffer flags
Now we have almost two distinct parts between tcp and http.
Only the connection establishment code still requires some
resynchronization, the rest does not.
2008-11-02 10:19:09 +01:00
Willy Tarreau
9a2d15429d [MEDIUM] buffers: add BF_READ_ATTACHED and BF_ANA_TIMEOUT
Those two flags will be used to wake up analysers only when
needed.
2008-11-02 10:19:09 +01:00
Willy Tarreau
48adac5db9 [MEDIUM] stream interface: add the ->shutw method as well as in and out buffers
Those entries were really needed for cleaner and better code. Using them
has permitted to automatically close a file descriptor during a shut write,
reducing by 20% the number of calls to process_session() and derived
functions.

Process_session() does not need to know the file descriptor anymore, though
it still remains very complicated due to the special case for the connect
mode.
2008-11-02 10:19:08 +01:00
Willy Tarreau
e5ed406715 [MAJOR] make stream sockets aware of the stream interface
As of now, a stream socket does not directly wake up the task
but it does contact the stream interface which itself knows the
task. This allows us to perform a few cleanups upon errors and
shutdowns, which reduces the number of calls to data_update()
from 8 per session to 2 per session, and make all the functions
called in the process_session() loop completely swappable.

Some improvements are required. We need to provide a shutw()
function on stream interfaces so that one side which closes
its read part on an empty buffer can propagate the close to
the remote side.
2008-11-02 10:19:08 +01:00
Willy Tarreau
eabf313df2 [MINOR] change type of fdtab[]->owner to void*
The owner of an fd was initially a task but this was sometimes
casted to a (struct listener *). We'll soon need more types,
so void* is more appropriate.
2008-11-02 10:19:08 +01:00
Willy Tarreau
fdccded0e8 [MEDIUM] indicate a reason for a task wakeup
It's very frequent to require some information about the
reason why a task is running. Some flags have been added
so that a task now knows if it got woken up due to I/O
completion, timeout, etc...
2008-11-02 10:19:08 +01:00
Willy Tarreau
4df8206832 [OPTIM] reduce the number of calls to task_wakeup()
A test has shown that more than 16% of the calls to task_wakeup()
could be avoided because the task is already woken up. So make it
inline and move the test to the inline part.
2008-11-02 10:19:07 +01:00
Willy Tarreau
cb651251f9 [OPTIM] ev_sepoll: detect newly created FDs and check them once
When an accept() creates a new FD, it is already marked as set for
reads. But the task will be woken up without first checking if the
socket could be read.

The speculative I/O gives us a chance to either read the FD if there
are data pending on it, or immediately mark it for poll mode if
nothing is pending.

Simply doing this reduces the number of calls to process_session
from 6 to 5 per session, 2 to 1 calls to process_request, 10% less
calls to epoll_ctl, fd_clr, fd_set, stream_sock_data_update, 20%
less eb32_insert/eb_delete, etc... General performance increase
seems to be around 3%.
2008-11-02 10:19:07 +01:00
Willy Tarreau
21e1be8152 [MINOR] do not check for BF_SHUTR when computing write timeout
This check was useless as !BF_SHUTR is already implied by tick_isset(rex).
2008-11-02 10:19:07 +01:00
Willy Tarreau
3da77c5abd [MINOR] re-arrange buffer flags and rename some of them
The buffer flags became a big bazaar. Re-arrange them
so that their names are more explicit and so that they
are more easily readable in hex form. Some aggregates
have also been adjusted.
2008-11-02 10:19:07 +01:00
Willy Tarreau
72b179a53c [MEDIUM] reintroduce BF_HIJACK with produce_content
The stats dump are back. Even very large config files with
5000 servers work fast and well. The SN_SELF_GEN flag has
completely been removed.
2008-11-02 10:19:06 +01:00
Willy Tarreau
36e6a41bc8 [MINOR] only call flow analysers when their read side is connected.
It's useless to call flow analysers when their read side has not
seen a connection yet.
2008-11-02 10:19:06 +01:00
Willy Tarreau
2bea3a1155 [OPTIM] stream_sock_read must check for null-reads more often
With small HTTP messages, stream_sock_read() tends to wake the
task up for a message read without indicating that it may be
the last one. The reason is that level-triggered pollers generally
don't report HUP with data, but only afterwards, so stream_sock_read
has no chance to detect this condition and needs a respin.

So now we return on incomplete buffers only when the buffer is known
as a streamer, because here it generally makes sense. The net result
is that the number of calls in a single HTTP session has dropped
from 5 to 3, with one less wake up and several less calls to
stream_sock_data_update().
2008-11-02 10:19:06 +01:00
Willy Tarreau
3a16b2c9cd [MEDIUM] split stream_sock_process_data
It was a waste to constantly update the file descriptor's status
and timeouts during a flags update. So stream_sock_process_data
has been slit in two parts :
  stream_sock_data_update()  => computes updated flags
  stream_sock_data_finish()  => computes timeouts

Only the first one is called during flag updates. The second one
is only called upon completion. The number of calls to fd_set/fd_clr
has now significantly dropped.

Also, it's useless to check for errors and timeouts in the
process_session() loop, it's enough to check for them at the
beginning.
2008-11-02 10:19:06 +01:00
Willy Tarreau
f9839bdffe [MAJOR] make the client side use stream_sock_process_data()
The client side now relies on stream_sock_process_data(). One
part has not yet been re-implemented, it concerns the calls
to produce_content().

process_session() has been adjusted to correctly check for
changing bits in order not to call useless functions too many
times.

It already appears that stream_sock_process_data() should be
split so that the timeout computations are only performed at
the exit of process_session().
2008-11-02 10:19:06 +01:00
Willy Tarreau
2d2127989c [MEDIUM] stream_sock_process_data moved to stream_sock.c
The old temporary process_srv_data function moved to stream_sock.c.
2008-11-02 10:19:05 +01:00
Willy Tarreau
8a8188301b [MEDIUM] process_srv_data: ensure that we always correctly re-arm timeouts
We really want to ensure that we don't miss a timeout update and do not
update them for nothing. So the code takes care of updating the timeout
in the two following circumstances :
  - it was not set
  - some I/O has been performed

Maybe we'll be able to remove that from stream_sock_{read|write}, or
we'll find a way to ensure that we never have to re-enable this.
2008-11-02 10:19:05 +01:00
Willy Tarreau
2ac679d9aa [MEDIUM] third cleanup and optimization of process_srv_data()
Some repeated tests were factored out.
Now the code makes sense and is fully understandable.
2008-11-02 10:19:05 +01:00
Willy Tarreau
8fbd3b4ce7 [MEDIUM] second level of code cleanup for process_srv_data
Now the function is 100% server-independant. Next step will
consist in using the same function for the client side too.
2008-11-02 10:19:05 +01:00
Willy Tarreau
376580a873 [MEDIUM] massive cleanup of process_srv()
Server-specific calls were extracted and moved to the caller.
The function is now nearly server-agnostic.
2008-11-02 10:19:05 +01:00
Willy Tarreau
8b46aa01ac [OPTIM] remove useless fd_set(read) upon shutdown(write)
Those old tricks are no longer needed and are overwritten
anyway. Remove them.
2008-11-02 10:19:05 +01:00
Willy Tarreau
fa7e10251d [MAJOR] rework of the server FSM
srv_state has been removed from HTTP state machines, and states
have been split in either TCP states or analyzers. For instance,
the TARPIT state has just become a simple analyzer.

New flags have been added to the struct buffer to compensate this.
The high-level stream processors sometimes need to force a disconnection
without touching a file-descriptor (eg: report an error). But if
they touched BF_SHUTW or BF_SHUTR, the file descriptor would not
be closed. Thus, the two SHUT?_NOW flags have been added so that
an application can request a forced close which the stream interface
will be forced to obey.

During this change, a new BF_HIJACK flag was added. It will
be used for data generation, eg during a stats dump. It
prevents the producer on a buffer from sending data into it.

  BF_SHUTR_NOW  /* the producer must shut down for reads ASAP  */
  BF_SHUTW_NOW  /* the consumer must shut down for writes ASAP */
  BF_HIJACK     /* the producer is temporarily replaced        */

BF_SHUTW_NOW has precedence over BF_HIJACK. BF_HIJACK has
precedence over BF_MAY_FORWARD (so that it does not need it).

New functions buffer_shutr_now(), buffer_shutw_now(), buffer_abort()
are provided to manipulate BF_SHUT* flags.

A new type "stream_interface" has been added to describe both
sides of a buffer. A stream interface has states and error
reporting. The session now has two stream interfaces (one per
side). Each buffer has stream_interface pointers to both
consumer and producer sides.

The server-side file descriptor has moved to its stream interface,
so that even the buffer has access to it.

process_srv() has been split into three parts :
  - tcp_get_connection() obtains a connection to the server
  - tcp_connection_failed() tests if a previously attempted
    connection has succeeded or not.
  - process_srv_data() only manages the data phase, and in
    this sense should be roughly equivalent to process_cli.

Little code has been removed, and a lot of old code has been
left in comments for now.
2008-11-02 10:19:04 +01:00
Willy Tarreau
41f40ede3b [MEDIUM] make it possible for analysers to follow the whole session
Some analysers will need to remain present after connection is
established. Change the way BF_MAY_FORWARD is set to allow this.
2008-11-02 10:19:04 +01:00
Willy Tarreau
788e284d93 [BUG] fix harmless but wrong fd insertion sequence
In backend.c, we had an EV_FD_SET() called before fd_insert().
This is wrong because fd_insert updates maxfd which might be
used by some of the pollers during EV_FD_SET(), although this
is not currently the case.
2008-08-26 13:25:39 +02:00
Willy Tarreau
79f5fe82f8 [BUG] Fix empty X-Forwarded-For header name when set in defaults section
The following patch introduced a minor bug :
   [MINOR] permit renaming of x-forwarded-for header

If "option forwardfor" is declared in a defaults section, the header name
is never set and we see an empty header name before the value. Also, the
header name was not reset between two defaults sections.
2008-08-26 13:22:19 +02:00
Willy Tarreau
c52164a1a8 [BUG] process_request: HTTP body analysis must return zero if missing data
This missing return and timeout check caused an infinite loop too.
2008-08-17 19:27:11 +02:00
Willy Tarreau
2500981dc1 [BUG] process_cli/process_srv: don't call shutdown when already done
A few missing checks of BF_SHUTR and BF_SHUTW caused busy loops upon
some error paths.
2008-08-17 18:16:38 +02:00
Willy Tarreau
ffab5b4ab0 [MEDIUM] merge inspect_exp and txn->exp into request buffer
Since we may have several analysers on a buffer, it's more
convenient to have the analyser timeout attached to the
buffer itself.
2008-08-17 18:03:28 +02:00
Willy Tarreau
c7e961e5f7 [BUILD] fix warning in proto_tcp.c with gcc >= 4
signedness issues.
2008-08-17 17:13:47 +02:00
Willy Tarreau
6d2889ba3d [OPTIM] process_cli/process_srv: reduce the number of tests
We can skip a number of tests by simply checking a few flags,
it saves a few CPU cycles in the fast path.
2008-08-17 16:25:06 +02:00
Willy Tarreau
2df28e8110 [MEDIUM] session: move the analysis bit field to the buffer
It makes more sense to store the list of analysers in the buffer
than in the session since they are precisely plugged onto one
buffer.
2008-08-17 15:20:19 +02:00
Willy Tarreau
f495ddf9d4 [MINOR] ensure the termination flags are set by process_xxx
When any processing remains on a buffer, it must be up to the
processing functions to set the termination flags, because they
are the only ones who know about higher levels.
2008-08-17 14:38:41 +02:00
Willy Tarreau
507385d0e1 [MEDIUM] centralize buffer timeout checks at the top of process_session
it's more efficient and easier to check all the timeouts at once and
always rely on the buffer flags than to check them everywhere.
2008-08-17 13:04:25 +02:00
Willy Tarreau
26ed74dadc [MEDIUM] use buffer->wex instead of buffer->cex for connect timeout
It's a shame not to use buffer->wex for connection timeouts since by
definition it cannot be used till the connection is not established.
Using it instead of ->cex also makes the buffer processing more
symmetric.
2008-08-17 12:11:14 +02:00
Willy Tarreau
dafde43410 [MAJOR] process_session: rely only on buffer flags
Instead of calling all functions in a loop, process_session now
calls them according to buffer flags changes. This ensures that
we almost never call functions for nothing. The flags settings
are still quite coarse, but the number of average functions
calls per session has dropped from 31 to 18 (the calls to
process_srv dropped from 13 to 7 and the calls to process_cli
dropped from 13 to 8).

This could still be improved by memorizing which flags each
function uses, but that would add a level of complexity which
is not desirable and maybe even not worth the small gain.
2008-08-17 01:15:41 +02:00
Willy Tarreau
e393fe224b [MEDIUM] buffers: add BF_EMPTY and BF_FULL to remove dependency on req/rep->l
It is not always convenient to run checks on req->l in functions to
check if a buffer is empty or full. Now the stream_sock functions
set flags BF_EMPTY and BF_FULL according to the buffer contents. Of
course, functions which touch the buffer contents adjust the flags
too.
2008-08-16 22:18:07 +02:00
Willy Tarreau
ba392cecf9 [CLEANUP] get rid of BF_SHUT*_PENDING
BF_SHUTR_PENDING and BF_SHUTW_PENDING were poor ideas because
BF_SHUTR is the pending of BF_SHUTW_DONE and BF_SHUTW is the
pending of BF_SHUTR_DONE. Remove those two useless and confusing
"pending" versions and rename buffer_shut{r,w}_* functions.
2008-08-16 21:13:23 +02:00
Willy Tarreau
d5382b4aaa [BUG] maintain_proxies must not disable backends
maintain_proxies could disable backends (p->maxconn == 0) which is
wrong (but apparently harmless). Add a check for p->maxconn == 0.
2008-08-16 18:41:13 +02:00
Willy Tarreau
a7c52761b4 [BUG] process_response: do not touch srv_state
process_response is not allowed to touch srv_state (this is an
incident which has survived the code migration). This bug was
causing connection exhaustion on frontend due to some closed
sockets marked SV_STDATA again.
2008-08-16 18:40:18 +02:00
Willy Tarreau
d9f483646d [BUG] buffers: remove BF_MAY_CONNECT and fix forwarding issue
It wasn't really wise to separate BF_MAY_CONNECT and BF_MAY_FORWARD,
as it caused trouble in TCP mode because the connection was allowed
but not the forwarding. Remove BF_MAY_CONNECT.
2008-08-16 16:39:26 +02:00
Willy Tarreau
9a8c5de375 [BUG] process_response must not enable the read FD
Since the separation of TCP and HTTP state machines, the HTTP
code must not play anymore with the file descriptor status
without checking if they are closed. Remains of such practice
have caused busy loops under some circumstances (mainly when
client closed during headers response).
2008-08-16 16:11:07 +02:00
Willy Tarreau
7a52a5c468 [BUG] ev_sepoll: closed file descriptors could persist in the spec list
If __fd_clo() was called on a file descriptor which was previously
disabled, it was not removed from the spec list. This apparently
could not happen on previous code because the TCP states prevented
this, but now it happens regularly. The effects are spec entries
stuck populated, leading to busy loops.
2008-08-16 16:06:02 +02:00
Willy Tarreau
f853320b44 [MINOR] term_trace: add better instrumentations to trace the code
A new member has been added to the struct session. It keeps a trace
of what block of code performs a close or a shutdown on a socket, and
in what sequence. This is extremely convenient for post-mortem analysis
where flag combinations and states seem impossible. A new ABORT_NOW()
macro has also been added to make the code immediately segfault where
called.
2008-08-16 14:55:08 +02:00
Willy Tarreau
1ae3a057df [MEDIUM] remove unused references to {CL|SV}_STSHUT*
All references to CL_STSHUT* and SV_STSHUT* were removed where
possible. Some of them could not be removed because they are
still in use by the unix sockets.

A bug remains at this stage. Injecting with a very short timeout
sometimes leads to a client in close state and a server in data
state with all buffer flags indicating a shutdown but the server
fd still enable, thus causing a busy loop.
2008-08-16 10:56:30 +02:00
Willy Tarreau
461f662846 [MAJOR] clearly separate HTTP response processing from TCP server state
The HTTP response is now processed in its own function, regardless of
the TCP state. All FSMs have become fairly simpler and must still be
improved by removing useless CL_STSHUT* and SV_STSHUT* (still used by
proto_uxst). The number of calls to process_* is still huge though.

Next steps consist in :
  - removing useless assignments of CL_STSHUT* and SV_STSHUT*
  - add a BF_EMPTY flag to buffers to indicate an empty buffer
  - returning smarter values in process_* so that each callee
    may explicitly indicate whom needs to be called after it.
  - unify read and write timeouts for a same side. The way it
    is now is too complicated and error-prone
  - auditing code for regression testing

We're close to getting something which works fairly better now.
2008-08-15 23:43:19 +02:00
Willy Tarreau
cebf57e0bf [MAJOR] better separation of response processing and server state
TCP timeouts are not managed anymore by the response FSM. Warning,
the FORCE_CLOSE state does not work anymore for now. All remaining
bugs causing stale connections have been swept.
2008-08-15 18:16:37 +02:00
Willy Tarreau
f5483bf639 [MAJOR] get rid of the SV_STHEADERS state
The HTTP response code has been moved to a specific function
called "process_response" and the SV_STHEADERS state has been
removed and replaced with the flag AN_RTR_HTTP_HDR.
2008-08-14 18:35:40 +02:00
Willy Tarreau
e46ab5524f [BUG] fix recently introduced loop when client closes early
Due to a recent change in the FSMs, if the client closes with buffer
full, then the server loops waiting for headers. We can safely ignore
this case since the server FSM will have to be reworked too. Let's
fix the root cause for now.
2008-08-14 00:18:39 +02:00
Willy Tarreau
c65a3ba3d4 [MAJOR] completely separate HTTP and TCP states on the request path
For the first time, HTTP and TCP are not merged anymore. All request
processing has moved to process_request while the TCP processing of
the frontend remains in process_cli. The code is a lot cleaner,
simpler, smaller (1%) and slightly faster (1% too).

Right now, the HTTP state machine cannot easily command the TCP
state machine, but it does not cause that many difficulties.

The response processing has not yet been extracted, and the unix-stream
state machines have to be broken down that way too.

The CL_STDATA, CL_STSHUTR and CL_STSHUTW states still exist and are
exactly the sames. They will have to be all merged into CL_STDATA
once the work has stabilized. It is also possible that this single
state will disappear in favor of just buffer flags.
2008-08-14 00:18:39 +02:00
Willy Tarreau
7f875f6c8f [MEDIUM] simplify and centralize request timeout cancellation and request forwarding
Instead of playing with req->flags and request timeout everywhere,
tweak them only at precise locations.
2008-08-14 00:18:38 +02:00
Willy Tarreau
adfb8569f7 [MAJOR] get rid of SV_STANALYZE (step 2)
The SV_STANALYZE state was installed on the server side but was really
meant to be processed with the rest of the request on the client side.
It suffered from several issues, mostly related to the way timeouts were
handled while waiting for data.

All known issues related to timeouts during a request - and specifically
a request involving body processing - have been raised and fixed. At this
point, the code is a bit dirty but works fine, so next steps might be
cleanups with an ability to come back to the current state in case of
trouble.
2008-08-14 00:18:38 +02:00
Willy Tarreau
67f0eead22 [MAJOR] kill CL_STINSPECT and CL_STHEADERS (step 1)
This is a first attempt at separating data processing from the
TCP state machine. Those two states have been replaced with flags
in the session indicating what needs to be analyzed. The corresponding
code is still called before and in lieu of TCP states.

Next change should get rid of the specific SV_STANALYZE which is in
fact a client state.

Then next change should consist in making it possible to analyze
TCP contents while being in CL_STDATA (or CL_STSHUT*).
2008-08-14 00:18:38 +02:00
Aleksandar Lazic
697bbb0106 [PATCH] appsessions: cleanup DEBUG_HASH and initialize request_counter
This patch cleanup the -DDEBUG=DEBUG_HASH output setting and initialize
the request_counter for the appsessions.
2008-08-13 23:43:26 +02:00
Willy Tarreau
9f1f24bb7f [BUG] client timeout incorrectly rearmed while waiting for server
Client timeout could be refreshed in stream_sock_*, but this is
undesired when the timeout is already set to eternity. The effect
is that a session could still be aborted if client timeout was
smaller than server timeout. A second effect is that sessions
expired on the server side would expire with "cD" flags.

The fix consists in not updating it if it was not previously set.
A cleaner method might consist in updating the buffer timeout. This
is probably what will be done later when the state machines only
deal with the buffers.
2008-08-11 11:34:18 +02:00
Willy Tarreau
ce09c52187 [BUG] server timeout was not considered in some circumstances
Due to a copy-paste typo, the client timeout was refreshed instead
of the server's when waiting for server response. This means that
the server's timeout remained eternity.
2008-08-11 11:34:16 +02:00
Willy Tarreau
fb0528bd56 [BUG] fix segfault with url_param + check_post
If an HTTP/0.9-like POST request is sent to haproxy while
configured with url_param + check_post, it will crash. The
reason is that the total buffer length was computed based
on req->total (which equals the number of bytes read) and
not req->l (number of bytes in the buffer), thus leading
to wrong size calculations when calling memchr().

The affected code does not look like it could have been
exploited to run arbitrary code, only reads were performed
at wrong locations.
2008-08-11 11:34:01 +02:00
Willy Tarreau
718f0ef129 [MEDIUM] process_cli: don't rely at all on server state
A new buffer flag BF_MAY_FORWARD has been added so that the client
FSM can check whether it is allowed to forward the response to the
client. The client FSM does not have to monitor the server state
anymore.
2008-08-10 16:21:32 +02:00
Willy Tarreau
dc0a6a0dea [MEDIUM] process_srv: don't rely at all on client state
A new buffer flag BF_MAY_CONNECT has been added so that the server
FSM can check whether it is allowed to establish a connection or
not. That way, the client FSM only has to move this flag and the
server side does not need to monitor client state anymore.
2008-08-03 22:47:10 +02:00
Willy Tarreau
6468d924ea [MEDIUM] process_srv: rely on buffer flags for client shutdown
The open/close nature of each half of the client side is known
to the buffer, so let the server state machine rely on this
instead of checking the client state for CL_STSHUT* or
CL_STCLOSE.
2008-08-03 20:48:51 +02:00
Willy Tarreau
89edf5e629 [MEDIUM] buffers: ensure buffer_shut* are properly called upon shutdowns
It is important that buffer states reflect the state of both sides so
that we can remove client and server state inter-dependencies.
2008-08-03 20:48:50 +02:00
Willy Tarreau
48d63db7a8 [MEDIUM] memory: update pool_free2() to support NULL pointers
In order to make pool usage more convenient, let pool_free2()
support NULL pointers by doing nothing, just like the standard
free(3) call does.

The various call places have been updated to remove the now
useless checks.
2008-08-03 20:48:50 +02:00
Willy Tarreau
a534fea478 [CLEANUP] remove 65 useless NULL checks before free
C specification clearly states that free(NULL) is a no-op.
So remove useless checks before calling free.
2008-08-03 20:48:50 +02:00
Ross West
af72a1d8ec [MINOR] permit renaming of x-forwarded-for header
Because I needed it in my situation - here's a quick patch to
allow changing of the "x-forwarded-for" header by using a suboption to
"option forwardfor".

Suboption "header XYZ" will set the header from "x-forwarded-for" to "XYZ".

Default is still "x-forwarded-for" if the header value isn't defined.
Also the suboption 'except a.b.c.d/z' still works on the same line.

So it's now: option forwardfor [except a.b.c.d[/z]] [header XYZ]
2008-08-03 10:51:45 +02:00
Willy Tarreau
dd64f8d394 [MEDIUM] acl: when possible, report the name and requirements of ACLs in warnings
When an ACL is referenced at a wrong place (eg: response during request, layer7
during layer4), try to indicate precisely the name and requirements of this ACL.

Only the first faulty ACL is returned. A small change consisting in iterating
that way may improve reports :
   cap = ACL_USE_any_unexpected
   while ((acl=cond_find_require(cond, cap))) {
     warning()
     cap &= ~acl->requires;
   }

This will report the first ACL of each unsupported type. But doing so will
mangle the error reporting a lot, so we need to rework error reports first.
2008-08-03 09:41:05 +02:00
Willy Tarreau
0ceba5af74 [MEDIUM] acl: set types on all currently known ACL verbs
All currently known ACL verbs have been assigned a type which makes
it possible to detect inconsistencies, such as response values used
in request rules.
2008-07-25 19:31:03 +02:00
Willy Tarreau
a9802633d8 [MEDIUM] acl: enforce ACL type checking
ACL now hold information on the availability of the data they rely
on. They can indicate which parts of the requests/responses they
require, and the rules parser may now report inconsistencies.

As an example, switching rules are now checked for response-specific
ACLs, though those are not still set. A warning is reported in case
of mismatch. ACLs keyword restrictions will now have to be specifically
set wherever a better control is expected.

The line number where an ACL condition is declared has been added to
the conditions in order to be able to report the faulty line number
during post-loading checks.
2008-07-25 19:13:19 +02:00
Willy Tarreau
b6fb420c7e [MINOR] acl: add the "wait_end" acl verb
The new "wait_end" acl delays evaluation of the rule (and the next ones)
to the end of the analysis period. This is intented to be used with TCP
content analysis. A rule referencing such an ACL will not match until
the delay is over. An equivalent default ACL "WAIT_END" has been created.
2008-07-20 11:18:28 +02:00
Willy Tarreau
58393e103f [MEDIUM] acl: get rid of dummy values in always_true/always_false
make use of last change in order to get rid of dummy values in
always_true/always_false.
2008-07-20 10:39:22 +02:00
Willy Tarreau
a79534fce1 [MEDIUM] acl: permit fetch() functions to set the result themselves
For protocol analysis, it's not always convenient to have to run through
a fetch then a match against dummy values. It's easier to let the fetch()
function set the result itself. This obviously works only for boolean
values.
2008-07-20 10:17:20 +02:00
Willy Tarreau
c6317703ce [MINOR] acl: add REQ_CONTENT to the list of default acls
With content inspection, checking the presence of data in the
request buffer is very important. It's getting boring to always
add such an ACL, so let's add it by default.
2008-07-20 09:29:50 +02:00
Willy Tarreau
177e2b0127 [CLEANUP] remove dependency on obsolete INTBITS macro
The INTBITS macro was found to be already defined on some platforms,
and to equal 32 (while INTBITS was 5 here). Due to pure luck, there
was no declaration conflict, but it's nonetheless a problem to fix.

Looking at the code showed that this macro was only used for left
shifts and nothing else anymore. So the replacement is obvious. The
new macro, BITS_PER_INT is more obviously correct.
2008-07-16 10:30:44 +02:00
Willy Tarreau
ec6c5df018 [CLEANUP] remove many #include <types/xxx> from C files
It should be stated as a rule that a C file should never
include types/xxx.h when proto/xxx.h exists, as it gives
less exposure to declaration conflicts (one of which was
caught and fixed here) and it complicates the file headers
for nothing.

Only types/global.h, types/capture.h and types/polling.h
have been found to be valid includes from C files.
2008-07-16 10:30:42 +02:00
Willy Tarreau
284648e079 [CLEANUP] remove unused include/types/client.h
This file is not used anymore.
2008-07-16 10:30:40 +02:00
Willy Tarreau
655e26af24 [MINOR] acl: add req_ssl_ver in TCP, to match an SSL version
This new keyword matches an dotted version mapped into an integer.
It permits to match an SSL message protocol version just as if it
was an integer, so that it is easy to map ranges, like this :

	acl obsolete_ssl  req_ssl_ver   lt 3
	acl correct_ssl   req_ssl_ver   3.0-3.1
	acl invalid_ssl   req_ssl_ver   gt 3.1

Both SSLv2 hello messages and SSLv3 messages are supported. The
test tries to be strict enough to avoid being easily fooled. In
particular, it waits for as many bytes as announced in the message
header if this header looks valid (bound to the buffer size).

The same decoder will be usable with minor changes to check the
response messages.
2008-07-16 10:30:06 +02:00
Willy Tarreau
4a26d2f2fa [MINOR] acl: add a new parsing function: parse_dotted_ver
This new function supports one major and one minor and makes an int of them.
It is very convenient to compare versions (eg: SSL) just as if they were plain
integers, as the comparison functions will still be based on integers.
2008-07-16 10:29:51 +02:00
Willy Tarreau
b686644ad8 [MAJOR] implement tcp request content inspection
Some people need to inspect contents of TCP requests before
deciding to forward a connection or not. A future extension
of this demand might consist in selecting a server farm
depending on the protocol detected in the request.

For this reason, a new state CL_STINSPECT has been added on
the client side. It is immediately entered upon accept() if
the statement "tcp-request inspect-delay <xxx>" is found in
the frontend configuration. Haproxy will then wait up to
this amount of time trying to find a matching ACL, and will
either accept or reject the connection depending on the
"tcp-request content <action> {if|unless}" rules, where
<action> is either "accept" or "reject".

Note that it only waits that long if no definitive verdict
can be found earlier. That generally implies calling a fetch()
function which does not have enough information to decode
some contents, or a match() function which only finds the
beginning of what it's looking for.

It is only at the ACL level that partial data may be processed
as such, because we need to distinguish between MISS and FAIL
*before* applying the term negation.

Thus it is enough to add "| ACL_PARTIAL" to the last argument
when calling acl_exec_cond() to indicate that we expect
ACL_PAT_MISS to be returned if some data is missing (for
fetch() or match()). This is the only case we may return
this value. For this reason, the ACL check in process_cli()
has become a lot simpler.

A new ACL "req_len" of type "int" has been added. Right now
it is already possible to drop requests which talk too early
(eg: for SMTP) or which don't talk at all (eg: HTTP/SSL).

Also, the acl fetch() functions have been extended in order
to permit reporting of missing data in case of fetch failure,
using the ACL_TEST_F_MAY_CHANGE flag.

The default behaviour is unchanged, and if no rule matches,
the request is accepted.

As a side effect, all layer 7 fetching functions have been
cleaned up so that they now check for the validity of the
layer 7 pointer before dereferencing it.
2008-07-16 10:29:07 +02:00
Willy Tarreau
9de1bbd004 [MEDIUM] modularize the "timeout" keyword configuration parser
The "timeout" keyword already relied on an external parser, let's
make use of the new keyword registration mechanism.
2008-07-09 20:34:27 +02:00
Willy Tarreau
39f23b6c7e [MINOR] cfgparse: add support for warnings in external functions
Some parsers will need to report warnings in some cases. Let's
use positive values for that.
2008-07-09 20:23:15 +02:00
Willy Tarreau
10522fd113 [MEDIUM] modularize the global "stats" keyword configuration parser
The "stats" keyword already relied on an external parser, let's
make use of the new keyword registration mechanism.
2008-07-09 20:12:41 +02:00
Willy Tarreau
5b2c33683b [MEDIUM] add support for configuration keyword registration
Any module which needs configuration keywords may now dynamically
register a keyword in a given section, and associate it with a
configuration parsing function using cfg_register_keywords() from
a constructor function. This makes the configuration parser more
modular because it is not required anymore to touch cfg_parse.c.
Example :

static int parse_global_blah(char **args, int section_type, struct proxy *curpx,
                             struct proxy *defpx, char *err, int errlen)
{
	printf("parsing blah in global section\n");
	return 0;
}

static int parse_listen_blah(char **args, int section_type, struct proxy *curpx,
		      struct proxy *defpx, char *err, int errlen)
{
	printf("parsing blah in listen section\n");
	if (*args[1]) {
		snprintf(err, errlen, "missing arg for listen_blah!!!");
		return -1;
	}
	return 0;
}

static struct cfg_kw_list cfg_kws = {{ },{
	{ CFG_GLOBAL, "blah", parse_global_blah },
	{ CFG_LISTEN, "blah", parse_listen_blah },
	{ 0, NULL, NULL },
}};

__attribute__((constructor))
static void __module_init(void)
{
	cfg_register_keywords(&cfg_kws);
}
2008-07-09 19:44:58 +02:00
Willy Tarreau
11382813a1 [TESTS] added test-acl.cfg to test some ACL combinations
various rules constructions can be tested with this test case.
2008-07-09 16:18:21 +02:00
Willy Tarreau
a8cfa34a9c [BUG] use_backend would not correctly consider "unless"
A copy-paste typo made use_backend not correctly consider the "unless"
case, depending on the previous "block" rule.
2008-07-09 11:23:31 +02:00
Willy Tarreau
0c303eec87 [MAJOR] convert all expiration timers from timeval to ticks
This is the first attempt at moving all internal parts from
using struct timeval to integer ticks. Those provides simpler
and faster code due to simplified operations, and this change
also saved about 64 bytes per session.

A new header file has been added : include/common/ticks.h.

It is possible that some functions should finally not be inlined
because they're used quite a lot (eg: tick_first, tick_add_ifset
and tick_is_expired). More measurements are required in order to
decide whether this is interesting or not.

Some function and variable names are still subject to change for
a better overall logics.
2008-07-07 00:09:58 +02:00
Willy Tarreau
ce44f12c1e [OPTIM] task_queue: assume most consecutive timers are equal
When queuing a timer, it's very likely that an expiration date is
equal to that of the previously queued timer, due to time rounding
to the millisecond. Optimizing for this case provides a noticeable
1% performance boost.
2008-07-05 18:16:19 +02:00
Willy Tarreau
91e99931b7 [MEDIUM] introduce task->nice and boot access to statistics
The run queue scheduler now considers task->nice to queue a task and
to pick a task out of the queue. This makes it possible to boost the
access to statistics (both via HTTP and UNIX socket). The UNIX socket
receives twice as much a boost as the HTTP socket because it is more
sensible.
2008-06-30 07:51:00 +02:00
Willy Tarreau
58b458d8ba [MAJOR] use an ebtree instead of a list for the run queue
We now insert tasks in a certain sequence in the run queue.
The sorting key currently is the arrival order. It will now
be possible to apply a "nice" value to any task so that it
goes forwards or backwards in the run queue.

The calls to wake_expired_tasks() and maintain_proxies()
have been moved to the main run_poll_loop(), because they
had nothing to do in process_runnable_tasks().

The task_wakeup() function is not inlined anymore, as it was
only used at one place.

The qlist member of the task structure has been removed now.
The run_queue list has been replaced for an integer indicating
the number of tasks in the run queue.
2008-06-29 22:40:23 +02:00
Willy Tarreau
af754fc88f [OPTIM] shrink wake_expired_tasks() by using task_wakeup()
It's not worth duplicating task_wakeup() in wake_expired_tasks().
Calling it reduces code size and slightly improves performance.
2008-06-29 19:25:52 +02:00
Willy Tarreau
69e989ccbc [BUILD] change declaration of base64tab to fix build with Intel C++
I got a report that Intel C++ complains about the size of the
base64tab in base64.c. Setting it to 65 chars to allow for the
trailing zero fixes the problem.
2008-06-29 17:17:38 +02:00
Willy Tarreau
28c41a4041 [MEDIUM] rework the wait queue mechanism
The wait queues now rely on 4 trees for past, present and future
timers. The computations are cleaner and more reliable. The
wake_expired_tasks function has become simpler. Also, a bug
previously introduced in task_queue() by the first introduction
of eb_trees has been fixed (the eb->key was never updated).
2008-06-29 17:00:59 +02:00
Willy Tarreau
284c7b3195 [BUG] disable buffer read timeout when reading stats
The buffer read timeouts were not reset when stats were produced. This
caused unneeded wakeups.
2008-06-29 16:38:43 +02:00
Willy Tarreau
e6313a37d6 [MINOR] introduce now_ms, the current date in milliseconds
This new time value will be used to compute timeouts and wait queue
positions. The operation is made once for all when time is retrieved.
A future improvement might consist in having it in ticks of 1/1024
second and to convert all timeouts into ticks.
2008-06-29 13:47:25 +02:00
Willy Tarreau
e62bdd4026 [BUG] wqueue: perform proper timeout comparisons with wrapping values
With wrapping keys, we cannot simply do "if (key > now)", but
we must at least do "if ((signed)(key-now) > 0)".
2008-06-29 10:32:02 +02:00
Willy Tarreau
accc4e1e86 [BUG] we could segfault during exit while freeing uri_auths
The following config makes haproxy segfault on exit :

defaults
	mode	http
	balance	roundrobin

listen  no-stats
        bind       :8001

listen  stats
        bind       :8002
	stats      uri /stats

The simple fix is to ensure that p->uri_auth is not NULL
before dereferencing it.
2008-06-24 11:14:45 +02:00
Willy Tarreau
9789f7bd68 [MAJOR] replace ultree with ebtree in wait-queues
The ultree code has been removed in favor of a simpler and
cleaner ebtree implementation. The eternity queue does not
need to exist anymore, and the pool_tree64 has been removed.

The ebtree node is stored in the task itself. The qlist list
header is still used by the run-queue, but will be able to
disappear once the run-queue uses ebtree too.
2008-06-24 08:17:16 +02:00
Willy Tarreau
b0b37bcd65 [MEDIUM] further improve monotonic clock by check forward jumps
The first implementation of the monotonic clock did not verify
forward jumps. The consequence is that a fast changing time may
expire a lot of tasks. While it does seem minor, in fact it is
problematic because most machines which boot with a wrong date
are in the past and suddenly see their time jump by several
years in the future.

The solution is to check if we spent more apparent time in
a poller than allowed (with a margin applied). The margin
is currently set to 1000 ms. It should be large enough for
any poll() to complete.

Tests with randomly jumping clock show that the result is quite
accurate (error less than 1 second at every change of more than
one second).
2008-06-23 14:00:57 +02:00
Willy Tarreau
b7f694f20e [MEDIUM] implement a monotonic internal clock
If the system date is set backwards while haproxy is running,
some scheduled events are delayed by the amount of time the
clock went backwards. This is particularly problematic on
systems where the date is set at boot, because it seldom
happens that health-checks do not get sent for a few hours.

Before switching to use clock_gettime() on systems which
provide it, we can at least ensure that the clock is not
going backwards and maintain two clocks : the "date" which
represents what the user wants to see (mostly for logs),
and an internal date stored in "now", used for scheduled
events.
2008-06-22 17:18:02 +02:00
Willy Tarreau
7c669d7e0f [BUG] fix the dequeuing logic to ensure that all requests get served
The dequeuing logic was completely wrong. First, a task was assigned
to all servers to process the queue, but this task was never scheduled
and was only woken up on session free. Second, there was no reservation
of server entries when a task was assigned a server. This means that
as long as the task was not connected to the server, its presence was
not accounted for. This was causing trouble when detecting whether or
not a server had reached maxconn. Third, during a redispatch, a session
could lose its place at the server's and get blocked because another
session at the same moment would have stolen the entry. Fourth, the
redispatch option did not work when maxqueue was reached for a server,
and it was not possible to do so without indefinitely hanging a session.

The root cause of all those problems was the lack of pre-reservation of
connections at the server's, and the lack of tracking of servers during
a redispatch. Everything relied on combinations of flags which could
appear similarly in quite distinct situations.

This patch is a major rework but there was no other solution, as the
internal logic was deeply flawed. The resulting code is cleaner, more
understandable, uses less magics and is overall more robust.

As an added bonus, "option redispatch" now works when maxqueue has
been reached on a server.
2008-06-20 15:08:06 +02:00
Willy Tarreau
7a63abd84f [BUG] log: reported queue position was offed-by-one
The reported queue position in the logs was 0 for the first pending request
in the queue, which is wrong because it means that one request will have to
be completed before the queued one may execute. It caused the undesired side
effect that 0/0 was reported when either 0 or 1 request was pending in the
queue. Thus, we have to increment the queue size before reporting the value.
2008-06-20 15:08:04 +02:00
Willy Tarreau
7008987813 [BUG] queue management: wake oldest request in queues
When a server terminates a connection, the next session in its
own queue was immediately processed. Because of this, if all
server queues are always filled, then no new anonymous request
will be processed. Consider oldest request between global and
server queues to choose from which to pick the request.

An improvement over this will consist in adding a configurable
offset when comparing expiration dates, so that cookie-less
requests can get either less or more priority.
2008-06-20 15:07:40 +02:00
Willy Tarreau
3a6281199a [BUG] event pollers must not wait if a task exists in the run queue
Under some circumstances, a task may already lie in the run queue
(eg: inter-task wakeup). It is disastrous to wait for an event in
this case because some processing gets delayed.
2008-06-20 15:05:56 +02:00
Willy Tarreau
b463dfb2de [MEDIUM] add support for conditional HTTP redirection
A new "redirect" keyword adds the ability to send an HTTP 301/302/303
redirection to either an absolute location or to a prefix followed by
the original URI. The redirection is conditionned by ACL rules, so it
becomes very easy to move parts of a site to another site using this.

This work was almost entirely done at Exceliance by Emeric Brun.

A test-case has been added in the tests/ directory.
2008-06-07 23:08:56 +02:00
Krzysztof Piotr Oledzki
8001d6162e [MEDIUM] Fix memory freeing at exit, part 2
- free oldpids
- call free(exp->preg), not only regfree(exp->preg): req_exp, rsp_exp
- build a list of unique uri_auths and eventually free it
- prune_acl_cond/free for switching_rules
- add a callback pointer to free ptr from acl_pattern (used for regexs) and execute it

==1180== malloc/free: in use at exit: 0 bytes in 0 blocks.
==1180== malloc/free: 5,599 allocs, 5,599 frees, 4,220,556 bytes allocated.
==1180== All heap blocks were freed -- no leaks are possible.
2008-06-07 11:06:14 +02:00
Krzysztof Piotr Oledzki
a643baf091 [MEDIUM] Fix memory freeing at exit
New functions implemented:
 - deinit_pollers: called at the end of deinit())
 - prune_acl: called via list_for_each_entry_safe

Add missing pool_destroy2 calls:
 - p->hdr_idx_pool
 - pool2_tree64

Implement all task stopping:
 - health-check: needs new "struct task" in the struct server
 - queue processing: queue_mgt
 - appsess_refresh: appsession_refresh

before (idle system):
==6079== LEAK SUMMARY:
==6079==    definitely lost: 1,112 bytes in 75 blocks.
==6079==    indirectly lost: 53,356 bytes in 2,090 blocks.
==6079==      possibly lost: 52 bytes in 1 blocks.
==6079==    still reachable: 150,996 bytes in 504 blocks.
==6079==         suppressed: 0 bytes in 0 blocks.

after (idle system):
==6945== LEAK SUMMARY:
==6945==    definitely lost: 7,644 bytes in 137 blocks.
==6945==    indirectly lost: 9,913 bytes in 587 blocks.
==6945==      possibly lost: 0 bytes in 0 blocks.
==6945==    still reachable: 0 bytes in 0 blocks.
==6945==         suppressed: 0 bytes in 0 blocks.

before (running system for ~2m):
==9343== LEAK SUMMARY:
==9343==    definitely lost: 1,112 bytes in 75 blocks.
==9343==    indirectly lost: 54,199 bytes in 2,122 blocks.
==9343==      possibly lost: 52 bytes in 1 blocks.
==9343==    still reachable: 151,128 bytes in 509 blocks.
==9343==         suppressed: 0 bytes in 0 blocks.

after (running system for ~2m):
==11616== LEAK SUMMARY:
==11616==    definitely lost: 7,644 bytes in 137 blocks.
==11616==    indirectly lost: 9,981 bytes in 591 blocks.
==11616==      possibly lost: 0 bytes in 0 blocks.
==11616==    still reachable: 4 bytes in 1 blocks.
==11616==         suppressed: 0 bytes in 0 blocks.

Still not perfect but significant improvement.
2008-05-30 07:07:19 +02:00
Krzysztof Piotr Oledzki
1acf217366 [BUG/CLEANUP] cookiedomain -> cookie_domain rename + free(p->cookie_domain)
Rename cookiedomain -> cookie_domain to be consistent with current
naming scheme. Also make sure cookie_domain is deallocated at deinit()
2008-05-30 07:03:22 +02:00
Willy Tarreau
8a7af60312 [MEDIUM] detect streaming buffers and tag them as such
Add the ability to detect streaming buffers, and set a
flag indicating it. It will later serve us in order to
dynamically resize them, and to prioritize file descriptors
during polls.
2008-05-25 10:41:12 +02:00
Willy Tarreau
f2e8ee2b46 [MEDIUM] reduce risk of event starvation in ev_sepoll
If too many events are set for spec I/O, those ones can starve the
polled events. Experiments show that when polled events starve, they
quickly turn into spec I/O, making the situation even worse. While
we can reduce the number of polled events processed at once, we
cannot do this on speculative events because most of them are new
ones (avg 2/3 new - 1/3 old from experiments).

The solution against this problem relies on those two factors :
  1) one FD registered as a spec event cannot be polled at the same time
  2) even during very high loads, we will almost never be interested in
     simultaneous read and write streaming on the same FD.

The first point implies that during starvation, we will not have more than
half of our FDs in the poll list, otherwise it means there is less than that
in the spec list, implying there is no starvation.

The second point implies that we're statically only interested in half of
the maximum number of file descriptors at once, because we will unlikely
have simultaneous read and writes for a same buffer during long periods.

So, if we make it possible to drain maxsock/2/2 during peak loads, then we
can ensure that there will be no starvation effect. This means that we must
always allocate maxsock/4 events for the poller.

Last, sepoll uses an optimization consisting in reducing the number of calls
to epoll_wait() to once every too polls. However, when dealing with many
spec events, we can wait very long and skipping epoll_wait() every second
time increases latency. For this reason, we try to detect if we are beyond
a reasonable limit and stop doing so at this stage.
2008-05-25 10:39:02 +02:00
Krzysztof Piotr Oledzki
efe3b6f524 [MINOR] Allow to specify a domain for a cookie
This patch allows to specify a domain used when inserting a cookie
providing a session stickiness. Usefull for example with wildcard domains.

The patch adds one new variable to the struct proxy: cookiedomain.
When set the domain is appended to a Set-Cookie header.

Domain name is validated using the new invalid_domainchar() function.
It is basically invalid_char() limited to [A-Za-z0-9_.-]. Yes, the test
is too trivial and does not cover all wrong situations, but the main
purpose is to detect most common mistakes, not intentional abuses.

The underscore ("_") character is not RFC-valid but as it is
often (mis)used so I decided to allow it.
2008-05-25 10:09:02 +02:00
Marek Majkowski
9c30fc161f [MEDIUM] add support for URI hash depth and length limits
This patch adds two optional arguments "len" and "depth" to
"balance uri". They are used to limit the length in characters
of the analysis, as well as the number of directory components
it applies to.
2008-04-28 00:43:55 +02:00
Krzysztof Piotr Oledzki
8e4b21d5eb [BUG] Flush buffers also where there are exactly 0 bytes left
I noticed it was possible to get truncated http/csv stats. Sometimes.
Usually the problem disappeared as fast as it appeared, but once it
happend that my http-stats page was truncated for about one hour.
It was quite weird as it happened independently for csv and http
output and it took me some time to track & fix this bug.

Both buffer_write & buffer_write_chunk used to return 0 in two
situations: is case of success or where there was exactly 0 bytes
left. The first one is intentional but I believe the second one
is not as it was not possible to distinguish between successful
write and unsuccessful one, which means that if the buffer was 100%
filled, it was never flushed and it was not possible to write
more data.

This patch fixes this problem.
2008-04-21 07:22:33 +02:00
Willy Tarreau
7b4c5aee55 [RELEASE] Released version 1.3.15
Released version 1.3.15 with the following main changes :
    - [BUILD] Added support for 'make install'
    - [BUILD] Added 'install-man' make target for installing the man page
    - [BUILD] Added 'install-bin' make target
    - [BUILD] Added 'install-doc' make target
    - [BUILD] Removed "/" after '$(DESTDIR)' in install targets
    - [BUILD] Changed 'install' target to install the binaries first
    - [BUILD] Replace hardcoded 'LD = gcc' with 'LD = $(CC)'
    - [MEDIUM]: Inversion for options
    - [MEDIUM]: Count retries and redispatches also for servers, fix redistribute_pending, extend logs, %d->%u cleanup
    - [BUG]: Restore clearing t->logs.bytes
    - [MEDIUM]: rework checks handling
    - [DOC] Update a "contrib" file with a hint about a scheme used for formathing subjects
    - [MEDIUM] Implement "track [<backend>/]<server>"
    - [MINOR] Implement persistent id for proxies and servers
    - [BUG] Don't increment server connections too much + fix retries
    - [MEDIUM]: Prevent redispatcher from selecting the same server, version #3
    - [MAJOR] proto_uxst rework -> SNMP support
    - [BUG] appsession lookup in URL does not work
    - [BUG] transparent proxy address was ignored in backend
    - [BUG] hot reconfiguration failed because of a wrong error check
    - [DOC] big update to the configuration manual
    - [DOC] large update to the configuration manual
    - [DOC] document more options
    - [BUILD] major rework of the GNU Makefile
    - [STATS] add support for "show info" on the unix socket
    - [DOC] document options forwardfor to logasap
    - [MINOR] add support for the "backlog" parameter
    - [OPTIM] introduce global parameter "tune.maxaccept"
    - [MEDIUM] introduce "timeout http-request" in frontends
    - [MINOR] tarpit timeout is also allowed in backends
    - [BUG] increment server connections for each connect()
    - [MEDIUM] add a turn-around state of one second after a connection failure
    - [BUG] fix typo in redispatched connection
    - [DOC] document options nolinger to ssl-hello-chk
    - [DOC] added documentation for "option tcplog" to "use_backend"
    - [BUG] connect_server: server might not exist when sending error report
    - [MEDIUM] support fully transparent proxy on Linux (USE_LINUX_TPROXY)
    - [MEDIUM] add non-local bind to connect() on Linux
    - [MINOR] add transparent proxy support for balabit's Tproxy v4
    - [BUG] use backend's source and not server's source with tproxy
    - [BUG] fix overlapping server flags
    - [MEDIUM] fix server health checks source address selection
    - [BUG] build failed on CONFIG_HAP_LINUX_TPROXY without CONFIG_HAP_CTTPROXY
    - [DOC] added "server", "source" and "stats" keywords
    - [DOC] all server parameters have been documented
    - [DOC] document all req* and rsp* keywords.
    - [DOC] added documentation about HTTP header manipulations
    - [BUG] log response byte count, not request
    - [BUILD] code did not build in full debug mode
    - [BUG] fix truncated responses with sepoll
    - [MINOR] use s->frt_addr as the server's address in transparent proxy
    - [MINOR] fix configuration hint about timeouts
    - [DOC] minor cleanup of the doc and notice to contributors
    - [MINOR] report correct section type for unknown keywords.
    - [BUILD] update MacOS Makefile to build on newer versions
    - [DOC] fix erroneous "useallbackups" option in the doc
    - [DOC] applied small fixes from early readers
    - [MINOR] add configuration support for "redir" server keyword
    - [MEDIUM] completely implement the server redirection method
    - [TESTS] add a test case for the server redirection mechanism
    - [DOC] add a configuration entry for "server ... redir <prefix>"
    - [BUILD] backend.c and checks.c did not build without tproxy !
    - Revert "[BUILD] backend.c and checks.c did not build without tproxy !"
    - [BUILD] backend.c and checks.c did not build without tproxy !
    - [OPTIM] used unsigned ints for HTTP state and message offsets
    - [OPTIM] GCC4's builtin_expect() is suboptimal
    - [BUG] failed conns were sometimes incremented in the frontend!
    - [BUG] timeout.check was not pre-set to eternity
    - [TESTS] add test-pollers.cfg to easily report pollers in use
    - [BUG] do not apply timeout.connect in checks if unset
    - [BUILD] ensure that makefile understands USE_DLMALLOC=1
    - [MINOR] silent gcc for a wrong warning
    - [CLEANUP] update .gitignore to ignore more temporary files
    - [CLEANUP] report dlmalloc's source path only if explictly specified
    - [BUG] str2sun could leak a small buffer in case of error during parsing
    - [BUG] option allbackups was not working anymore in roundrobin mode
    - [MAJOR] implementation of the "leastconn" load balancing algorithm
    - [BUILD] ensure that users don't build without setting the target anymore.
    - [DOC] document the leastconn LB algo
    - [MEDIUM] fix stats socket limitation to 16 kB
    - [DOC] fix unescaped space in httpchk example.
    - [BUG] fix double-decrement of server connections
    - [TESTS] add a test case for port mapping
    - [TESTS] add a benchmark for integer hashing
    - [TESTS] add new methods in ip-hash test file
    - [MAJOR] implement parameter hashing for POST requests
2008-04-19 21:25:12 +02:00
Willy Tarreau
192ee3e630 [BUILD] fix build of POST analysis code with gcc < 3
move variable declarations at beginning of blocks.
2008-04-19 21:24:56 +02:00
matt.farnsworth@nokia.com
1c2ab96be5 [MAJOR] implement parameter hashing for POST requests
This patch extends the "url_param" load balancing method by introducing
the "check_post" option. Using this option enables analysis of the beginning
of POST requests to search for the specified URL parameter.

The patch also fixes a few minor typos in comments that were discovered
during code review.
2008-04-15 15:30:41 +02:00
Willy Tarreau
f899b94e63 [BUG] fix double-decrement of server connections
If a client does a sudden dirty close (CL_STCLOSE) during a server
connect turn-around, then the number of server connections is
decremented twice. This causes huge problems on the affected
server because when its connection number becomes negative, it
overflows and prevents the server from accepting new connections
due to an apparent saturation.

The fix consists in not decrementing the counter if the server is
in a turn-around state.
2008-03-28 18:19:05 +01:00
Willy Tarreau
39f7e6d516 [MEDIUM] fix stats socket limitation to 16 kB
Due to the way the stats socket work, it was not possible to
maintain the information related to the command entered, so
after filling a whole buffer, the request was lost and it was
considered that there was nothing to write anymore.

The major reason was that some flags were passed directly
during the first call to stats_dump_raw() instead of being
stored persistently in the session.

To definitely fix this problem, flags were added to the stats
member of the session structure.

A second problem appeared. When the stats were produced, a first
call to client_retnclose() was performed, then one or multiple
subsequent calls to buffer_write_chunks() were done. But once the
stats buffer was full and a reschedule operated, the buffer was
flushed, the write flag cleared from the buffer and nothing was
done to re-arm it.

For this reason, a check was added in the proto_uxst_stats()
function in order to re-call the client FSM when data were added
by stats_dump_raw(). Finally, the whole unix stats dump FSM was
rewritten to avoid all the magics it depended on. It is now
simpler and looks more like the HTTP one.
2008-03-17 22:08:01 +01:00
Willy Tarreau
51406233bb [MAJOR] implementation of the "leastconn" load balancing algorithm
The new "leastconn" LB algorithm selects the server which has the
least established or pending connections. The weights are considered,
so that a server with a weight of 20 will get twice as many connections
as the server with a weight of 10.

The algorithm respects the minconn/maxconn settings, as well as the
slowstart since it is a dynamic algorithm. It also correctly supports
backup servers (one and all).

It is generally suited for protocols with long sessions (such as remote
terminals and databases), as it will ensure that upon restart, a server
with no connection will take all new ones until its load is balanced
with others.

A test configuration has been added in order to ease regression testing.
2008-03-10 22:04:30 +01:00
Willy Tarreau
f4cca45b5e [BUG] option allbackups was not working anymore in roundrobin mode
Commit 3168223a7b broke option
"allbackups" in roundrobin mode due to an erroneous structure
member replacement in backend.c. The PR_O_USE_ALL_BK flag was
not tested in the right member anymore.

This bug uncoverred another one, by which all backup servers would
be used whatever the option's value, if all of them had been seen
as simultaneously failed at one moment.

This patch fixes the two stupid errors. Correctness has been tested
using the test-fwrr.cfg config example.
2008-03-08 21:42:54 +01:00
Willy Tarreau
caf720d3ff [BUG] str2sun could leak a small buffer in case of error during parsing
Matt Farnsworth reported a memory leak in str2sun() in case a too large
socket path is passed. The bug is very minor because it only happens
once during config parsing, but has to be fixed nevertheless. The patch
Matt provided could even be improved by completely removing the useless
strdup() in this function.
2008-03-07 10:07:04 +01:00
Krzysztof Piotr Oledzki
2c6962c3c0 [MAJOR] proto_uxst rework -> SNMP support
Currently there is a ~16KB limit for a data size passed via unix socket.
It is caused by a trivial bug ttat is going to fixed soon, however
in most cases there is no need to dump a full stats.

This patch makes possible to select a scope of dumped data by extending
current "show stat" to "show stat [<iid> <type> <sid>]":
 - iid is a proxy id, -1 to dump all proxies
 - type selects type of dumpable objects: 1 for frontend, 2 for backend, 4 for
   server, -1 for all types. Values can be ORed, for example:
     1+2=3   -> frontend+backend.
     1+2+4=7 -> frontend+backend+server.
 - sid is a service id, -1 to dump everything from the selected proxy.

To do this I implemented a new session flag (SN_STAT_BOUND), added three
variables in data_ctx.stats (iid, type, sid), modified dumpstats.c and
completely revorked the process_uxst_stats: now it waits for a "\n"
terminated string, splits args and uses them. BTW: It should be quite easy
to add new commands, for example to enable/disable servers, the only problem
I can see is a not very lucky config name (*stats* socket). :|

During the work I also fixed two bug:
 - s->flags were not initialized for proto_uxst
 - missing comma if throttling not enabled (caused by a stupid change in
     "Implement persistent id for proxies and servers")

Other changes:
 - No more magic type valuse, use STATS_TYPE_FE/STATS_TYPE_BE/STATS_TYPE_SV
 - Don't memset full s->data_ctx (it was clearing s->data_ctx.stats.{iid/type/sid},
    instead initialize stats.sv & stats.sv_st (stats.px and stats.px_st were already
    initialized)

With all that changes it was extremely easy to write a short perl plugin
for a perl-enabled net-snmp (also included in this patch).

29385 is my PEN (Private Enterprise Number) and I'm willing to donate
the SNMPv2-SMI::enterprises.29385.106.* OIDs for HAProxy if there
is nothing assigned already.
2008-03-04 06:32:16 +01:00
Krzysztof Piotr Oledzki
5a329cf017 [MEDIUM]: Prevent redispatcher from selecting the same server, version #3
When haproxy decides that session needs to be redispatched it chose a server,
but there is no guarantee for it to be a different one. So, it often
happens that selected server is exactly the same that it was previously, so
a client ends up with a 503 error anyway, especially when one sever has
much bigger weight than others.

Changes from the previous version:
 - drop stupid and unnecessary SN_DIRECT changes

 - assign_server(): use srvtoavoid to keep the old server and clear s->srv
    so SRV_STATUS_NOSRV guarantees that t->srv == NULL (again)
    and get_server_rr_with_conns has chances to work (previously
    we were passing a NULL here)

 - srv_redispatch_connect(): remove t->srv->cum_sess and t->srv->failed_conns
   incrementing as t->srv was guaranteed to be NULL

 - add avoididx to get_server_rr_with_conns. I hope I correctly understand this code.

 - fix http_flush_cookie_flags() and move it to assign_server_and_queue()
   directly. The code here was supposed to set CK_DOWN and clear CK_VALID,
   but: (TX_CK_VALID | TX_CK_DOWN) == TX_CK_VALID == TX_CK_MASK so:
	if ((txn->flags & TX_CK_MASK) == TX_CK_VALID)
		txn->flags ^= (TX_CK_VALID | TX_CK_DOWN);
   was really a:
	if ((txn->flags & TX_CK_MASK) == TX_CK_VALID)
		txn->flags &= TX_CK_VALID

   Now haproxy logs "--DI" after redispatching connection.

 - defer srv->redispatches++ and s->be->redispatches++ so there
   are called only if a conenction was redispatched, not only
   supposed to.

 - don't increment lbconn if redispatcher selected the same sarver

 - don't count unsuccessfully redispatched connections as redispatched
   connections

 - don't count redispatched connections as errors, so:

 - the number of connections effectively served by a server is:
 srv->cum_sess - srv->failed_conns - srv->retries - srv->redispatches
   and
 SUM(servers->failed_conns) == be->failed_conns

 - requires the "Don't increment server connections too much + fix retries" patch

 - needs little more testing and probably some discussion so reverting to the RFC state

Tests #1:
 retries 4
 redispatch

i) 1 server(s): b (wght=1, down)
  b) sessions=5, lbtot=1, err_conn=1, retr=4, redis=0
  -> request failed

ii) server(s): b (wght=1, down), u (wght=1, down)
  b) sessions=4, lbtot=1, err_conn=0, retr=3, redis=1
  u) sessions=1, lbtot=1, err_conn=1, retr=0, redis=0
  -> request FAILED

iii) 2 server(s): b (wght=1, down), u (wght=1, up)
  b) sessions=4, lbtot=1, err_conn=0, retr=3, redis=1
  u) sessions=1, lbtot=1, err_conn=0, retr=0, redis=0
  -> request OK

iv) 2 server(s): b (wght=100, down), u (wght=1, up)
  b) sessions=4, lbtot=1, err_conn=0, retr=3, redis=1
  u) sessions=1, lbtot=1, err_conn=0, retr=0, redis=0
  -> request OK

v) 1 server(s): b (down for first 4 SYNS)
  b) sessions=5, lbtot=1, err_conn=0, retr=4, redis=0
  -> request OK

Tests #2:
 retries 4

i) 1 server(s): b (down)
  b) sessions=5, lbtot=1, err_conn=1, retr=4, redis=0
  -> request FAILED
2008-03-04 06:16:37 +01:00
Krzysztof Piotr Oledzki
626a19b66f [BUG] Don't increment server connections too much + fix retries
Commit 98937b8757 while fixing
one bug introduced another one. With "retries 4" and
"option redispatch" haproxy tries to connect 4 times to
one server server and 1 time to a second one. However
logs showed 5 connections to the first server (the
last one was counted twice) and 2 to the second.

This patch also fixes srv->retries and be->retries increments.

Now I get: 3 retries and 1 error in a first server (4 cum_sess)
and 1 error in a second server (1 cum_sess) with:
 retries 4
 option redispatch

and: 4 retries and 1 error (5 cum_sess) with:
 retries 4

So, the number of connections effectively served by a server is:
 srv->cum_sess - srv->failed_conns - srv->retries
2008-03-04 06:11:17 +01:00
Krzysztof Piotr Oledzki
f58a962247 [MINOR] Implement persistent id for proxies and servers
This patch adds a possibility to set a persistent id for a proxy/server.
Now, even if some proxies/servers are inserted/deleted/moved, iids and
sids can be still used reliable.

Some people add servers with tricky names (BACKEND or FRONTEND for example).
So I also added one more field ('type') to distinguish between a
backend (0), frontend (1) and server (2) without complicated logic:
if name==BACKEND and sid==0 then type is BACKEND else type is SERVER,
etc for a FRONTEND. It also makes possible to have one frontend with more
than one IP (a patch coming soon) with independed stats - for example to
differs between remote and local traffic.

Finally, I added documentation about the CSV format.

This patch depends on '[MEDIUM] Implement "track [<backend>/]<server>"'
2008-02-28 17:23:59 +01:00
Krzysztof Piotr Oledzki
c8b16fc948 [MEDIUM] Implement "track [<backend>/]<server>"
This patch implements ability to set the current state of one server
by tracking another one. It:
 - adds two variables: *tracknext, *tracked to struct server
 - implements findserver(), similar to findproxy()
 - adds "track" keyword accepting both "proxy/server" and "server" (assuming current proxy)
 - verifies if both checks and tracking is not enabled at the same time
 - changes set_server_down() to notify tracking server
 - creates set_server_up(), set_server_disabled(), set_server_enabled() by
   moving the code from process_chk() and adding notifications
 - changes stats to show a name of tracked server instead of Chk/Dwn/Dwntime(html)
   or by adding new variable (csv)

Changes from the previuos version:
 - it is possibile to track independently of the declaration order
 - one extra comma bug is fixed
 - new condition to check if there is no disable-on-404 inconsistency
2008-02-27 10:39:53 +01:00
Willy Tarreau
6054819a70 [BUG] do not apply timeout.connect in checks if unset
tv_bound() does not consider infinite timeouts, so we must
check that timeout.connect is set before applying it to the
checks.
2008-02-17 11:34:10 +01:00
Ryan Warnick
6d0b1fac23 [BUG] appsession lookup in URL does not work
We've been trying to use the latest release (1.3.14.2) of haproxy  to do
sticky sessions.  Cookie insertion is not an option for us, although we
would much rather use it, as we are trying to work around a problem where
cookies are unreliable.  The appsession functionality only partially worked
(it wouldn't read the session id out of a query string) until we made the
following code change to the get_srv_from_appsession function in
proto_http.c.
2008-02-17 11:24:35 +01:00
Willy Tarreau
3a70f94991 [BUG] timeout.check was not pre-set to eternity
If timeout.check was not set, check were using 0 as the timeout, causing
odd behaviours.
2008-02-15 11:15:34 +01:00
Willy Tarreau
50fd1e1e3b [BUG] failed conns were sometimes incremented in the frontend! 2008-02-15 10:09:15 +01:00
Willy Tarreau
70bcfb77a7 [OPTIM] GCC4's builtin_expect() is suboptimal
GCC4 is stupid (unbelievable news!).

When some code uses __builtin_expect(x != 0, 1), it really performs
the check of x != 0 then tests that the result is not zero! This is
a double check when only one was expected. Some performance drops
of 10% in the HTTP parser code have been observed due to this bug.

GCC 3.4 is fine though.

A solution consists in expecting that the tested value is 1. In
this case, it emits the correct code, but it's still not optimal
it seems. Finally the best solution is to ignore likely() and to
pray for the compiler to emit correct code. However, we still have
to fix unlikely() to remove the test there too, and to fix all
code which passed pointers overthere to pass integers instead.
2008-02-14 23:14:33 +01:00
Willy Tarreau
e69eada057 [OPTIM] used unsigned ints for HTTP state and message offsets
State and offsets within http_msg were incorrectly set to signed int.
Turning them into unsigned slightly improved performance while reducing
code size.
2008-02-14 23:14:30 +01:00
Willy Tarreau
cf1d572f2a [BUILD] backend.c and checks.c did not build without tproxy !
missing #ifdefs. The right patch this time!
2008-02-14 20:28:18 +01:00
Willy Tarreau
21d2af3e9f Revert "[BUILD] backend.c and checks.c did not build without tproxy !"
This reverts commit 3c3c0122f8.
This commit was buggy as it also removed previous tproxy changes !
2008-02-14 20:25:24 +01:00
Willy Tarreau
3c3c0122f8 [BUILD] backend.c and checks.c did not build without tproxy !
missing #ifdefs.
2008-02-13 22:22:56 +01:00
Willy Tarreau
9c33612f53 [MEDIUM] completely implement the server redirection method
Now when a server has "redir <prefix>" on its config line, any HEAD or GET
request addressing it will lead to a 302 with Location set to "<prefix>"
immediately followed by the relative URI of the incoming request. This makes
it very easy to send redirect to browsers to check remote static servers, as
well as to provide redirection for remote sites when the local one is down.
2008-02-13 00:55:49 +01:00
Willy Tarreau
7a58a72e85 [MINOR] add configuration support for "redir" server keyword
The servers now support the "redir" keyword, making it possible to
return a 302 with the specified prefix in front of the request instead
of connecting to them. This is generally useful for multi-site load
balancing but may also serve in order to achieve very high traffic
rate.

The keyword has only been added to the config parser and to structures,
it's not used yet.
2008-02-13 00:55:49 +01:00
Willy Tarreau
6daf34352f [MINOR] report correct section type for unknown keywords.
An unknown keyword was always reported in section "listen" for any
section type (defaults, listen, frontend, backend, ...).
2008-01-22 16:44:08 +01:00
Krzysztof Piotr Oledzki
5259dfedd1 [MEDIUM]: rework checks handling
This patch adds two new variables: fastinter and downinter.
When server state is:
 - non-transitionally UP -> inter (no change)
 - transitionally UP (going down), unchecked or transitionally DOWN (going up) -> fastinter
 - down -> downinter

It allows to set something like:
        server sr6 127.0.51.61:80 cookie s6 check inter 10000 downinter 20000 fastinter 500 fall 3 weight 40
In the above example haproxy uses 10000ms between checks but as soon as
one check fails fastinter (500ms) is used. If server is down
downinter (20000) is used or fastinter (500ms) if one check pass.
Fastinter is also used when haproxy starts.

New "timeout.check" variable was added, if set haproxy uses it as an additional
read timeout, but only after a connection has been already established. I was
thinking about using "timeout.server" here but most people set this
with an addition reserve but still want checks to kick out laggy servers.
Please also note that in most cases check request is much simpler
and faster to handle than normal requests so this timeout should be smaller.

I also changed the timeout used for check connections establishing.

Changes from the previous version:
 - use tv_isset() to check if the timeout is set,
 - use min("timeout connect", "inter") but only if "timeout check" is set
   as this min alone may be to short for full (connect + read) check,
 - debug code (fprintf) commented/removed
 - documentation

Compile tested only (sorry!) as I'm currently traveling but changes
are rather small and trivial.
2008-01-22 11:29:06 +01:00
Krzysztof Piotr Oledzki
f1e1cb463f [BUG]: Restore clearing t->logs.bytes
Commit 8b3977ffe3 removed "t->logs.bytes_in = 0;"
but instead it should change it into "t->logs.bytes_out = 0;" as since
583bc96606 counters are incremented not set.

It should be incremented in session_process_counters while sending data to a
client:
        bytes = s->rep->total - s->logs.bytes_out;
        s->logs.bytes_out = s->rep->total;

However, if we increment (set) s->logs.bytes_out while processing
"logasap", statistics get wrong values added for headers: 0 or even
negative if haproxy adds some headers itself.

To test it, please enable logasap and download one empty file and look at
stats. Without my fix information available on that page are invalid, for
example:

# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,
www,b,0,0,0,1,,1,24,-92,,0,,0,0,0,,UP,1,1,0,0,0,3121,0,,1,2,1,,1,
www,BACKEND,0,0,0,1,0,1,24,-92,0,0,,0,0,0,0,UP,1,1,0,,0,3121,0,,1,2,0,,1,
2008-01-22 10:30:26 +01:00
Willy Tarreau
0f68eaca1a [MINOR] fix configuration hint about timeouts
Do not talk about "clitimeout", "contimeout" or "srvtimeout"
anymore.
2008-01-20 23:25:06 +01:00
Willy Tarreau
bd41428fee [MINOR] use s->frt_addr as the server's address in transparent proxy
There's no point trying to check original dest addr with only one
method when doing transparent proxy as in full transparent mode,
the real destination address is required. Let's copy the one from
the frontend.
2008-01-19 13:46:35 +01:00