Commit Graph

421 Commits

Author SHA1 Message Date
Olivier Houchard
a4d4fdfaa3 MEDIUM: sessions: Don't keep an infinite number of idling connections.
In session, don't keep an infinite number of connection that can idle.
Add a new frontend parameter, "max-session-srv-conns" to set a max number,
with a default value of 5.
2018-12-15 23:50:10 +01:00
Olivier Houchard
f502aca5c2 MEDIUM: mux: provide the session to the init() and attach() method.
Instead of trying to get the session from the connection, which is not
always there, and of course there could be multiple sessions per connection,
provide it with the init() and attach() methods, so that we know the
session for each outgoing stream.
2018-12-15 23:50:09 +01:00
Olivier Houchard
006e3101f9 MEDIUM: servers: Add a command to limit the number of idling connections.
Add a new command, "pool-max-conn" that sets the maximum number of connections
waiting in the orphan idling connections list (as activated with idle-timeout).
Using "-1" means unlimited. Using pools is now dependant on this.
2018-12-15 23:50:08 +01:00
Willy Tarreau
cc79ed28f6 BUG/MAJOR: backend: only update server's counters when the server exists
PiBa-NL reported that since this commit f157384 ("MINOR: backend: count
the number of connect and reuse per server and per backend"), reg-test
connection/h00001 fails. Indeed it does, the server is not checked for
existing prior to updating its counter. It should also fail with
transparent mode.
2018-12-15 15:13:10 +01:00
Willy Tarreau
f157384803 MINOR: backend: count the number of connect and reuse per server and per backend
Sadly we didn't have the cumulated number of connections established to
servers till now, so let's now update it per backend and per-server and
report it in the stats. On the stats page it appears in the tooltip
when hovering over the total sessions count field.
2018-12-14 11:35:36 +01:00
Olivier Houchard
9a86fcbd47 MEDIUM: mux: Add an optional "reset" method.
Add a new method to mux, "reset", that is used to let the mux know the
connection attempt failed, and we're about to retry, so it just have to
reinit itself. Currently only the H1 mux needs it.
2018-12-13 17:32:15 +01:00
Olivier Houchard
ab8b075ff0 BUG/MEDIUM: connections: Remove CS_FL_EOS | CS_FL_REOS on retry.
CS_FL_EOS | CS_FL_REOS can be set by the mux if the connection failed, so make
sure we remove them before retrying to connect, or it may lead to a premature
close of the connection.
2018-12-13 17:32:15 +01:00
Olivier Houchard
ac1ce6f9b8 BUG/MEDIUM: connections: Remove error flags when retrying.
In connect_server(), when retrying to connect, remove the error flags from
the connection and the conn_stream, we're trying to connect again, anyway.
2018-12-08 21:56:07 +01:00
Olivier Houchard
eb2bbba547 BUG/MEDIUM: connection: Don't use the provided conn_stream if it was tried.
In connect_server(), don't attempt to reuse the conn_stream associated to
the stream_interface, if we already attempted a connection with it.
Using that conn_stream is only there for the cases where a connection and
a conn_stream was created ahead, mostly by http_proxy or by the LUA code.
If we already attempted to connect, that means we fail, and so we should
create a new connection.

No backport needed.
2018-12-08 18:13:46 +01:00
Olivier Houchard
0fa989f4c0 BUG/MEDIUM: connections: Reuse an already attached conn_stream.
In connect_server(), if we already have a conn_stream, reuse it
instead of trying to create a new one. http_proxy and LUA both
manually create a conn_stream and a connection, and we want
to use it.
2018-12-06 15:06:19 +01:00
Olivier Houchard
0c18a6fe34 MEDIUM: servers: Add a way to keep idle connections alive.
Add a new keyword for servers, "idle-timeout". If set, unused connections are
kept alive until the timeout happens, and will be picked for reuse if no
other connection is available.
2018-12-02 18:16:53 +01:00
Olivier Houchard
2442f68dd3 BUG/MEDIUM: Special-case http_proxy when dealing with outgoing connections.
http_proxy is special, because it creates its connection and conn_stream
earlier. So in assign_server(), check that the connection associated with
the conn_stream has a destination address set, and in connect_server(),
use the connection and the conn_stream already attached to the
stream_interface, instead of looking for a connection in the session, and
creating a new conn_stream.
2018-12-01 17:20:03 +01:00
Olivier Houchard
ba4fff5fd2 MEDIUM: server: Be smarter about deciding to reuse the last server.
Instead of parsing all the available connections owned by the session
each time we choose a server, even if prefer-last-server is not set,
just do it if prefer-last-server is used, and check if the server is usable,
before checking the connections.
2018-12-01 15:45:30 +01:00
Olivier Houchard
00cf70f28b MAJOR: sessions: Store multiple outgoing connections in the session.
Instead of just storing the last connection in the session, store all of
the connections, for at most MAX_SRV_LIST (currently 5) targets.
That way we can do keepalive on more than 1 outgoing connection when the
client uses HTTP/2.
2018-12-01 10:47:18 +01:00
Olivier Houchard
bf024f0a15 MEDIUM: connections: Put H2 connections in the idle list if http-reuse always.
When creating a new outgoing H2 connection, put it in the idle list so that
it's immediately available for others to use, if http-reuse always is used.
2018-12-01 10:47:18 +01:00
Olivier Houchard
a30a40bcca BUG/MEDIUM: connections: Remove the connection from the idle list before destroy.
Before calling the destroy() method, remove the connection from the idle list,
so that no new session will pick it.
2018-12-01 10:47:16 +01:00
Olivier Houchard
a49d41a9af BUG/MEDIUM: connections: Don't assume we have a mux in connect_server().
When dealing with the previous connection, don't assume it has a mux, as it
may not yet be the case if we're waiting for the ALPN.
2018-12-01 10:47:16 +01:00
Olivier Houchard
d76bd2d40b BUG/MEDIUM: connections: Don't forget to detach the connection from the SI.
When we're deferring the mux choice until the ALPN is negociated, we
attach the connection to the stream_interface until it's done, so that we
can destroy it if something goes wrong and the stream is destroy.
Before calling si_attach_cs() to attach the conn_stream once we have it,
call si_detach_endpoint(), or is_attach_cs() would destroy the connection.
2018-11-29 17:39:04 +01:00
Olivier Houchard
70d9b2fdb0 BUG/MEDIUM: connections: Wake the stream once the mux is chosen.
When we defer the mux choice until the ALPN is negociated, don't forget
to wake the stream once it's done, or it will never have the opportunity
to send data.
2018-11-29 17:39:04 +01:00
Willy Tarreau
0108d90c6c MEDIUM: init: convert all trivial registration calls to initcalls
This switches explicit calls to various trivial registration methods for
keywords, muxes or protocols from constructors to INITCALL1 at stage
STG_REGISTER. All these calls have in common to consume a single pointer
and return void. Doing this removes 26 constructors. The following calls
were addressed :

- acl_register_keywords
- bind_register_keywords
- cfg_register_keywords
- cli_register_kw
- flt_register_keywords
- http_req_keywords_register
- http_res_keywords_register
- protocol_register
- register_mux_proto
- sample_register_convs
- sample_register_fetches
- srv_register_keywords
- tcp_req_conn_keywords_register
- tcp_req_cont_keywords_register
- tcp_req_sess_keywords_register
- tcp_res_cont_keywords_register
- flt_register_keywords
2018-11-26 19:50:32 +01:00
Lukas Tribus
da95fd901b BUILD/MINOR: ssl: fix build with non-alpn/non-npn libssl
In commit c7566001 ("MINOR: server: Add "alpn" and "npn" keywords") and
commit 201b9f4e ("MAJOR: connections: Defer mux creation for outgoing
connection if alpn is set"), the build was broken on older OpenSSL
releases.

Move the #ifdef's around so that we build again with older OpenSSL
releases (0.9.8 was tested).
2018-11-26 08:34:40 +01:00
Olivier Houchard
ee23b2a1e3 MEDIUM: servers: Store the connection in the SI until we have a mux.
When we create a connection, if we have to defer the conn_stream and the
mux creation until we can decide it (ie until the SSL handshake is done, and
the ALPN is decided), store the connection in the stream_interface, so that
we're sure we can destroy it if needed.
2018-11-23 19:11:14 +01:00
Olivier Houchard
1295016873 BUG/MEDIUM: servers: Don't check if we have a conn_stream too soon.
The creation of the conn_stream for an outgoing connection has been delayed
a bit, and when using dispatch, a check was made to see if a conn_stream
was attached before the conn_stream was created, so remove the test, as
it's done later anyway, and create and install the conn_stream right away
when we don't have a server, as is done when we don't have an alpn/npn
defined.
2018-11-23 14:56:21 +01:00
Olivier Houchard
c6e0bb4944 MINOR: server: Only defined conn_complete_server if USE_OPENSSL is set.
conn_complete_server() is only used when using ALPN/NPN, so only define it
if USE_OPENSSL is set.
2018-11-23 14:56:13 +01:00
Olivier Houchard
201b9f4eb5 MAJOR: connections: Defer mux creation for outgoing connection if alpn is set.
If an ALPN (or a NPN) was chosen for a server, defer choosing the mux until
after the SSL handshake is done, and the ALPN/NPN has been negociated, so
that we know which mux to pick.
2018-11-22 19:52:23 +01:00
Olivier Houchard
7c6f8b146d MAJOR: connections: Detach connections from streams.
Do not destroy the connection when we're about to destroy a stream. This
prevents us from doing keepalive on server connections when the client is
using HTTP/2, as a new stream is created for each request.
Instead, the session is now responsible for destroying connections.
When reusing connections, the attach() mux method is now used to create a new
conn_stream.
2018-11-18 21:45:45 +01:00
Olivier Houchard
47e9a1ad4e MEDIUM: connections: Wait until the connection is established to try to recv.
Instead of trying to receive as soon as the connection is created, and to
eventually have to transfer subscription if we move connections, wait
until the connection is established before attempting to recv.
2018-11-18 21:41:50 +01:00
Willy Tarreau
cde1bc64cb BUG/MINOR: backend: assign the wait list after the error check
Commit 85b73e9 ("BUG/MEDIUM: stream: Make sure polling is right on retry.")
introduced a possible null dereference on the error path detected by gcc-7.
Let's simply assign srv_conn after checking the error and not before.

No backport is needed.
2018-10-28 20:36:00 +01:00
Lukas Tribus
80512b186f BUG/MINOR: only auto-prefer last server if lb-alg is non-deterministic
While "option prefer-last-server" only applies to non-deterministic load
balancing algorithms, 401/407 responses actually caused haproxy to prefer
the last server unconditionally.

As this breaks deterministic load balancing algorithms like uri, this
patch applies the same condition here.

Should be backported to 1.8 (together with "BUG/MINOR: only mark
connections private if NTLM is detected").
2018-10-27 22:10:32 +02:00
Olivier Houchard
85b73e9427 BUG/MEDIUM: stream: Make sure polling is right on retry.
When retrying to connect to a server, because the previous connection failed,
make sure if we subscribed to the previous connection, the polling flags will
be true for the new fd.

No backport is needed.
2018-10-21 05:55:32 +02:00
Willy Tarreau
33dd4ef812 BUG/MINOR: backend: check that the mux installed properly
The return value from conn_install_mux() was not checked, so if an
inconsistency happens in the code, or a memory allocation fails while
initializing the mux, we can crash while using an uninitialized mux.
In practice the code inconsistency does not really happen since we
cannot configure such a situation, except during development, but
the out of memory condition could definitely happen.

This should be backported to 1.8 (the code is a bit different there,
there are two calls to conn_install_mux()).
2018-10-03 10:24:05 +02:00
Willy Tarreau
1e582e5e5c BUILD: backend: fix 3 build warnings related to null-deref at -Wextra
These ones are not valid either since the checks are performed a few
lines above the call. Let's switch to __objt_server() instead.
2018-09-20 11:42:15 +02:00
Patrick Hemmer
155e93e570 MINOR: Add srv_conn_free sample fetch
This adds the 'srv_conn_free([<backend>/]<server>)' sample fetch. This fetch
provides the number of available connections on the designated server.
2018-08-27 16:38:56 +02:00
Patrick Hemmer
4cdf3abaa0 MINOR: add be_conn_free sample fetch
This adds the sample fetch 'be_conn_free([<backend>])'. This sample fetch
provides the total number of unused connections across available servers in the
specified backend.
2018-08-27 14:10:16 +02:00
Christopher Faulet
7ce0c891ab MEDIUM: mux: Use the mux protocol specified on bind/server lines
To do so, mux choices are split to handle incoming and outgoing connections in a
different way. The protocol specified on the bind/server line is used in
priority. Then, for frontend connections, the ALPN is retrieved and used to
choose the best mux. For backend connection, there is no ALPN. Finaly, if no
protocol is specified and no protocol matches the ALPN, we fall back on a
default mux, choosing in priority the first mux with exactly the same mode.
2018-08-08 10:42:08 +02:00
Christopher Faulet
b75bb21092 MEDIUM: backend: don't rely on mux_pt_ops in connect_server()
The comment above the change remains true. We assume there is always 1
conn_stream per outgoing connectionq. Today, it is always true because H2 is not
supported yet for server connections.
2018-08-08 09:54:22 +02:00
Christopher Faulet
6cc7afa04e MINOR: backend: Try to find the best mux for outgoing connections
For now, there is no effect. mux-pt will always be used because this is only
available mux for backend connections.
2018-08-08 09:54:22 +02:00
Christopher Faulet
2bf88c05d0 CLEANUP: backend: Move mux install to call it at only one place
It makes the code readability simpler. It will also ease futur changes.
2018-08-07 14:37:37 +02:00
Christopher Faulet
4507351a2f BUG/MINOR: build: Fix compilation with debug mode enabled
It remained some fragments of the old buffers API in debug messages, here and
there.

This was caused by the recent buffer API changes, no backport is needed.
2018-07-20 10:45:20 +02:00
Willy Tarreau
843b7cbe9d MEDIUM: chunks: make the chunk struct's fields match the buffer struct
Chunks are only a subset of a buffer (a non-wrapping version with no head
offset). Despite this we still carry a lot of duplicated code between
buffers and chunks. Replacing chunks with buffers would significantly
reduce the maintenance efforts. This first patch renames the chunk's
fields to match the name and types used by struct buffers, with the goal
of isolating the code changes from the declaration changes.

Most of the changes were made with spatch using this coccinelle script :

  @rule_d1@
  typedef chunk;
  struct chunk chunk;
  @@
  - chunk.str
  + chunk.area

  @rule_d2@
  typedef chunk;
  struct chunk chunk;
  @@
  - chunk.len
  + chunk.data

  @rule_i1@
  typedef chunk;
  struct chunk *chunk;
  @@
  - chunk->str
  + chunk->area

  @rule_i2@
  typedef chunk;
  struct chunk *chunk;
  @@
  - chunk->len
  + chunk->data

Some minor updates to 3 http functions had to be performed to take size_t
ints instead of ints in order to match the unsigned length here.
2018-07-19 16:23:43 +02:00
Willy Tarreau
c9fa0480af MAJOR: buffer: finalize buffer detachment
Now the buffers only contain the header and a pointer to the storage
area which can be anywhere. This will significantly simplify buffer
swapping and will make it possible to map chunks on buffers as well.

The buf_empty variable was removed, as now it's enough to have size==0
and area==NULL to designate the empty buffer (thus a non-allocated head
is the empty buffer by default). buf_wanted for now is indicated by
size==0 and area==(void *)1.

The channels and the checks now embed the buffer's head, and the only
pointer is to the storage area. This slightly increases the unallocated
buffer size (3 extra ints for the empty buffer) but considerably
simplifies dynamic buffer management. It will also later permit to
detach unused checks.

The way the struct buffer is arranged has proven quite efficient on a
number of tests, which makes sense given that size is always accessed
and often first, followed by the othe ones.
2018-07-19 16:23:43 +02:00
Willy Tarreau
6a445ebc8a MINOR: backend: use new buffer API
The few locations dealing with the buffer rewind were updated not to
touch ->o nor ->p anymore and to use the channel's functions instead.
2018-07-19 16:23:42 +02:00
Willy Tarreau
188e230704 MINOR: buffer: convert most b_ptr() calls to c_ptr()
The latter uses the channel wherever a channel is known.
2018-07-19 16:23:40 +02:00
Willy Tarreau
bcbd39370f MINOR: channel/buffer: replace b_{adv,rew} with c_{adv,rew}
These ones manipulate the output data count which will be specific to
the channel soon, so prepare the call points to use the channel only.
The b_* functions are now unused and were removed.
2018-07-19 16:23:40 +02:00
Willy Tarreau
760e81d356 MINOR: backend: implement random-based load balancing
For large farms where servers are regularly added or removed, picking
a random server from the pool can ensure faster load transitions than
when using round-robin and less traffic surges on the newly added
servers than when using leastconn.

This commit introduces "balance random". It internally uses a random as
the key to the consistent hashing mechanism, thus all features available
in consistent hashing such as weights and bounded load via hash-balance-
factor are usable. It is extremely convenient because one common concern
when using random is what happens when a server is hammered a bit too
much. Here that can trivially be avoided, like in the configuration below :

    backend bk0
        balance random
        hash-balance-factor 110
        server-template s 1-100 127.0.0.1:8000 check inter 1s

Note that while "balance random" internally relies on a hash algorithm,
it holds the same properties as round-robin and as such is compatible with
reusing an existing server connection with "option prefer-last-server".
2018-05-03 07:20:40 +02:00
Christopher Faulet
767a84bcc0 CLEANUP: log: Rename Alert/Warning in ha_alert/ha_warning 2017-11-24 17:19:12 +01:00
Christopher Faulet
56803b1c98 CLEANUP: debug: Use DPRINTF instead of fprintf into #ifdef DEBUG_FULL/#endif 2017-11-24 17:19:03 +01:00
Willy Tarreau
46c9d3e6cb BUILD: ssl: fix build of backend without ssl
Commit 522eea7 ("MINOR: ssl: Handle sending early data to server.") added
a dependency on SRV_SSL_O_EARLY_DATA which only exists when USE_OPENSSL
is defined (which is probably not the best solution) and breaks the build
when ssl is not enabled. Just add an ifdef USE_OPENSSL around the block
for now.
2017-11-08 14:28:08 +01:00
Olivier Houchard
522eea7110 MINOR: ssl: Handle sending early data to server.
This adds a new keyword on the "server" line, "allow-0rtt", if set, we'll try
to send early data to the server, as long as the client sent early data, as
in case the server rejects the early data, we no longer have them, and can't
resend them, so the only option we have is to send back a 425, and we need
to be sure the client knows how to interpret it correctly.
2017-11-08 14:11:10 +01:00
Olivier Houchard
9aaf778129 MAJOR: connection : Split struct connection into struct connection and struct conn_stream.
All the references to connections in the data path from streams and
stream_interfaces were changed to use conn_streams. Most functions named
"something_conn" were renamed to "something_cs" for this. Sometimes the
connection still is what matters (eg during a connection establishment)
and were not always renamed. The change is significant and minimal at the
same time, and was quite thoroughly tested now. As of this patch, all
accesses to the connection from upper layers go through the pass-through
mux.
2017-10-31 18:03:23 +01:00
Willy Tarreau
53a4766e40 MEDIUM: connection: start to introduce a mux layer between xprt and data
For HTTP/2 and QUIC, we'll need to deal with multiplexed streams inside
a connection. After quite a long brainstorming, it appears that the
connection interface to the existing streams is appropriate just like
the connection interface to the lower layers. In fact we need to have
the mux layer in the middle of the connection, between the transport
and the data layer.

A mux can exist on two directions/sides. On the inbound direction, it
instanciates new streams from incoming connections, while on the outbound
direction it muxes streams into outgoing connections. The difference is
visible on the mux->init() call : in one case, an upper context is already
known (outgoing connection), and in the other case, the upper context is
not yet known (incoming connection) and will have to be allocated by the
mux. The session doesn't have to create the new streams anymore, as this
is performed by the mux itself.

This patch introduces this and creates a pass-through mux called
"mux_pt" which is used for all new connections and which only
calls the data layer's recv,send,wake() calls. One incoming stream
is immediately created when init() is called on the inbound direction.
There should not be any visible impact.

Note that the connection's mux is purposely not set until the session
is completed so that we don't accidently run with the wrong mux. This
must not cause any issue as the xprt_done_cb function is always called
prior to using mux's recv/send functions.
2017-10-31 18:03:23 +01:00
Christopher Faulet
5b51755aef MEDIUM: threads/lb: Make LB algorithms (lb_*.c) thread-safe
A lock for LB parameters has been added inside the proxy structure and atomic
operations have been used to update server variables releated to lb.

The only significant change is about lb_map. Because the servers status are
updated in the sync-point, we can call recalc_server_map function synchronously
in map_set_server_status_up/down function.
2017-10-31 13:58:31 +01:00
Christopher Faulet
29f77e846b MEDIUM: threads/server: Add a lock per server and atomically update server vars
The server's lock is use, among other things, to lock acces to the active
connection list of a server.
2017-10-31 13:58:31 +01:00
Christopher Faulet
40a007cf2a MEDIUM: threads/server: Make connection list (priv/idle/safe) thread-safe
For now, we have a list of each type per thread. So there is no need to lock
them. This is the easiest solution for now, but not the best one because there
is no sharing between threads. An idle connection on a thread will not be able
be used by a stream on another thread. So it could be a good idea to rework this
patch later.
2017-10-31 13:58:30 +01:00
Christopher Faulet
ff8abcd31d MEDIUM: threads/proxy: Add a lock per proxy and atomically update proxy vars
Now, each proxy contains a lock that must be used when necessary to protect
it. Moreover, all proxy's counters are now updated using atomic operations.
2017-10-31 13:58:30 +01:00
Willy Tarreau
f098fd061f MINOR: backend: use conn_full_close() instead of conn_force_close()
There's no point in using conn_force_close() in outgoing connect()
since XPRT_TRACKED is not set so both functions are equivalent.
2017-10-22 09:54:18 +02:00
Willy Tarreau
ff2b7afe0b MINOR: server: add the srv_queue() sample fetch method
srv_queue([<backend>/]<server>) : integer
  Returns an integer value corresponding to the number of connections currently
  pending in the designated server's queue. If <backend> is omitted, then the
  server is looked up in the current backend. It can sometimes be used together
  with the "use-server" directive to force to use a known faster server when it
  is not much loaded. See also the "srv_conn", "avg_queue" and "queue" sample
  fetch methods.
2017-10-13 11:47:18 +02:00
Emeric Brun
52a91d3d48 MEDIUM: check: server states and weight propagation re-work
The server state and weight was reworked to handle
"pending" values updated by checks/CLI/LUA/agent.
These values are commited to be propagated to the
LB stack.

In further dev related to multi-thread, the commit
will be handled into a sync point.

Pending values are named using the prefix 'next_'
Current values used by the LB stack are named 'cur_'
2017-09-05 15:23:16 +02:00
Christopher Faulet
8fe4891b11 MINOR: backends: Make get_server_* functions explicitly static
Not used outside.
2017-09-05 10:20:00 +02:00
Christopher Faulet
f0614e8111 MINOR: backends: Change get_server_sh/get_server_uh into private function 2017-06-27 14:38:02 +02:00
Nenad Merdanovic
2754fbcfd6 CLEANUP: Replace repeated code to count usable servers with be_usable_srv()
2 places were using an open-coded implementation of this function to count
available servers. Note that the avg_queue_size() fetch didn't check that
the proxy was in STOPPED state so it would possibly return a wrong server
count here but that wouldn't impact the returned value.

Signed-off-by: Nenad Merdanovic <nmerdan@haproxy.com>
2017-03-13 18:26:05 +01:00
Nenad Merdanovic
b7e7c4720a MINOR: Add nbsrv sample converter
This is like the nbsrv() sample fetch function except that it works as
a converter so it can count the number of available servers of a backend
name retrieved using a sample fetch or an environment variable.

Signed-off-by: Nenad Merdanovic <nmerdan@haproxy.com>
2017-03-13 18:26:05 +01:00
Willy Tarreau
04276f3d6e MEDIUM: server: split the address and the port into two different fields
Keeping the address and the port in the same field causes a lot of problems,
specifically on the DNS part where we're forced to cheat on the family to be
able to keep the port. This causes some issues such as some families not being
resolvable anymore.

This patch first moves the service port to a new field "svc_port" so that the
port field is never used anymore in the "addr" field (struct sockaddr_storage).
All call places were adapted (there aren't that many).
2017-01-06 19:29:33 +01:00
Olivier Doucet
1ca1b6fe3c BUG/MINOR: option prefer-last-server must be ignored in some case
when using "option prefer-last-server", we may not always stay on
the same backend if option balance told us otherwise.
For example, backend may change in the following cases:
balance hdr()
balance rdp-cookie
balance source
balance uri
balance url_param

[wt: backport this to 1.7 and 1.6]
2017-01-02 14:26:22 +01:00
Marcin Deranek
57b877147d BUG/MINOR: backend: nbsrv() should return 0 if backend is disabled
According to nbsrv() documentation this fetcher should return "an
integer value corresponding to the number of usable servers".
In case backend is disabled none of servers is usable, so I believe
fetcher should return 0.

This patch should be backported to 1.7, 1.6, 1.5.
2016-12-23 00:09:12 +01:00
Willy Tarreau
a261e9b094 CLEANUP: connection: remove all direct references to raw_sock and ssl_sock
Now we exclusively use xprt_get(XPRT_RAW) instead of &raw_sock or
xprt_get(XPRT_SSL) for &ssl_sock. This removes a bunch of #ifdef and
include spread over a number of location including backend, cfgparse,
checks, cli, hlua, log, server and session.
2016-12-22 23:26:38 +01:00
Marcin Deranek
d2471c2bdc MINOR: proxy: Add fe_name/be_name fetchers next to existing fe_id/be_id
These 2 patches add ability to fetch frontend/backend name in your
logic, so they can be used later to make routing decisions (fe_name) or
taking some actions based on backend which responded to request (be_name).
In our case we needed a fetcher to be able to extract information we
needed from frontend name.
2016-12-12 15:10:43 +01:00
Tim Düsterhus
4896c440b3 DOC: Spelling fixes
[wt: this contains spelling fixes for both doc and code comments,
 should be backported, ignoring the parts which don't apply]
2016-11-29 07:29:57 +01:00
Willy Tarreau
b3e111b4fd BUG/MEDIUM: proxy: return "none" and "unknown" for unknown LB algos
When a backend doesn't use any known LB algorithm, backend_lb_algo_str()
returns NULL. It used to cause "nil" to be printed in the stats dump
since version 1.4 but causes 1.7 to try to parse this NULL to encode
it as a CSV string, causing a crash on "show stat" in this case.

The only situation where this can happen is when "transparent" or
"dispatch" are used in a proxy, in which case the LB algorithm is
BE_LB_ALGO_NONE. Thus now we explicitly report "none" when this
situation is detected, and we preventively report "unknown" if any
unknown algorithm is detected, which may happen if such an algo is
added in the future and the function is not updated.

This fix must be backported to 1.7 and may be backported as far as
1.4, though it has less impact there.
2016-11-26 15:58:27 +01:00
Willy Tarreau
6fb8dc1a5a MINOR: server: do not emit warnings/logs/alerts on server state changes at boot
We'll have to use srv_set_admin_flag() to propagate some server flags
during the startup, and we don't want the resulting actions to cause
warnings, logs nor e-mail alerts to be generated since we're just applying
the config or a state file. So let's condition these notifications to the
fact that we're starting.
2016-11-07 14:31:45 +01:00
Andrew Rodland
13d5ebb913 MINOR: server: compute a "cumulative weight" to allow chash balancing to hit its target
For active servers, this is the sum of the eweights of all active
servers before this one in the backend, and
[srv->cumulative_weight .. srv_cumulative_weight + srv_eweight) is a
space occupied by this server in the range [0 .. lbprm.tot_wact), and
likewise for backup servers with tot_wbck. This allows choosing a
server or a range of servers proportional to their weight, by simple
integer comparison.

Signed-off-by: Andrew Rodland <andrewr@vimeo.com>
2016-10-25 20:21:32 +02:00
Willy Tarreau
2e0565cc09 BUG/MAJOR: server: the "sni" directive could randomly cause trouble
The "sni" server directive does some bad stuff on many occasions because
it works on a sample of type string and limits len to size-1 by hand. The
problem is that size used to be zero on many occasions before the recent
changes to smp_dup() and that it effectively results in setting len to -1
and writing the zero byte *before* the string (and not terminating the
string).

This patch makes use of the recently introduced smp_make_safe() to address
this issue.

This fix must be backported to 1.6.
2016-08-09 14:30:57 +02:00
Willy Tarreau
be508f1580 BUG/MAJOR: samples: check smp->strm before using it
Since commit 6879ad3 ("MEDIUM: sample: fill the struct sample with the
session, proxy and stream pointers") merged in 1.6-dev2, the sample
contains the pointer to the stream and sample fetch functions as well
as converters use it heavily.

The problem is that earlier commit 87b0966 ("REORG/MAJOR: session:
rename the "session" entity to "stream"") had split the session and
stream resulting in the possibility for smp->strm to be NULL before
the stream was initialized. This is what happens in tcp-request
connection rulesets, as discovered by Baptiste.

The sample fetch functions must now check that smp->strm is valid
before using it. An alternative could consist in using a dummy stream
with nothing in it to avoid some checks but it would only result in
deferring them to the next step anyway, and making it harder to detect
that a stream is valid or the dummy one.

There is still an issue with variables which requires a complete
independant fix. They use strm->sess to find the session with strm
possibly NULL and passed as an argument. All call places indirectly
use smp->strm to build strm. So the problem is there but the API needs
to be changed to remove this duplicate argument that makes it much
harder to know what pointer to use.

This fix must be backported to 1.6, as well as the next one fixing
variables.
2016-03-10 16:42:58 +01:00
Willy Tarreau
0aae4806a3 BUG/MAJOR: http-reuse: fix risk of orphaned connections
There is a bug in connect_server() : we use si_attach_conn() to offer
the current session's connection to the session we're stealing the
connection from. Unfortunately, si_attach_conn() uses the standard data
connection operations while here we need to use the idle connection
operations.

This results in a situation where when the server's idle timeout strikes,
the read0 is silently ignored, causes the response channel to be shut down
for reads, and the connection remains attached. Next attempt to send a
request when using this connection simply results in nothing being done
because we try to send over an already closed connection. Worse, if the
client aborts, then no timeout remains at all and the session waits
forever and remains assigned to the server.

A more-or-less easy way to reproduce this bug is to have two concurrent
streams each connecting to a different server with "http-reuse aggressive",
typically a cache farm using a URL hash :

   stream1: GET /1 HTTP/1.1
   stream2: GET /2 HTTP/1.1
   stream1: GET /2 HTTP/1.1
   wait for the server 1's connection to timeout
   stream2: GET /1 HTTP/1.1

The connection hangs here, and "show sess all" shows a closed connection
with a SHUTR on the response channel.

The fix is very simple though not optimal. It consists in calling
si_idle_conn() again after attaching the connection. But in practise
it should not be done like this. The real issue is that there's no way
to cleanly attach a connection to a stream interface without changing
the connection's operations. So the API clearly needs to be revisited
to make such operations easier.

Many thanks to Yves Lafon from W3C for providing lots of useful dumps
and testing patches to help figure the root cause!

This fix must be backported to 1.6.
2016-02-03 21:23:08 +01:00
Willy Tarreau
29fbe51490 MAJOR: tproxy: remove support for cttproxy
This was the first transparent proxy technology supported by haproxy
circa 2005 but it was obsoleted in 2007 by Tproxy 4.0 which removed a
lot of the earlier versions' shortcomings and was finally merged into
the kernel. Since nobody has been using cttproxy for many years now
and nobody has even just tried to compile the files, it's time to
remove it. The doc was updated as well.
2015-08-20 19:35:14 +02:00
Thierry FOURNIER
136f9d34a9 MINOR: samples: rename union from "data" to "u"
The union name "data" is a little bit heavy while we read the source
code because we can read "data.data.sint". The rename from "data" to "u"
makes the read easiest like "data.u.sint".
2015-08-20 17:13:46 +02:00
Thierry FOURNIER
8c542cac07 MEDIUM: samples: Use the "struct sample_data" in the "struct sample"
This patch remove the struct information stored both in the struct
sample_data and in the striuct sample. Now, only thestruct sample_data
contains data, and the struct sample use the struct sample_data for storing
his own data.
2015-08-20 17:13:46 +02:00
Willy Tarreau
449d74a906 MEDIUM: backend: add the "http-reuse aggressive" strategy
This strategy is less extreme than "always", it only dispatches first
requests to validated reused connections, and moves a connection from
the idle list to the safe list once it has seen a second request, thus
proving that it could be reused.
2015-08-06 16:29:01 +02:00
Willy Tarreau
161d45ffc7 MEDIUM: backend: implement "http-reuse safe"
The "safe" mode consists in picking existing connections only when
processing a request that's not the first one from a connection. This
ensures that in case where the server finally times out and closes, the
client can decide to replay idempotent requests.
2015-08-06 11:50:53 +02:00
Willy Tarreau
efb90f9dd3 MAJOR: backend: improve the connection reuse mechanism
Now instead of closing the existing connection attached to the
stream interface, we first check if the one we pick was attached to
another stream interface, in which case the connections are swapped
if possible (eg: if the current connection is not private). That way
the previous connection remains attached to an existing session and
significantly increases the chances of being reused.
2015-08-06 11:41:06 +02:00
Willy Tarreau
8dff998b91 MAJOR: backend: initial work towards connection reuse
In connect_server(), if we don't have a connection attached to the
stream-int, we first look into the server's idle_conns list and we
pick the first one there, we detach it from its owner if it had one.
If we used to have a connection, we close it.

This mechanism works well but doesn't scale : as servers increase,
the likeliness that the connection attached to the stream interface
doesn't match the server and gets closed increases.
2015-08-06 11:34:21 +02:00
Willy Tarreau
387ebf84dd MINOR: connection: add a new flag CO_FL_PRIVATE
This flag is set on an outgoing connection when this connection gets
some properties that must not be shared with other connections, such
as dynamic transparent source binding, SNI or a proxy protocol header,
or an authentication challenge from the server. This will be needed
later to implement connection reuse.
2015-08-06 11:14:17 +02:00
Willy Tarreau
323a2d925c MEDIUM: stream-int: queue idle connections at the server
Now we get a per-server list of all idle connections. That way we'll
be able to reclaim them upon shortage later.
2015-08-06 11:06:25 +02:00
Willy Tarreau
973a54235f MEDIUM: stream-int: simplify si_alloc_conn()
Since we now always call this function with the reuse parameter cleared,
let's simplify the function's logic as it cannot return the existing
connection anymore. The savings on this inline function are appreciable
(240 bytes) :

$ size haproxy.old haproxy.new
   text    data     bss     dec     hex filename
1020383   40816   36928 1098127  10c18f haproxy.old
1020143   40816   36928 1097887  10c09f haproxy.new
2015-08-05 21:51:09 +02:00
Willy Tarreau
c12b5e663d MEDIUM: backend: don't call si_alloc_conn() when we reuse a valid connection
connect_server() already does most of the check that is done again in
si_alloc_conn(), so let's simply reuse the existing connection instead
of calling the function again. It will also simplify the connection
reuse.

Indeed, for reuse to be set, it also requires srv_conn to be valid. In the
end, the only situation where we have to release the existing connection
and allocate a new one is when reuse == 0.
2015-08-05 21:42:12 +02:00
Willy Tarreau
7b00492ce3 CLEANUP: backend: factor out objt_server() in connect_server()
objt_server() is called multiple times at various places while some
places already make use of srv for this. Let's move the call at the
top of the function and use it all over the place.
2015-08-05 10:12:47 +02:00
Thierry FOURNIER
07ee64ef4d MAJOR: sample: converts uint and sint in 64 bits signed integer
This patch removes the 32 bits unsigned integer and the 32 bit signed
integer. It replaces these types by a unique type 64 bit signed.

This makes easy the usage of integer and clarify signed and unsigned use.
With the previous version, signed and unsigned are used ones in place of
others, and sometimes the converter loose the sign. For example, divisions
are processed with "unsigned", if one entry is negative, the result is
wrong.

Note that the integer pattern matching and dotted version pattern matching
are already working with signed 64 bits integer values.

There is one user-visible change : the "uint()" and "sint()" sample fetch
functions which used to return a constant integer have been replaced with
a new more natural, unified "int()" function. These functions were only
introduced in the latest 1.6-dev2 so there's no impact on regular
deployments.
2015-07-22 00:48:23 +02:00
Willy Tarreau
732eac41f4 MEDIUM: ssl: add sni support on the server lines
The new "sni" server directive takes a sample fetch expression and
uses its return value as a hostname sent as the TLS SNI extension.
A typical use case consists in forwarding the front connection's SNI
value to the server in a bridged HTTPS forwarder :

   sni ssl_fc_sni
2015-07-10 11:43:15 +02:00
Thierry FOURNIER
0786d05a04 MEDIUM: sample: change the prototype of sample-fetches functions
This patch removes the "opt" entry from the prototype of the
sample-fetches fucntions. This permits to remove some weight
in the prototype call.
2015-05-11 20:03:08 +02:00
Thierry FOURNIER
0a9a2b8cec MEDIUM: sample change the prototype of sample-fetches and converters functions
This patch removes the structs "session", "stream" and "proxy" from
the sample-fetches and converters function prototypes.

This permits to remove some weight in the prototype call.
2015-05-11 20:01:42 +02:00
Willy Tarreau
f69d4ff006 BUG/MAJOR: http: prevent risk of reading past end with balance url_param
The get_server_ph_post() function assumes that the buffer is contiguous.
While this is true for all the header part, it is not necessarily true
for the end of data the fit in the reserve. In this case there's a risk
to read past the end of the buffer for a few hundred bytes, and possibly
to crash the process if what follows is not mapped.

The fix consists in truncating the analyzed length to the length of the
contiguous block that follows the headers.

A config workaround for this bug would be to disable balance url_param.

This fix must be backported to 1.5. It seems 1.4 did have the check.
2015-05-02 00:10:43 +02:00
Willy Tarreau
d0d8da989b MINOR: stream: provide a few helpers to retrieve frontend, listener and origin
Expressions are quite long when using strm_sess(strm)->whatever, so let's
provide a few helpers : strm_fe(), strm_li(), strm_orig().
2015-04-06 11:37:29 +02:00
Willy Tarreau
192252e2d8 MAJOR: sample: pass a pointer to the session to each sample fetch function
Many such function need a session, and till now they used to dereference
the stream. Once we remove the stream from the embryonic session, this
will not be possible anymore.

So as of now, sample fetch functions will be called with this :

   - sess = NULL,  strm = NULL                     : never
   - sess = valid, strm = NULL                     : tcp-req connection
   - sess = valid, strm = valid, strm->txn = NULL  : tcp-req content
   - sess = valid, strm = valid, strm->txn = valid : http-req / http-res
2015-04-06 11:37:25 +02:00
Willy Tarreau
15e91e1b36 MAJOR: sample: don't pass l7 anymore to sample fetch functions
All of them can now retrieve the HTTP transaction *if it exists* from
the stream and be sure to get NULL there when called with an embryonic
session.

The patch is a bit large because many locations were touched (all fetch
functions had to have their prototype adjusted). The opportunity was
taken to also uniformize the call names (the stream is now always "strm"
instead of "l4") and to fix indent where it was broken. This way when
we later introduce the session here there will be less confusion.
2015-04-06 11:35:53 +02:00
Willy Tarreau
eee5b51248 MAJOR: http: move http_txn out of struct stream
Now this one is dynamically allocated. It means that 280 bytes of memory
are saved per TCP stream, but more importantly that it will become
possible to remove the l7 pointer from fetches and converters since
it will be deduced from the stream and will support being null.

A lot of care was taken because it's easy to forget a test somewhere,
and the previous code used to always trust s->txn for being valid, but
all places seem to have been visited.

All HTTP fetch functions check the txn first so we shouldn't have any
issue there even when called from TCP. When branching from a TCP frontend
to an HTTP backend, the txn is properly allocated at the same time as the
hdr_idx.
2015-04-06 11:35:52 +02:00
Willy Tarreau
9ad7bd48d2 MEDIUM: session: use the pointer to the origin instead of s->si[0].end
When s->si[0].end was dereferenced as a connection or anything in
order to retrieve information about the originating session, we'll
now use sess->origin instead so that when we have to chain multiple
streams in HTTP/2, we'll keep accessing the same origin.
2015-04-06 11:34:29 +02:00
Willy Tarreau
e36cbcb3b0 MEDIUM: stream: move the frontend's pointer to the session
Just like for the listener, the frontend is session-wide so let's move
it to the session. There are a lot of places which were changed but the
changes are minimal in fact.
2015-04-06 11:23:58 +02:00
Willy Tarreau
e7dff02dd4 REORG/MEDIUM: stream: rename stream flags from SN_* to SF_*
This is in order to keep things consistent.
2015-04-06 11:23:57 +02:00
Willy Tarreau
87b09668be REORG/MAJOR: session: rename the "session" entity to "stream"
With HTTP/2, we'll have to support multiplexed streams. A stream is in
fact the largest part of what we currently call a session, it has buffers,
logs, etc.

In order to catch any error, this commit removes any reference to the
struct session and tries to rename most "session" occurrences in function
names to "stream" and "sess" to "strm" when that's related to a session.

The files stream.{c,h} were added and session.{c,h} removed.

The session will be reintroduced later and a few parts of the stream
will progressively be moved overthere. It will more or less contain
only what we need in an embryonic session.

Sample fetch functions and converters will have to change a bit so
that they'll use an L5 (session) instead of what's currently called
"L4" which is in fact L6 for now.

Once all changes are completed, we should see approximately this :

   L7 - http_txn
   L6 - stream
   L5 - session
   L4 - connection | applet

There will be at most one http_txn per stream, and a same session will
possibly be referenced by multiple streams. A connection will point to
a session and to a stream. The session will hold all the information
we need to keep even when we don't yet have a stream.

Some more cleanup is needed because some code was already far from
being clean. The server queue management still refers to sessions at
many places while comments talk about connections. This will have to
be cleaned up once we have a server-side connection pool manager.
Stream flags "SN_*" still need to be renamed, it doesn't seem like
any of them will need to move to the session.
2015-04-06 11:23:56 +02:00
Willy Tarreau
350f487300 CLEANUP: session: simplify references to chn_{prod,cons}(&s->{req,res})
These 4 combinations are needlessly complicated since the session already
has direct access to the associated stream interfaces without having to
check an indirect pointer.
2015-03-11 20:41:47 +01:00