Commit Graph

160 Commits

Author SHA1 Message Date
Willy Tarreau
66435e5f63 CLEANUP: applet: use the appctx's endp instead of cs->endp
The few applets that set CS_EP_EOI or CS_EP_ERROR used to set it on the
endpoint retrieved from the conn_stream while it's already available on
the appctx itself. Better use the appctx one to limit the unneeded
interactions between the two sides.
2022-05-13 14:28:46 +02:00
Willy Tarreau
1eea6657fb CLEANUP: httpclient: do not use the appctx.ctx anymore
The httpclient already uses its own pointer and only used to store this
single pointer into the appctx.ctx field. Let's just move it to the
svcctx and remove this entry from the appctx union.
2022-05-06 18:13:36 +02:00
Willy Tarreau
89a7c41e24 CLEANUP: httpclient/cli: use a locally-defined context instead of ctx.cli
The httpclient's CLI uses ctx.cli.i0 for its flags and .p0 for the client
instance. Let's have a locally defined structure for this so that we don't
need the generic cli variables anymore.
2022-05-06 18:13:36 +02:00
William Lallemand
7867f63313 MEDIUM: resolvers: create a "default" resolvers section at startup
Try to create a "default" resolvers section at startup, but does not
display any error nor warning. This section is initialized using the
/etc/resolv.conf of the system.

This is opportunistic and with no guarantee that it will work (but it should
on most systems).

This is useful for the httpclient as it allows to use the DNS resolver
without any configuration in most of the cases.

The function is called from the httpclient_pre_check() function to
ensure than we tried to create the section before trying to initiate the
httpclient. But it is also called from the resolvers.c to ensure the
section is created when the httpclient init was disabled.
2022-05-06 17:02:15 +02:00
William Lallemand
7c5a7ef32b MINOR: httpclient: allow ipv4 or ipv6 preference for resolving
The httpclient.resolvers.prefer global keyword allows to configure an
ipv4 or ipv6 preference when resolving. This could be useful in
environment where the ipv6 is not supported.
2022-05-04 16:14:42 +02:00
William Lallemand
8a734cbae3 MINOR: httpclient: configure the resolvers section to use
By default the httpclient uses the resolvers section whose ID is
"default", the httpclient.resolvers.id global option allows to configure
another section to use.
2022-05-04 16:14:35 +02:00
William Lallemand
683fbb86be MINOR: httpclient: allow to configure the ca-file
The global keyword httpclient.ssl.ca-file allows to configure the
ca-file used for the httpclient verify.
2022-05-04 16:14:35 +02:00
William Lallemand
6fce46a910 MEDIUM: httpclient: hard-error when SSL is configured
The hard_error_ssl flag is set when the configuration is explicitely
done for the ssl in the httpclient.

If no configuration was made, the features are simply disabled and no
alert is emitted.
2022-05-04 16:13:17 +02:00
William Lallemand
85af49c5c8 MINOR: httpclient: cleanup the error handling in init
Cleanup the error handling in the initialization so we rely on the
ERR_CODE and use memprintf() to set the errmsg before printing it at the
end of the functions.
2022-05-04 14:33:57 +02:00
William Lallemand
9ff95e2269 MINOR: httpclient: rename dash by dot in global option
Rename the httpclient-ssl-verify into httpclient.ssl.verify.
2022-05-04 13:52:29 +02:00
William Lallemand
853327390e MINOR: httpclient: handle unix and other socket types in dst
httpclient_set_dst() allows one to set the destination address instead
of using the one in the URL or resolving one from the host.
This function also support other types of socket like sockpair@, unix@,
anything that could be used on a server line.

In order to still support this behavior, the address must be set on the
backend in this particular case because the frontend connection does not
support anything other than ipv4 or ipv6.
2022-05-04 11:21:01 +02:00
William Lallemand
0e23526a2c CLEANUP: httpclient: remove the comment about resolving
remove the comment of httpclient_start which states there is no
resolver.
2022-05-04 11:21:01 +02:00
William Lallemand
1218d19921 MEDIUM: httpclient: allow address and port change for resolving
To allow the http-request set-dst to work for the httpclient DNS
resolving, some changes need to be done:

- The destination address need to be set in the frontend (s->csf->dst)
  instead of the backend (s->csb->dst) to be able to use
  tcp_action_req_set_dst()

- SRV_F_MAPPORTS need to be set on the proxy in order to allow the port
  change in alloc_dst_address()
2022-05-04 11:21:01 +02:00
William Lallemand
5392ff6e3c MEDIUM: httpclient: http-request rules for resolving
The httpclient_resolve_init() adds http-request actions which does the
resolving using the Host header of the HTTP request.

The parse_http_req_cond function is directly used over an array of http
rules.

The do-resolve rule uses the "default" resolvers section. If this
section does not exist in the configuration, the resolving is disabling.
2022-05-04 11:21:01 +02:00
William Lallemand
7f1df8ff1a MEDIUM: httpclient: remove url2sa to use a more flexible parser
The httpclient DNS resolver will need a more efficient URL parser which
splits the URL into parts and does not try to resolve.

httpclient_spliturl uses the http_uri_parser in order to split the URL
into several ist.

The result of the function host part is then processed into str2ip2(),
and fails if it it not an IP, allowing us to resolve the domain later.
2022-05-04 11:21:01 +02:00
Willy Tarreau
03bd3952a6 MEDIUM: stream: remove the confusing SF_ADDR_SET flag
This flag is no longer needed now that it must always match the presence
of a destination address on the backend conn_stream. Worse, before previous
patch, if it were to be accidently removed while the address is present, it
could result in a leak of that address since alloc_dst_address() would first
be called to flush it.

Its usage has a long history where addresses were stored in an area shared
with the connection, but as this is no longer the case, there's no reason
for putting this burden onto application-level code that should not focus
on setting obscure flags.

The only place where that made a small difference is in the dequeuing code
in case of queue redistribution, because previously the code would first
clear the flag, and only later when trying to deal with the queue, would
release the address. It's not even certain whether there would exist a
code path going to connect_server() without calling pendconn_dequeue()
first (e.g. retries on queue timeout maybe?).

Now the pendconn_dequeue() code will rely on SF_ASSIGNED to decide to
clear and release the address, since that flag is always set while in
a server's queue, and its clearance implies that we don't want to keep
the address. At least it remains consistent and there's no more risk of
leaking it.
2022-05-02 16:56:01 +02:00
Christopher Faulet
0055d5693e MINOR: httpclient: Don't use co_set_data() to decrement output
The use of co_set_data() should be strictly limited to setting the amount of
existing data to be transmitted. It ought not be used to decrement the
output after the data have left the buffer, because doing so involves
performing incorrect calculations using co_data() that still comprises data
that are not in the buffer anymore. Let's use c_rew() for this, which is
made exactly for this purpose, i.e. decrement c->output by as much as
requested.
2022-04-29 14:12:42 +02:00
Christopher Faulet
6b4f1f64a8 BUG/MINOR: httpclient: Count metadata in size to transfer via htx_xfer_blks()
When HTX blocks are transfer from the HTTP client context to the request
channel, via htx_xfer_blks() function, the metadata must also be counted, in
addition to the data size. Otherwise, expected payload size will not be
copied because the metadata of an HTX block (8 bytes) will be reserved. And
if the payload size is lower than 8 bytes, nothing will be copied. Thus only
a zero-copy will be able to copy the payload.

This issue is 2.6-specific, no backport is needed.
2022-04-29 14:12:42 +02:00
Christopher Faulet
534645d6c0 BUG/MEDIUM: httpclient: Fix loop consuming HTX blocks from the response channel
When the HTTP client consumes the response, it loops on the HTX message to
copy blocks content and it removes blocks by calling htx_remove_blk(). But
this function removes a block and returns the next one in the HTX
message. The result must be used instead of using htx_get_next(). It is
especially important because the block used in htx_get_next() loop was
removed. It only works because the message is not defragmented during the
loop.

In addition, the loop on the response was simplified to iter on blocks
instead of positions.

This patch must be backported to 2.5.
2022-04-29 14:12:42 +02:00
William Lallemand
04994de642 BUG/MINOR: httpclient/ssl: use the correct verify constant
The SSL_SERVER_VERIFY_* constants were incorrectly set on the httpclient
server verify. The right constants are SSL_SOCK_VERIFY_* .

This could cause issues when using "httpclient-ssl-verify" or when the
SSL certificates can't be loaded.

No backport needed
2022-04-28 19:35:21 +02:00
William Lallemand
f1344b3cee MEDIUM: httpclient: re-enable the verify by default
Since the httpclient verify now has a fallback which disable the SSL in
the httpclient without exiting haproxy at startup, we can safely
re-enable it by default.

It could still be disabled with "httpclient-ssl-verify none".
2022-04-26 16:15:23 +02:00
William Lallemand
4006b0f130 MEDIUM: httpclient: disable SSL when the ca-file couldn't be loaded
Emit a warning when the ca-file couldn't be loaded for the httpclient,
and disable the SSL of the httpclient.

We must never be in a case where the verify is disabled without any
configuration, so better disable the SSL completely.

Move the check on the scheme above the initialization of the applet so
we could abort before initializing the appctx.
2022-04-26 16:15:23 +02:00
William Lallemand
eaa703ef06 MEDIUM: httpclient/ssl: verify is configurable and disabled by default
Disable temporary the SSL verify by default in the httpclient. The
initialization of the @system-ca during the init of the httpclient is a
problem in some cases.

The verify can be reactivated with "httpclient-ssl-verify required" in
the global section.
2022-04-22 18:05:17 +02:00
William Lallemand
c6ceba3170 MINOR: httpclient/mworker: disable in the master process
Disable the httpclient in the master process.
2022-04-22 16:49:53 +02:00
William Lallemand
cf5cb0b524 MEDIUM: httpclient/ssl: verify required
The httpclient HTTPS requests now enable the "verify required" option.
To achieve this, the "@system-ca" ca-file is configured in the
httpclient ssl server. Which means all the system CAs will be loaded at
haproxy startup.
2022-04-22 15:45:47 +02:00
William Lallemand
2c8b0842bb MEDIUM: httpclient: change the init sequence
Change the init order of the httpclient, a different init sequence is
required to allow a more complicated init.

The init is splitted in two parts:

- the first part is executed before config_check_validity(), which
  allows to create proxy and more advanced stuff than STG_INIT, because
  we might want to use stuff already initialized in haproxy (trash
  buffers for example)

- the second part is executed after the config_check_validity(),
  currently it is used for the log configuration.
2022-04-22 15:45:47 +02:00
Christopher Faulet
eb50c01fef MINOR: conn-stream: Make cs_detach_* private and use cs_destroy() from outside
A conn-stream is never detached from an endpoint or an application alone,
except on a reset. Thus, to avoid any error, these functions are now
private. And cs_destroy() function is added to destroy a conn-stream. This
function is called when a stream is released, on the front and back
conn-streams, and when a health-check is finished.
2022-04-22 14:32:30 +02:00
Willy Tarreau
e1e9f6bbe5 BUG/MEDIUM: logs: fix http-client's log srv initialization
As anticipated in commit 211ea252d ("BUG/MINOR: logs: fix logsrv leaks
on clean exit"), there were indeed other corner cases that were not
properly covered. Setting the http client's ring_name to NULL make the
sink lookup crash on startup in sink_find () with a config as simple as:

    global
        log ring@buf0 local0

The fields must be properly initialized (both config file name and
the ring_name). This only needs to be backported if/when the commit
above is backported.
2022-04-22 09:40:44 +02:00
Christopher Faulet
67df95a8a2 BUILD: http-client: Avoid dead code when compiled without SSL support
When an HTTP client is started on an HAProxy compiled without the SSL
support, an error is triggered when HTTPS is used. In this case, the freshly
created conn-stream is released. But this code is specific to the non-SSL
part. Thus it is moved the in right #if/#else section.

This patch should fix the issue #1655.
2022-04-14 12:02:35 +02:00
Christopher Faulet
6b0a0fb2f9 CLEANUP: tree-wide: Remove any ref to stream-interfaces
Stream-interfaces are gone. Corresponding files can be safely be removed. In
addition, comments are updated accordingly.
2022-04-13 15:10:16 +02:00
Christopher Faulet
a0bdec350f MEDIUM: stream-int/conn-stream: Move blocking flags from SI to CS
Remaining flags and associated functions are move in the conn-stream
scope. These flags are added on the endpoint and not the conn-stream
itself. This way it will be possible to get them from the mux or the
applet. The functions to get or set these flags are renamed accordingly with
the "cs_" prefix and updated to manipualte a conn-stream instead of a
stream-interface.
2022-04-13 15:10:15 +02:00
Christopher Faulet
da098e6c17 MINOR: stream-int/conn-stream: Move si_shut* and si_chk* in conn-stream scope
si_shutr(), si_shutw(), si_chk_rcv() and si_chk_snd() are moved in the
conn-stream scope and renamed, respectively, cs_shutr(), cs_shutw(),
cs_chk_rcv(), cs_chk_snd() and manipulate a conn-stream instead of a
stream-interface.
2022-04-13 15:10:15 +02:00
Christopher Faulet
8abe712749 MINOR: stream-int: Remove SI_FL_NOLINGER/NOHALF to rely on CS flags instead
Flags to disable lingering and half-close are now handled at the conn-stream
level. Thus SI_FL_NOLINGER and SI_FL_NOHALF stream-int flags are replaced by
CS_FL_NOLINGER and CS_FL_NOHALF conn-stream flags.
2022-04-13 15:10:14 +02:00
Christopher Faulet
8da67aae3e MEDIUM: stream-int/conn-stream: Move src/dst addresses in the conn-stream
The source and destination addresses at the applicative layer are moved from
the stream-interface to the conn-stream. This simplifies a bit the code and
it is a logicial step to remove the stream-interface.
2022-04-13 15:10:14 +02:00
Christopher Faulet
908628c4c0 MEDIUM: tree-wide: Use CS util functions instead of SI ones
At many places, we now use the new CS functions to get a stream or a channel
from a conn-stream instead of using the stream-interface API. It is the
first step to reduce the scope of the stream-interfaces. The main change
here is about the applet I/O callback functions. Before the refactoring, the
stream-interface was the appctx owner. Thus, it was heavily used. Now, as
far as possible,the conn-stream is used. Of course, it remains many calls to
the stream-interface API.
2022-04-13 15:10:14 +02:00
Christopher Faulet
b041b23ae4 MEDIUM: conn-stream: Move remaning flags from CS to endpoint
All old flags CS_FL_* are now moved in the endpoint scope and renamed
CS_EP_* accordingly. It is a systematic replacement. There is no true change
except for the health-check and the endpoint reset. Here it is a bit special
because the same conn-stream is reused. Thus, we must handle endpoint
allocation errors. To do so, cs_reset_endp() has been adapted.

Thanks to this last change, it will now be possible to simplify the
multiplexer and probably the applets too. A review must also be performed to
remove some flags in the channel or the stream-interface. The HTX will
probably be simplified too. Finally, there is now some place in the
conn-stream to move info from the stream-interface.
2022-04-13 15:10:14 +02:00
Christopher Faulet
9ec2f4dc7c MAJOR: conn-stream: Share endpoint struct between the CS and the mux/applet
The conn-stream endpoint is now shared between the conn-stream and the
applet or the multiplexer. If the mux or the applet is created first, it is
responsible to also create the endpoint and share it with the conn-stream.
If the conn-stream is created first, it is the opposite.

When the endpoint is only owned by an applet or a mux, it is called an
orphan endpoint (there is no conn-stream). When it is only owned by a
conn-stream, it is called a detached endpoint (there is no mux/applet).

The last entity that owns an endpoint is responsible to release it. When a
mux or an applet is detached from a conn-stream, the conn-stream
relinquishes the endpoint to recreate a new one. This way, the endpoint
state is never lost for the mux or the applet.
2022-04-13 15:10:14 +02:00
Christopher Faulet
a9e8b3979d MEDIUM: conn-stream: Pre-allocate endpoint to create CS from muxes and applets
It is a transient commit to prepare next changes. Now, when a conn-stream is
created from an applet or a multiplexer, an endpoint is always provided. In
addition, the API to create a conn-stream was specialized to have one
function per type.

The next step will be to share the endpoint structure.
2022-04-13 15:10:14 +02:00
Christopher Faulet
b669d684c0 MEDIUM: conn-stream: Be able to pass endpoint to create a conn-stream
It is a transient commit to prepare next changes. It is possible to pass a
pre-allocated endpoint to create a new conn-stream. If it is NULL, a new
endpoint is created, otherwise the existing one is used. There no more
change at the conn-stream level.

In the applets, all conn-stream are created with no pre-allocated
endpoint. But for multiplexers, an endpoint is systematically created before
creating the conn-stream.
2022-04-13 15:10:14 +02:00
Christopher Faulet
9388204db1 MAJOR: conn-stream: Invert conn-stream endpoint and its context
This change is only significant for the multiplexer part. For the applets,
the context and the endpoint are the same. Thus, there is no much change. For
the multiplexer part, the connection was used to set the conn-stream
endpoint and the mux's stream was the context. But it is a bit strange
because once a mux is installed, it takes over the connection. In a
wonderful world, the connection should be totally hidden behind the mux. The
stream-interface and, in a lesser extent, the stream, still access the
connection because that was inherited from the pre-multiplexer era.

Now, the conn-stream endpoint is the mux's stream (an opaque entity for the
conn-stream) and the connection is the context. Dedicated functions have
been added to attached an applet or a mux to a conn-stream.
2022-04-13 15:10:14 +02:00
Christopher Faulet
2479e5f775 MEDIUM: applet: Set the appctx owner during allocation
The appctx owner is now always a conn-stream. Thus, it can be set during the
appctx allocation. But, to do so, the conn-stream must be created first. It
is not a problem on the server side because the conn-stream is created with
the stream. On the client side, we must take care to create the conn-stream
first.

This change should ease other changes about the applets bootstrapping.
2022-04-13 15:10:13 +02:00
Christopher Faulet
dca3b5b2c6 BUG/MINOR: http_client: Don't add input data on an empty request buffer
When compiled in debug mode, a BUG_ON triggers an error when the payload is
fully transfered from the http-client buffer to the request channel
buffer. In fact, when channel_add_input() is called, the request buffer is
empty. So an error is reported when those data are directly forwarded,
because we try to add some output data on a buffer with no data.

To fix the bug, we must be sure to call channel_add_input() after the data
transfer.

The bug was introduced by the commit ccc7ee45f ("MINOR: httpclient: enable
request buffering"). So, this patch must be backported if the above commit
is backported.
2022-04-07 11:04:39 +02:00
William Lallemand
eb0d4c40ac BUG/MINOR: httpclient: end callback in applet release
In case an error provokes the release of the applet, we will never call
the end callback of the httpclient.

In the case of a lua script, it would mean that the lua task will never
be waked up after a yield, letting the lua script stuck forever.

Fix the issue by moving the callback from the end of the iohandler to
the applet release function.

Must be backported in 2.5.
2022-04-06 14:19:36 +02:00
William Lallemand
71abad050a MEDIUM: httpclient: enable l7-retry
Enable the layer-7 retry in the httpclient. This way the client will
retry upon a connection error or a timeout.
2022-04-06 11:50:10 +02:00
William Lallemand
ccc7ee45f9 MINOR: httpclient: enable request buffering
The request buffering is required for doing l7 retry. The IO handler of
the httpclient need to be rework for that.

This patch change the IO handler so it copies partially the data instead
of swapping buffer. This is needed because the b_xfer won't never work
if the destination buffer is not empty, which is the case when
buffering.
2022-04-06 11:43:01 +02:00
William Lallemand
58a81aeb91 BUG/MINOR: httpclient: CF_SHUTW_NOW should be tested with channel_is_empty()
CF_SHUTW_NOW shouldn't be a condition alone to exit the io handler, it
must be tested with the emptiness of the response channel.

Must be backported to 2.5.
2022-03-18 11:34:10 +01:00
William Lallemand
1eca894321 BUG/MINOR: httpclient: process the response when received before the end of the request
A server could reply a response with a shut before the end of the htx
transfer, in this case the httpclient would leave before computing the
received response.

This patch fixes the issue by calling the "process_data" label instead of
the "more" label which don't do the si_shut.

Must be bacported in 2.5.
2022-03-18 11:34:10 +01:00
William Lallemand
a625b03e83 BUG/MINOR: httpclient: only check co_data() instead of HTTP_MSG_DATA
Checking msg >= HTTP_MSG_DATA was useful to check if we received all the
data. However it does not work correctly in case of errors because we
don't reach this state, preventing to catch the error in the httpclient.

The consequence of this problem is that we don't get the status code of
the error response upon an error.

Fix the issue by only checking co_data().

Must be backported to 2.5.
2022-03-18 11:34:10 +01:00
Willy Tarreau
211ea252d9 BUG/MINOR: logs: fix logsrv leaks on clean exit
Log servers are a real mess because:
  - entries are duplicated using memcpy() without their strings being
    reallocated, which results in these ones not being freeable every
    time.

  - a new field, ring_name, was added in 2.2 by commit 99c453df9
    ("MEDIUM: ring: new section ring to declare custom ring buffers.")
    but it's never initialized during copies, causing the same issue

  - no attempt is made at freeing all that.

Of course, running "haproxy -c" under ASAN quickly notices that and
dumps a core.

This patch adds the missing strdup() and initialization where required,
adds a new free_logsrv() function to cleanly free() such a structure,
calls it from the proxy when iterating over logsrvs instead of silently
leaking their file names and ring names, and adds the same logsrv loop
to the proxy_free_defaults() function so that we don't leak defaults
sections on exit.

It looks a bit entangled, but it comes as a whole because all this stuff
is inter-dependent and was missing.

It's probably preferable not to backport this in the foreseable future
as it may reveal other jokes if some obscure parts continue to memcpy()
the logsrv struct.
2022-03-17 19:53:46 +01:00
William Lallemand
715c101a19 BUILD: httpclient: fix build without SSL
src/http_client.c: In function ‘httpclient_cfg_postparser’:
src/http_client.c:1065:8: error: unused variable ‘errmsg’ [-Werror=unused-variable]
 1065 |  char *errmsg = NULL;
      |        ^~~~~~
src/http_client.c:1064:6: error: unused variable ‘err_code’ [-Werror=unused-variable]
 1064 |  int err_code = 0;
      |      ^~~~~~~~

Fix the build of the httpclient without SSL, the problem was introduced
with previous patch 71e3158 ("BUG/MINOR: httpclient: send the SNI using
the host header")

Must be backported in 2.5 as well.
2022-03-16 16:39:23 +01:00
William Lallemand
71e3158395 BUG/MINOR: httpclient: send the SNI using the host header
Generate an SNI expression which uses the Host header of the request.
This is mandatory for most of the SSL servers nowadays.

Must be backported in 2.5 with the previous patch which export
server_parse_sni_expr().
2022-03-16 15:55:30 +01:00
William Lallemand
97f69c6fb5 BUG/MEDIUM: httpclient: must manipulate head, not first
The httpclient mistakenly use the htx_get_first{_blk}() functions instead
of the htx_get_head{_blk}() functions. Which could stop the httpclient
because it will be without the start line, waiting for data that won't never
come.

Must be backported in 2.5.
2022-03-14 15:10:12 +01:00
William Lallemand
c020b2505d BUG/MINOR: httpclient: remove the UNUSED block when parsing headers
Remove the UNUSED blocks when iterating on headers, we should not stop
when encountering one. We should only stop iterating once we found the
EOH block. It doesn't provoke a problem, since we don't manipulates
the headers before treating them, but it could evolve in the future.

Must be backported to 2.5.
2022-03-14 15:10:12 +01:00
William Lallemand
c8f1eb99b4 BUG/MINOR: httpclient: consume partly the blocks when necessary
Consume partly the blocks in the httpclient I/O handler when there is
not enough room in the destination buffer for the whole block or when
the block is not contained entirely in the channel's output.

It prevents the I/O handler to be stuck in cases when we need to modify
the buffer with a filter for exemple.

Must be backported in 2.5.
2022-03-14 15:10:12 +01:00
William Lallemand
2b7dc4edb0 BUG/MEDIUM: httpclient: don't consume data before it was analyzed
In httpclient_applet_io_handler(), on the response path, we don't check
if the data are in the output part of the channel, and could consume
them before they were analyzed.

To fix this issue, this patch checks for the stline and the headers if
the msg_state is >= HTTP_MSG_DATA which means the stline and headers
were analyzed. For the data part, it checks if each htx blocks is in the
output before copying it.

Must be backported in 2.5.
2022-03-14 15:10:12 +01:00
Christopher Faulet
3d4332419c BUG/MINOR: httpclient: Set conn-stream/channel EOI flags at the end of request
In HTX, HTX_FL_EOM flag is added on the message to notifiy the end of the
message was received. In addition, the producer must set CS_FL_EOI flag on
the conn-stream. If it is a mux, the stream-interface is responsible to set
CF_EOI flag on the input channel. But, for now, if the producer is an
applet, in addition to the conn-stream flag, it must also set the channel
one.

These flags are used to notify the stream that the message is finished and
no more data are expected. It is especially important when the message
itself it directly forwarded from one side to the other. Because in this
case, the stream has no way to see the HTX_FL_EOM flag on the
message. Otherwise, the stream will detect a client or a server abort,
depending on the side.

For the HTTP client, it is not really easy to diagnose this error because
there is also another bug hiding this one. All HTTP request analyzers are
not set on the input channel. This will be fixed by another patch.

This patch must be backported to 2.5. It is related to the issue #1593.
2022-03-08 16:33:56 +01:00
Christopher Faulet
cda94accb1 MAJOR: stream/conn_stream: Move the stream-interface into the conn-stream
Thanks to all previous changes, it is now possible to move the
stream-interface into the conn-stream. To do so, some SI functions are
removed and their conn-stream counterparts are added. In addition, the
conn-stream is now responsible to create and release the
stream-interface. While the stream-interfaces were inlined in the stream
structure, there is now a pointer in the conn-stream. stream-interfaces are
now dynamically allocated. Thus a dedicated pool is added. It is a temporary
change because, at the end, the stream-interface structure will most
probably disappear.
2022-02-24 11:00:03 +01:00
Christopher Faulet
b91afea91c MINOR: httpclient: Always access the stream-int via the conn-stream
To be able to move the stream-interface from the stream to the conn-stream,
all access to the SI is done via the conn-stream. This patch is limited to
the httpclient part.
2022-02-24 11:00:02 +01:00
Christopher Faulet
f835dea939 MEDIUM: conn_stream: Add a pointer to the app object into the conn-stream
In the same way the conn-stream has a pointer to the stream endpoint , this
patch adds a pointer to the application entity in the conn-stream
structure. For now, it is a stream or a health-check. It is mandatory to
merge the stream-interface with the conn-stream.
2022-02-24 11:00:02 +01:00
Christopher Faulet
86e1c3381b MEDIUM: applet: Set the conn-stream as appctx owner instead of the stream-int
Because appctx is now an endpoint of the conn-stream, there is no reason to
still have the stream-interface as appctx owner. Thus, the conn-stream is
now the appctx owner.
2022-02-24 11:00:02 +01:00
Christopher Faulet
13a35e5752 MAJOR: conn_stream/stream-int: move the appctx to the conn-stream
Thanks to previous changes, it is now possible to set an appctx as endpoint
for a conn-stream. This means the appctx is no longer linked to the
stream-interface but to the conn-stream. Thus, a pointer to the conn-stream
is explicitly stored in the stream-interface. The endpoint (connection or
appctx) can be retrieved via the conn-stream.
2022-02-24 11:00:02 +01:00
Willy Tarreau
5b4b6ca823 CLEANUP: httpclient: initialize the client in stage INIT not REGISTER
REGISTER is meant to only assemble static lists, not to initialize
code that may depend on some elements possibly initialized at this
level. For example the init code currently looks up transport protocols
such as XPRT_RAW and XPRT_SSL which ought to be themselves registered
from at REGISTER stage, and which currently work only because they're
still registered directly from a constructor. INIT is perfectly suited
for this level.
2022-02-23 17:11:33 +01:00
William Lallemand
b4a4ef6a29 MINOR: httpclient/lua: ability to set a server timeout
Add the ability to set a "server timeout" on the httpclient with either
the httpclient_set_timeout() API or the timeout argument in a request.

Issue #1470.
2022-02-23 15:11:11 +01:00
Willy Tarreau
11adb1d8fc BUG/MEDIUM: httpclient: limit transfers to the maximum available room
A bug was uncovered by commit fc5912914 ("MINOR: httpclient: Don't limit
data transfer to 1024 bytes"), it happens that callers of b_xfer() and
b_force_xfer() are expected to check for available room in the target
buffer. Previously it was unlikely to be full but now with full buffer-
sized transfers, it happens more often and in practice it is possible
to crash the process with the debug command "httpclient" on the CLI by
going beyond a the max buffer size. Other call places ought to be
rechecked by now and it might be time to rethink this API if it tends
to generalize.

This must be backported to 2.5.
2022-02-18 17:32:12 +01:00
Willy Tarreau
2c8f984441 CLEANUP: httpclient/cli: fix indentation alignment of the help message
The output was not aligned with other commands, let's fix it.
2022-02-18 16:29:50 +01:00
William Lallemand
7b2e0ee1c1 MINOR: httpclient: sets an alternative destination
httpclient_set_dst() allows to set an alternative destination address
using HAProxy addres format. This will ignore the address within the
URL.
2022-02-17 20:07:00 +01:00
William Lallemand
5085bc3103 BUG/MINOR: httpclient: reinit flags in httpclient_start()
When starting for the 2nd time a request from the same httpclient *hc
context, the flags are not reinitialized and the httpclient will stop
after the first call to the IO handler, because the END flag is always
present.

This patch also add a test before httpclient_start() to ensure we don't
start a client already started.

Must be backported in 2.5.
2022-02-17 12:59:52 +01:00
William Lallemand
de6ecc3ace BUG/MINOR: httpclient/cli: display junk characters in vsn
ist are not ended by '\0', leading to junk characters being displayed
when using %s for printing the HTTP start line.

Fix the issue by replacing %s by %.*s + istlen.

Must be backported in 2.5.
2022-02-16 11:37:02 +01:00
Christopher Faulet
fc5912914b MINOR: httpclient: Don't limit data transfer to 1024 bytes
For debug purpose, no more 1024 bytes were copied at a time. But there is no
reason to keep this limitation. Thus, it is removed.

This patch may be backported to 2.5.
2022-02-02 16:19:19 +01:00
Christopher Faulet
6ced61dd0a BUG/MEDIUM: httpclient: Xfer the request when the stream is created
Since the HTTP legacy mode was removed, it is unexpected to create an HTTP
stream without a valid request. Thanks to this change, the wait_for_request
analyzer was significatly simplified. And it is possible because HTTP
multiplexers already take care to have a valid request to create a stream.

But it means that any HTTP applet on the client side must do the same. The
httpclient client is one of them. And it is not a problem because the
request is generated before starting the applet. We must just take care to
set the right state.

For now it works "by chance", because the applet seems to be scheduled
before the stream itself. But if this change, this will lead to crash
because the stream expects to have a request when wait_for_request analyzer.

This patch should be backported to 2.5.
2022-02-02 16:19:17 +01:00
Christopher Faulet
600985df41 BUG/MINOR: httpclient: Revisit HC request and response buffers allocation
For now, these buffers are allocated when the httpclient is created and
freed when it is released. Usually, we try to avoid to keep buffer allocated
if it is not required. Empty buffers should be released ASAP. Apart for
that, there is no issue with the response side because a copy is always
performed. However, for the request side, a swap with the channel's buffer
is always performed. And there is no guarantee the channel's buffer is
allocated. Thus, after the swap, the httpclient can retrieve a null
buffer. In practice, this never happens. But this may change. And it will be
required for a futur fix.

So, now, we systematically take care to have an allocated buffer when we
want to write in it. And it is released as soon as it becomes empty.

This patch should be backported to 2.5.
2022-02-02 16:19:16 +01:00
William Lallemand
bad9c8cac4 BUG/MINOR: httpclient: set default Accept and User-Agent headers
Some servers require at least an Accept and a User-Agent header in the
request. This patch sets some default value.

Must be backported in 2.5.
2022-01-14 20:46:21 +01:00
William Lallemand
e1e045f4d7 BUG/MINOR: httpclient: don't send an empty body
Forbid the httpclient to send an empty chunked client when there is no
data to send. It does happen when doing a simple GET too.

Must be backported in 2.5.
2022-01-14 18:10:43 +01:00
William Lallemand
f03b53c81d BUG/MINOR: httpclient: allow to replace the host header
This patch allows to replace the host header generated by the
httpclient instead of adding a new one, resulting in the server replying
an error 400.

The host header is now generated from the uri only if it wasn't found in
the list of headers.

Also add a new request in the VTC file to test this.

This patch must be backported in 2.5.
2021-11-24 15:44:36 +01:00
William Lallemand
67b778418e BUG/MEDIUM: httpclient/cli: free of unallocated hc->req.uri
httpclient_new() sets the hc->req.uri ist without duplicating its
memory, which is a problem since the string in the ist could be
inaccessible at some point. The API was made to use a ist which was
allocated dynamically, but httpclient_new() didn't do that, which result
in a crash when calling istfree().

This patch fixes the problem by doing an istdup()

Fix issue #1452.
2021-11-10 17:02:50 +01:00
William Lallemand
db8a1f391d BUG/MEDIUM: httpclient: channel_add_input() must use htx->data
The httpclient uses channel_add_input() to notify the channel layer that
it must forward some data. This function was used with b_data(&req->buf)
which ask to send the size of a buffer (because of the HTX metadata
which fill the buffer completely).

This is wrong and will have the consequence of trying to send data that
doesn't exist, letting HAProxy looping at 100% CPU.

When using htx channel_add_input() must be used with the size of the htx
payload, and not the size of a buffer.

When sending the request payload it also need to sets the buffer size to
0, which is achieved with a htx_to_buf() when the htx payload is empty.
2021-11-08 17:36:31 +01:00
William Lallemand
933fe394bb BUG/MINOR: httpclient/lua: rcv freeze when no request payload
This patch fixes the receive part of the lua httpclient when no payload
was sent.

The lua task was not awoken once it jumped into
hlua_httpclient_rcv_yield(), which caused the lua client to freeze.

It works with a payload because the payload push is doing the wakeup.

A change in the state machine of the IO handler is also require to
achieve correctly the change from the REQ state to the RES state, it has
to detect if there is the right EOM flag in the request.
2021-11-08 17:36:31 +01:00
William Lallemand
0f41c384ea BUG/MINOR: httpclient: use a placeholder value for Host header
A Host header must be present for http_update_host() to success.
htx_add_header(htx, ist("Host"), IST_NULL) was used but this is not a
good idea from a semantic point of view. It also tries to make a memcpy
with a len of 0, which is unrequired.

Use an ist("h") instead as a placeholder value.

This patch fixes bug #1439.
2021-11-02 15:53:09 +01:00
William Lallemand
0da616ee18 MINOR: httpclient: request streaming with a callback
This patch add a way to handle HTTP requests streaming using a
callback.

The end of the data must be specified by using the "end" parameter in
httpclient_req_xfer().
2021-10-28 16:24:14 +02:00
Christopher Faulet
16f16afb31 MINOR: stream: Use backend stream-interface dst address instead of target_addr
target_addr field in the stream structure is removed. The backend
stream-interface destination address is now used.
2021-10-27 11:35:59 +02:00
William Lallemand
dec25c3e14 MINOR: httpclient: support payload within a buffer
httpclient_req_gen() takes a payload argument which can be use to put a
payload in the request. This payload can only fit a request buffer.

This payload can also be specified by the "body" named parameter within
the lua. httpclient.

It is also used within the CLI httpclient when specified as a CLI
payload with "<<".
2021-10-27 10:19:41 +02:00
William Lallemand
34b3a93655 MINOR: httpclient/cli: access should be only done from expert mode
Only enable the usage of the CLI HTTP client in expert mode.
2021-10-19 15:02:42 +02:00
Ilya Shipitsin
bd6b4be721 CLEANUP: assorted typo fixes in the code and comments
This is 27th iteration of typo fixes
2021-10-18 07:26:19 +02:00
Willy Tarreau
1df20428f1 BUILD: httpclient: include missing ssl_sock-t
It's needed for SSL_SOCK_VERIFY_NONE.
2021-10-07 01:36:51 +02:00
William Lallemand
2a879001b5 MINOR: httpclient: destroy checks if a client was started but not stopped
During httpclient_destroy, add a condition in the BUG_ON which checks
that the client was started before it has ended. A httpclient structure
could have been created without being started.
2021-10-06 15:15:03 +02:00
William Lallemand
4d60184887 BUG/MEDIUM: httpclient/lua: crash because of b_xfer and get_trash_chunk()
When using the lua httpclient, haproxy could crash because a b_xfer is
done in httpclient_xfer, which will do a zero-copy swap of the data in
the buffers. The ptr will then be free() by the pool.

However this can't work with a trash buffer, because the area was not
allocated from the pool buffer, so the pool is not suppose to free it
because it does not know this ptr, using -DDEBUG_MEMORY_POOLS will
result with a crash during the free.

Fix the problem by using b_force_xfer() instead of b_xfer which copy
the data instead. The problem still exist with the trash however, and
the trash API must be reworked.
2021-10-06 15:15:03 +02:00
William Lallemand
b8b1370307 MINOR: httpclient: test if started during stop_and_destroy()
If the httpclient was never started, it is safe to destroy completely
the httpclient.
2021-10-06 15:15:03 +02:00
William Lallemand
ecb83e13eb MINOR: httpclient: stop_and_destroy() ask the applet to autokill
httpclient_stop_and_destroy() tries to destroy the httpclient structure
if the client was stopped.

In the case the client wasn't stopped, it ask the client to stop itself
and to destroy the httpclient structure itself during the release of the
applet.
2021-10-06 15:15:03 +02:00
William Lallemand
739f90a6ef MINOR: httpclient: set HTTPCLIENT_F_ENDED only in release
Only set the HTTPCLIENT_F_ENDED flag in httpclient_applet_release()
function so we are sure that the appctx is not used anymore once the
flag is set.
2021-10-06 15:15:03 +02:00
William Lallemand
03f5a1c77d MINOR: httpclient: destroy() must free the headers and the ists
httpclient_destroy() must free all the ist in the httpclient structure,
the URL in the request, the vsn and reason in the response.

It also must free the list of headers of the response.
2021-10-06 15:15:03 +02:00
William Lallemand
614e68337d BUG/MEDIUM: httpclient: replace ist0 by istptr
ASAN reported a buffer overflow in the httpclient. This overflow is the
consequence of ist0() which is incorrect here.

Replace all occurences of ist0() by istptr() which is more appropried
here since all ist in the httpclient were created from strings.
2021-09-26 18:19:55 +02:00
William Lallemand
1123dde6dd MINOR: httpclient: httpclient_ended() returns 1 if the client ended
httpclient_ended() returns 1 if there is no more data to collect,
because the client received everything or the connection ended.
2021-09-24 14:21:26 +02:00
William Lallemand
518878e007 MINOR: httpclient: httpclient_data() returns the available data
httpclient_data() returns the available data in the httpclient.
2021-09-24 14:21:26 +02:00
William Lallemand
79a3478c24 MINOR: httpclient: add the EOH when no headers where provided
httpclient_req_gen() now adds the end of headers block when no header
was provided, which avoid adding it manually.
2021-09-20 16:24:54 +02:00
Willy Tarreau
e61244631a MINOR: applet: remove the thread mask from appctx_new()
appctx_new() is exclusively called with tid_bit and it only uses the
mask to pass it to the accompanying task. There is no point requiring
the caller to know about a mask there, nor is there any point in
creating an applet outside of the context of its own thread anyway.
Let's drop this and pass tid_bit to task_new() directly.
2021-09-17 16:08:34 +02:00
Amaury Denoyelle
bc2ebfa5a4 MEDIUM: server: extend refcount for all servers
In a future patch, it will be possible to remove at runtime every
servers, both static and dynamic. This requires to extend the server
refcount for all instances.

First, refcount manipulation functions have been renamed to better
express the API usage.

* srv_refcount_use -> srv_take
The refcount is always initialize to 1 on the server creation in
new_server. It's also incremented for each check/agent configured on a
server instance.

* free_server -> srv_drop
This decrements the refcount and if null, the server is freed, so code
calling it must not use the server reference after it. As a bonus, this
function now returns the next server instance. This is useful when
calling on the server loop without having to save the next pointer
before each invocation.

In these functions, remove the checks that prevent refcount on
non-dynamic servers. Each reference to "dynamic" in variable/function
naming have been eliminated as well.
2021-08-25 15:53:54 +02:00
William Lallemand
957ab13d7b BUILD: httpclient: fix build without OpenSSL
Add some defines around the ssl server so we can build without OpenSSL.
2021-08-24 18:33:28 +02:00
William Lallemand
4463b17fe3 BUG/MINOR: httpclient: fix Host header
THe http_update_update_host function takes an URL and extract the domain
to use as a host header. However it only update an existing host header
and does not create one.

This patch add an empty host header so the function can update it.
2021-08-24 17:53:03 +02:00
William Lallemand
211c9679c8 MINOR: httpclient: add the server to the proxy
Add the raw and ssl server to the proxy list so they can be freed during
the deinit() of HAProxy. As a side effect the 2 servers need to have a
different ID so the SSL one was renamed "<HTTPSCLIENT>".
2021-08-24 17:18:13 +02:00
William Lallemand
cfcbe9ebd9 MINOR: httpclient: set verify none on the https server
There is currently no way to specify the CA to verify from the
httpclient API. Sets the verify to none so we can still do https
request.
2021-08-24 17:15:58 +02:00