mirrors/haproxy
1
0
Fork 0
mirror of https://git.haproxy.org/git/haproxy.git/ synced 2025-09-29 09:41:21 +02:00
Code Issues Projects Releases Wiki Activity

MEDIUM: http: add option-ignore-probes to get rid of the floods of 408

Browse Source 
Recently some browsers started to implement a "pre-connect" feature
consisting in speculatively connecting to some recently visited web sites
just in case the user would like to visit them. This results in many
connections being established to web sites, which end up in 408 Request
Timeout if the timeout strikes first, or 400 Bad Request when the browser
decides to close them first. These ones pollute the log and feed the error
counters. There was already "option dontlognull" but it's insufficient in
this case. Instead, this option does the following things :
   - prevent any 400/408 message from being sent to the client if nothing
     was received over a connection before it was closed ;
   - prevent any log from being emitted in this situation ;
   - prevent any error counter from being incremented

That way the empty connection is silently ignored. Note that it is better
not to use this unless it is clear that it is needed, because it will hide
real problems. The most common reason for not receiving a request and seeing
a 408 is due to an MTU inconsistency between the client and an intermediary
element such as a VPN, which blocks too large packets. These issues are
generally seen with POST requests as well as GET with large cookies. The logs
are often the only way to detect them.

This patch should be backported to 1.5 since it avoids false alerts and
makes it easier to monitor haproxy's status.
This commit is contained in:
Willy Tarreau 2015-05-01 15:37:53 +02:00
parent 13317669d5
commit 0f228a037a
4 changed files with 68 additions and 20 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

76
doc/configuration.txt
View File

@ -4336,7 +4336,9 @@ no option dontlognull
simple port probe or scan will produce a log. If those connections pollute simple port probe or scan will produce a log. If those connections pollute
the logs too much, it is possible to enable option "dontlognull" to indicate the logs too much, it is possible to enable option "dontlognull" to indicate
that a connection on which no data has been transferred will not be logged, that a connection on which no data has been transferred will not be logged,
which typically corresponds to those probes. which typically corresponds to those probes. Note that errors will still be
returned to the client and accounted for in the stats. If this is not what is
desired, option http-ignore-probes can be used instead.
It is generally recommended not to use this option in uncontrolled It is generally recommended not to use this option in uncontrolled
environments (eg: internet), otherwise scans and other malicious activities environments (eg: internet), otherwise scans and other malicious activities
@ -4345,7 +4347,8 @@ no option dontlognull
If this option has been enabled in a "defaults" section, it can be disabled If this option has been enabled in a "defaults" section, it can be disabled
in a specific instance by prepending the "no" keyword before it. in a specific instance by prepending the "no" keyword before it.
See also : "log", "monitor-net", "monitor-uri" and section 8 about logging. See also : "log", "http-ignore-probes", "monitor-net", "monitor-uri", and
section 8 about logging.
option forceclose option forceclose
@ -4441,6 +4444,40 @@ option forwardfor [ except <network> ] [ header <name> ] [ if-none ]
"option forceclose", "option http-keep-alive" "option forceclose", "option http-keep-alive"
option http-ignore-probes
no option http-ignore-probes
Enable or disable logging of null connections and request timeouts
May be used in sections : defaults | frontend | listen | backend
yes | yes | yes | no
Arguments : none
Recently some browsers started to implement a "pre-connect" feature
consisting in speculatively connecting to some recently visited web sites
just in case the user would like to visit them. This results in many
connections being established to web sites, which end up in 408 Request
Timeout if the timeout strikes first, or 400 Bad Request when the browser
decides to close them first. These ones pollute the log and feed the error
counters. There was already "option dontlognull" but it's insufficient in
this case. Instead, this option does the following things :
- prevent any 400/408 message from being sent to the client if nothing
was received over a connection before it was closed ;
- prevent any log from being emitted in this situation ;
- prevent any error counter from being incremented
That way the empty connection is silently ignored. Note that it is better
not to use this unless it is clear that it is needed, because it will hide
real problems. The most common reason for not receiving a request and seeing
a 408 is due to an MTU inconsistency between the client and an intermediary
element such as a VPN, which blocks too large packets. These issues are
generally seen with POST requests as well as GET with large cookies. The logs
are often the only way to detect them.
If this option has been enabled in a "defaults" section, it can be disabled
in a specific instance by prepending the "no" keyword before it.
See also : "log", "dontlognull", "errorfile", and section 8 about logging.
option http-keep-alive option http-keep-alive
no option http-keep-alive no option http-keep-alive
Enable or disable HTTP keep-alive from client to server Enable or disable HTTP keep-alive from client to server
@ -8334,8 +8371,8 @@ timeout http-request <timeout>
about the problem, and the connection is closed. The logs will report about the problem, and the connection is closed. The logs will report
termination codes "cR". Some recent browsers are having problems with this termination codes "cR". Some recent browsers are having problems with this
standard, well-documented behaviour, so it might be needed to hide the 408 standard, well-documented behaviour, so it might be needed to hide the 408
code using "errorfile 408 /dev/null". See more details in the explanations of code using "option http-ignore-probes" or "errorfile 408 /dev/null". See
the "cR" termination code in section 8.5. more details in the explanations of the "cR" termination code in section 8.5.
Note that this timeout only applies to the header part of the request, and Note that this timeout only applies to the header part of the request, and
not to any data. As soon as the empty line is received, this timeout is not not to any data. As soon as the empty line is received, this timeout is not
@ -8353,7 +8390,8 @@ timeout http-request <timeout>
effect, unless the frontend is in TCP mode, in which case the HTTP backend's effect, unless the frontend is in TCP mode, in which case the HTTP backend's
timeout will be used. timeout will be used.
See also : "errorfile", "timeout http-keep-alive", "timeout client". See also : "errorfile", "http-ignore-probes", "timeout http-keep-alive", and
"timeout client".
timeout queue <timeout> timeout queue <timeout>
@ -13664,7 +13702,8 @@ easier finding and understanding.
the request was typed by hand using a telnet client, and aborted the request was typed by hand using a telnet client, and aborted
too early. The HTTP status code is likely a 400 here. Sometimes this too early. The HTTP status code is likely a 400 here. Sometimes this
might also be caused by an IDS killing the connection between haproxy might also be caused by an IDS killing the connection between haproxy
and the client. and the client. "option http-ignore-probes" can be used to ignore
connections without any data transfer.
cR The "timeout http-request" stroke before the client sent a full HTTP cR The "timeout http-request" stroke before the client sent a full HTTP
request. This is sometimes caused by too large TCP MSS values on the request. This is sometimes caused by too large TCP MSS values on the
@ -13672,19 +13711,18 @@ easier finding and understanding.
packets, or by clients sending requests by hand and not typing fast packets, or by clients sending requests by hand and not typing fast
enough, or forgetting to enter the empty line at the end of the enough, or forgetting to enter the empty line at the end of the
request. The HTTP status code is likely a 408 here. Note: recently, request. The HTTP status code is likely a 408 here. Note: recently,
some browsers such as Google Chrome started to break the deployed Web some browsers started to implement a "pre-connect" feature consisting
infrastructure by aggressively implementing a new "pre-connect" in speculatively connecting to some recently visited web sites just
feature, consisting in sending connections to sites recently visited in case the user would like to visit them. This results in many
without sending any request on them until the user starts to browse connections being established to web sites, which end up in 408
the site. This mechanism causes massive disruption among resource- Request Timeout if the timeout strikes first, or 400 Bad Request when
limited servers, and causes a lot of 408 errors in HAProxy logs. the browser decides to close them first. These ones pollute the log
Worse, some people report that sometimes the browser displays the 408 and feed the error counters. Some versions of some browsers have even
error when the user expects to see the actual content (Mozilla fixed been reported to display the error code. It is possible to work
this bug in 2004, while Chrome users continue to report it in 2014), around the undesirable effects of this behaviour by adding "option
so in this case, using "errorfile 408 /dev/null" can be used as a http-ignore-probes" in the frontend, resulting in connections with
workaround. More information on the subject is available here : zero data transfer to be totally ignored. This will definitely hide
https://bugzilla.mozilla.org/show_bug.cgi?id=248827 the errors of people experiencing connectivity issues though.
https://code.google.com/p/chromium/issues/detail?id=85229
CT The client aborted while its session was tarpitted. It is important to CT The client aborted while its session was tarpitted. It is important to
check if this happens on valid requests, in order to be sure that no check if this happens on valid requests, in order to be sure that no

2
include/types/proxy.h
View File

@ -81,7 +81,7 @@ enum pr_mode {
#define PR_O_DISPATCH 0x00000040 /* use dispatch mode */ #define PR_O_DISPATCH 0x00000040 /* use dispatch mode */
/* unused: 0x00000080 */ /* unused: 0x00000080 */
#define PR_O_FWDFOR 0x00000100 /* conditionally insert x-forwarded-for with client address */ #define PR_O_FWDFOR 0x00000100 /* conditionally insert x-forwarded-for with client address */
/* unused: 0x00000200 */ #define PR_O_IGNORE_PRB 0x00000200 /* ignore empty requests (aborts and timeouts) */
#define PR_O_NULLNOLOG 0x00000400 /* a connect without request will not be logged */ #define PR_O_NULLNOLOG 0x00000400 /* a connect without request will not be logged */
/* unused: 0x0800, 0x1000 */ /* unused: 0x0800, 0x1000 */
#define PR_O_FF_ALWAYS 0x00002000 /* always set x-forwarded-for */ #define PR_O_FF_ALWAYS 0x00002000 /* always set x-forwarded-for */

1
src/cfgparse.c
View File

@ -154,6 +154,7 @@ static const struct cfg_opt cfg_opts[] =
{ "contstats", PR_O_CONTSTATS, PR_CAP_FE, 0, 0 }, { "contstats", PR_O_CONTSTATS, PR_CAP_FE, 0, 0 },
{ "dontlognull", PR_O_NULLNOLOG, PR_CAP_FE, 0, 0 }, { "dontlognull", PR_O_NULLNOLOG, PR_CAP_FE, 0, 0 },
{ "http_proxy", PR_O_HTTP_PROXY, PR_CAP_FE | PR_CAP_BE, 0, PR_MODE_HTTP }, { "http_proxy", PR_O_HTTP_PROXY, PR_CAP_FE | PR_CAP_BE, 0, PR_MODE_HTTP },
{ "http-ignore-probes", PR_O_IGNORE_PRB, PR_CAP_FE, 0, PR_MODE_HTTP },
{ "prefer-last-server", PR_O_PREF_LAST, PR_CAP_BE, 0, PR_MODE_HTTP }, { "prefer-last-server", PR_O_PREF_LAST, PR_CAP_BE, 0, PR_MODE_HTTP },
{ "logasap", PR_O_LOGASAP, PR_CAP_FE, 0, 0 }, { "logasap", PR_O_LOGASAP, PR_CAP_FE, 0, 0 },
{ "nolinger", PR_O_TCP_NOLING, PR_CAP_FE | PR_CAP_BE, 0, 0 }, { "nolinger", PR_O_TCP_NOLING, PR_CAP_FE | PR_CAP_BE, 0, 0 },

9
src/proto_http.c
View File

@ -2709,6 +2709,9 @@ int http_wait_for_request(struct stream *s, struct channel *req, int an_bit)
if (txn->flags & TX_WAIT_NEXT_RQ) if (txn->flags & TX_WAIT_NEXT_RQ)
goto failed_keep_alive; goto failed_keep_alive;
if (sess->fe->options & PR_O_IGNORE_PRB)
goto failed_keep_alive;
/* we cannot return any message on error */ /* we cannot return any message on error */
if (msg->err_pos >= 0) { if (msg->err_pos >= 0) {
http_capture_bad_message(&sess->fe->invalid_req, s, msg, msg->msg_state, sess->fe); http_capture_bad_message(&sess->fe->invalid_req, s, msg, msg->msg_state, sess->fe);
@ -2739,6 +2742,9 @@ int http_wait_for_request(struct stream *s, struct channel *req, int an_bit)
if (txn->flags & TX_WAIT_NEXT_RQ) if (txn->flags & TX_WAIT_NEXT_RQ)
goto failed_keep_alive; goto failed_keep_alive;
if (sess->fe->options & PR_O_IGNORE_PRB)
goto failed_keep_alive;
/* read timeout : give up with an error message. */ /* read timeout : give up with an error message. */
if (msg->err_pos >= 0) { if (msg->err_pos >= 0) {
http_capture_bad_message(&sess->fe->invalid_req, s, msg, msg->msg_state, sess->fe); http_capture_bad_message(&sess->fe->invalid_req, s, msg, msg->msg_state, sess->fe);
@ -2768,6 +2774,9 @@ int http_wait_for_request(struct stream *s, struct channel *req, int an_bit)
if (txn->flags & TX_WAIT_NEXT_RQ) if (txn->flags & TX_WAIT_NEXT_RQ)
goto failed_keep_alive; goto failed_keep_alive;
if (sess->fe->options & PR_O_IGNORE_PRB)
goto failed_keep_alive;
if (msg->err_pos >= 0) if (msg->err_pos >= 0)
http_capture_bad_message(&sess->fe->invalid_req, s, msg, msg->msg_state, sess->fe); http_capture_bad_message(&sess->fe->invalid_req, s, msg, msg->msg_state, sess->fe);
txn->status = 400; txn->status = 400;