[MINOR] add options dontlog-normal and log-separate-errors

Some big traffic sites have trouble dealing with logs and tend to
disable them. Here are two new options to help cope with massive
logs.

  - dontlog-normal only disables logging for 100% successful
    connections, other ones will still be logged

  - log-separate-errors will cause non-100% successful connections
    to be logged at level "err" instead of level "info" so that a
    properly configured syslog daemon can send them to a different
    file for longer conservation.

doc/configuration.txt

@@ -600,6 +600,7 @@ monitor-uri                 X          X         X         -
 [no] option checkcache      X          -         X         X
 [no] option clitcpka        X          X         X         -
 [no] option contstats       X          X         X         -
+[no] option dontlog-normal  X          X         X         -
 [no] option dontlognull     X          X         X         -
 [no] option forceclose      X          -         X         X
 option forwardfor           X          X         X         X
@@ -608,6 +609,8 @@ option originalto           X          X         X         X
 option httpchk              X          -         X         X
 [no] option httpclose       X          X         X         X
 option httplog              X          X         X         X
+[no] option log-separate-
+            errors          X          X         X         -
 [no] option logasap         X          X         X         -
 [no] option nolinger        X          X         X         X
 [no] option persist         X          -         X         X
@@ -1999,6 +2002,29 @@ option contstats
   it is not enabled by default, as it has small performance impact (~0.5%).
 
 
+option dontlog-normal
+no option dontlog-normal
+  Enable or disable logging of normal, successful connections
+  May be used in sections :   defaults | frontend | listen | backend
+                                 yes   |    yes   |   yes  |   no
+  Arguments : none
+
+  There are large sites dealing with several thousand connections per second
+  and for which logging is a major pain. Some of them are even forced to turn
+  logs off and cannot debug production issues. Setting this option ensures that
+  normal connections, those which experience no error, no timeout, no retry nor
+  redispatch, will not be logged. This leaves disk space for anomalies. In HTTP
+  mode, the response status code is checked and return codes 5xx will still be
+  logged.
+
+  It is strongly discouraged to use this option as most of the time, the key to
+  complex issues is in the normal logs which will not be logged here. If you
+  need to separate logs, see the "log-separate-errors" option instead.
+
+  See also : "log", "dontlognull", "log-separate-errors" and section 2.6 about
+             logging.
+
+
 option dontlognull
 no option dontlognull
   Enable or disable logging of null connections
@@ -2300,6 +2326,29 @@ option httplog
   See also :  section 2.6 about logging.
 
 
+option log-separate-errors
+no option log-separate-errors
+  Change log level for non-completely successful connections
+  May be used in sections :   defaults | frontend | listen | backend
+                                 yes   |    yes   |   yes  |   no
+  Arguments : none
+
+  Sometimes looking for errors in logs is not easy. This option makes haproxy
+  raise the level of logs containing potentially interesting information such
+  as errors, timeouts, retries, redispatches, or HTTP status codes 5xx. The
+  level changes from "info" to "err". This makes it possible to log them
+  separately to a different file with most syslog daemons. Be careful not to
+  remove them from the original file, otherwise you would lose ordering which
+  provides very important information.
+
+  Using this option, large sites dealing with several thousand connections per
+  second may log normal traffic to a rotating buffer and only archive smaller
+  error logs.
+
+  See also : "log", "dontlognull", "dontlog-normal" and section 2.6 about
+             logging.
+
+
 option logasap
 no option logasap
   Enable or disable early logging of HTTP requests
@@ -4979,9 +5028,10 @@ emitted earlier if "option logasap" is specified, which makes sense in most
 environments with long sessions such as remote terminals. Sessions which match
 the "monitor" rules are never logged. It is also possible not to emit logs for
 sessions for which no data were exchanged between the client and the server, by
-specifying "option dontlognull" in the frontend. A few fields may slightly vary
+specifying "option dontlognull" in the frontend. Successful connections will
-depending on some configuration options, those are marked with a star ('*')
+not be logged if "option dontlog-normal" is specified in the frontend. A few
-after the field name below.
+fields may slightly vary depending on some configuration options, those are
+marked with a star ('*') after the field name below.
 
   Example :
         frontend fnt
@@ -5141,7 +5191,8 @@ emitted at the end of the session, unless "option logasap" is specified, which
 generally only makes sense for download sites. A session which matches the
 "monitor" rules will never logged. It is also possible not to log sessions for
 which no data were sent by the client by specifying "option dontlognull" in the
-frontend.
+frontend. Successful connections will not be logged if "option dontlog-normal"
+is specified in the frontend.
 
 Most fields are shared with the TCP log, some being different. A few fields may
 slightly vary depending on some configuration options. Those ones are marked
@@ -5419,6 +5470,35 @@ with normal logs, the total time field and the number of bytes are prefixed
 with a '+' sign which means that real numbers are certainly larger.
 
 
+2.6.3.3) Raising log level upon errors
+--------------------------------------
+
+Sometimes it is more convenient to separate normal traffic from errors logs,
+for instance in order to ease error monitoring from log files. When the option
+"log-separate-errors" is used, connections which experience errors, timeouts,
+retries, redispatches or HTTP status codes 5xx will see their syslog level
+raised from "info" to "err". This will help a syslog daemon store the log in
+a separate file. It is very important to keep the errors in the normal traffic
+file too, so that log ordering is not altered. You should also be careful if
+you already have configured your syslog daemon to store all logs higher than
+"notice" in an "admin" file, because the "err" level is higher than "notice".
+
+
+2.6.3.4) Disabling logging of successful connections
+----------------------------------------------------
+
+Although this may sound strange at first, some large sites have to deal with
+multiple thousands of logs per second and are experiencing difficulties keeping
+them intact for a long time or detecting errors within them. If the option
+"dontlog-normal" is set on the frontend, all normal connections will not be
+logged. In this regard, a normal connection is defined as one without any
+error, timeout, retry nor redispatch. In HTTP, the status code is checked too,
+and a response with a status 5xx is not considered normal and will be logged
+too. Of course, doing is is really discouraged as it will remove most of the
+useful information from the logs. Do this only if you have no other
+alternative.
+
+
 2.6.4) Timing events
 --------------------
 

include/types/proxy.h

@@ -115,6 +115,8 @@
 #define PR_O2_SPLIC_ANY	(PR_O2_SPLIC_REQ|PR_O2_SPLIC_RTR|PR_O2_SPLIC_AUT)
 #define PR_O2_REQBUG_OK	0x00000008      /* let buggy requests pass through */
 #define PR_O2_RSPBUG_OK	0x00000010      /* let buggy responses pass through */
+#define PR_O2_NOLOGNORM	0x00000020      /* don't log normal traffic, only errors and retries */
+#define PR_O2_LOGERRORS	0x00000040      /* log errors and retries at level LOG_ERR */
 
 /* This structure is used to apply fast weighted round robin on a server group */
 struct fwrr_group {

src/cfgparse.c

@@ -125,6 +125,8 @@ static const struct cfg_opt cfg_opts2[] =
 #endif
 	{ "accept-invalid-http-request",  PR_O2_REQBUG_OK, PR_CAP_FE, 0 },
 	{ "accept-invalid-http-response", PR_O2_RSPBUG_OK, PR_CAP_BE, 0 },
+	{ "dontlog-normal",               PR_O2_NOLOGNORM, PR_CAP_FE, 0 },
+	{ "log-separate-errors",          PR_O2_LOGERRORS, PR_CAP_FE, 0 },
 	{ NULL, 0, 0, 0 }
 };
 

src/log.c

@@ -321,10 +321,15 @@ void tcp_sess_log(struct session *s)
 	struct proxy *fe = s->fe;
 	struct proxy *be = s->be;
 	struct proxy *prx_log;
-	int tolog;
+	int tolog, level, err;
 	char *svid;
 	struct tm tm;
 
+	/* if we don't want to log normal traffic, return now */
+	err = (s->flags & (SN_ERR_MASK | SN_REDISP)) || (s->conn_retries != be->conn_retries);
+	if (!err && (fe->options2 & PR_O2_NOLOGNORM))
+		return;
+
 	if (s->cli_addr.ss_family == AF_INET)
 		inet_ntop(AF_INET,
 			  (const void *)&((struct sockaddr_in *)&s->cli_addr)->sin_addr,
@@ -353,7 +358,11 @@ void tcp_sess_log(struct session *s)
 	tolog = fe->to_log;
 	svid = (tolog & LW_SVID) ? (s->srv != NULL) ? s->srv->id : "<NOSRV>" : "-";
 
-	send_log(prx_log, LOG_INFO, "%s:%d [%02d/%s/%04d:%02d:%02d:%02d.%03d]"
+	level = LOG_INFO;
+	if (err && (fe->options2 & PR_O2_LOGERRORS))
+		level = LOG_ERR;
+
+	send_log(prx_log, level, "%s:%d [%02d/%s/%04d:%02d:%02d:%02d.%03d]"
 		 " %s %s/%s %ld/%ld/%s%ld %s%lld"
 		 " %c%c %d/%d/%d/%d/%s%u %ld/%ld\n",
 		 pn,

src/proto_http.c

@@ -755,7 +755,7 @@ void http_sess_log(struct session *s)
 	struct proxy *be = s->be;
 	struct proxy *prx_log;
 	struct http_txn *txn = &s->txn;
-	int tolog;
+	int tolog, level, err;
 	char *uri, *h;
 	char *svid;
 	struct tm tm;
@@ -763,6 +763,13 @@ void http_sess_log(struct session *s)
 	int t_request;
 	int hdr;
 
+	/* if we don't want to log normal traffic, return now */
+	err = (s->flags & (SN_ERR_MASK | SN_REDISP)) ||
+		(s->conn_retries != be->conn_retries) ||
+		txn->status >= 500;
+	if (!err && (fe->options2 & PR_O2_NOLOGNORM))
+		return;
+
 	if (fe->logfac1 < 0 && fe->logfac2 < 0)
 		return;
 	prx_log = fe;
@@ -830,7 +837,11 @@ void http_sess_log(struct session *s)
 	if (tv_isge(&s->logs.tv_request, &s->logs.tv_accept))
 		t_request = tv_ms_elapsed(&s->logs.tv_accept, &s->logs.tv_request);
 
-	send_log(prx_log, LOG_INFO,
+	level = LOG_INFO;
+	if (err && (fe->options2 & PR_O2_LOGERRORS))
+		level = LOG_ERR;
+
+	send_log(prx_log, level,
 		 "%s:%d [%02d/%s/%04d:%02d:%02d:%02d.%03d]"
 		 " %s %s/%s %d/%ld/%ld/%ld/%s%ld %d %s%lld"
 		 " %s %s %c%c%c%c %d/%d/%d/%d/%s%u %ld/%ld%s\n",