In certain circumstances (eg: Lua HTTP applet called from a
TCP ruleset before http_process_request()), the HTTP TXN is not
yet fully initialized so some information it contains cannot be
relied on. Such information include the HTTP version, the state
of the expect: 100-continue header, the connection header and
the transfer-encoding header.
Here the bug only turns something which already doesn't work
into something wrong, but better avoid any references to the
http_txn from the Lua code to avoid future mistakes.
This patch should be backported into 1.6 for code consistency.
If a sample fetch needing http_txn is called from an HTTP Lua applet,
the result will be invalid and may even cause a crash because some HTTP
data can be forwarded and the HTTP txn is no longer valid.
Here the solution is to ensure that a fetch called from Lua never
needs http_txn. This is done thanks to a new flag HLUA_F_MAY_USE_HTTP
which indicates whether or not it is safe to call a fetch which needs
HTTP.
This fix needs to be backported to 1.6.
This patch converts a boolean "int" to a bitfiled. The main
reason is to save space in the struct if another flag may will
be require.
Note that this patch is required for next fix and will need to be
backported to 1.6.
When a POST is processed by a Lua service, the HTTP header are
potentially gone. So, we cannot retrieve their content using
the standard "hdr" sample fetchs (which will soon become invalid
anyway) from an applet.
This patch add an entry "headers" to the object applet_http. This
entry is an array containing all the headers. It permits to use the
HTTP headers during the processing of the service.
Many thanks to Jan Bruder for reporting this issue with enough
details to reproduce it.
This patch will have to be backported to 1.6 since it will be the
only way to access headers from Lua applets.
Sander Klein reported an error messages about SSLv3 not
being supported on Debian 8, although he didn't force-sslv3.
Vincent Bernat tracked this down to the LUA initialization, which
actually does force-sslv3.
This patch removes force-sslv3 from the LUA initialization, so
the LUA SSL socket can actually use TLS and doesn't trigger
warnings when SSLv3 is not supported by libssl (such as in
Debian 8).
This should be backported to 1.6.
When the txn.done() fiunction is called, the ouput buffer is cleaned,
but the associated relative pointer on the HTTP requests elements
is not reseted.
This patch remove this cleanup, because the output buffer may contain
data to forward.
The direction (request or response) is not propagated in the
sample fecthes called throught Lua. This patch adds the direction
status in some structs (hlua_txn and hlua_smp) to make sure that
the sample fetches will be called with all the information.
The converters can not access to a TXN object, so there are not
impacted the direction. However, the samples used as input of the
Lua converter wrapper are initiliazed with the direction. Thereby,
the struct smp stay consistent.
[wt: needs to be backported to 1.6]
This patch cleanups the direction names. It replaces numeric values,
by the associated defines. It ensure the compliance with values found
somwhere else in HAProxy.
It is required by the bugfix patch which is following.
[wt: needs to be backported to 1.6]
Lua needs to known the direction of the http data processed (request or
response). It checks the flag SMP_OPT_DIR_REQ, buf this flag is 0. This patch
correctly checks the flags after applying the SMP_OPT_DIR mask.
Thierry reported that keep-alive still didn't cope well with Lua
services. The reason is that for now applets have to be closed at
the end of a transaction so we want to work in server-close mode,
which isn't noticeable by the client since it still sees keep-alive.
Additionally we want to enable the request body transfer analyser
which will be needed to synchronize with the response analyser to
indicate the end of the transfer.
Only the main execution function can set the run flag, because it is
the last function before the execution time.
This patch removes the flag set by another function. It will be used
by the new lua timeout counter.
The garbage collector is a little bit heavy to run, and it was added
only for cosockets. This patch prevent useless executions when no
cosockets are used.
When the channel is down, the applet is waked up. Before this patch,
the applet closes the stream and the unread data are discarded.
After this patch, the stream is not closed except if the buffers are
empty.
It will be closed later by the close function, or by the garbage collector.
The HAProxy Lua socket respects the Lua Socket tcp specs. these specs
are a little bit limited, it not permits to connect to unix socket.
this patch extends a little it the specs. It permit to accept a
syntax that begin with "unix@", "ipv4@", "ipv6@", "fd@" or "abns@".
Actions may yield but must not do it during the final call from a ruleset
because it indicates there will be no more opportunity to complete or
clean up. This is indicated by ACT_FLAG_FINAL in the action's flags,
which must be passed to hlua_resume().
Thanks to this, an action called from a TCP ruleset is properly woken
up and possibly finished when the client disconnects.
Since commit bc4c1ac ("MEDIUM: http/tcp: permit to resume http and tcp
custom actions"), some actions may yield and be called back when new
information are available. Unfortunately some of them may continue to
yield because they simply don't know that it's the last call from the
rule set. For this reason we'll need to pass a flag to the custom
action to pass such information and possibly other at the same time.
When a thread stops this patch forces a garbage collection which
cleanup and free the memory.
The HAProxy Lua Socket class contains an HAProxy session, so it
uses a big amount of memory, in other this Socket class uses a
filedescriptor and maintain a connection open.
If the class Socket is stored in a global variable, the Socket stay
alive along of the life of the process (except if it is closed by the
other size, or if a timeout is reached). If the class Socket is stored
in a local variable, it must die with this variable.
The socket is closed by a call from the garbage collector. And the
portability and use of a variable is known by the same garbage collector.
so, running the GC just after the end of all Lua code ensure that the
heavy resources like the socket class are freed quickly.
Not having the target set on the connection causes it to be released
at the last moment, and the destination address to randomly be valid
depending on the data found in the memory at this moment. In practice
it works as long as memory poisonning is disabled. The deep reason is
that connect_server() doesn't expect to be called with SF_ADDR_SET and
an existing connection with !reuse. This causes the release of the
connection, its reallocation (!reuse), and taking the address from the
newly allocated connection. This should certainly be improved.
Commit d75cb0f ("BUG/MAJOR: lua: segfault after the channel data is
modified by some Lua action.") introduced a regression causing an
action run from a TCP rule in an HTTP proxy to end in HTTP error
if it terminated cleanly, because it didn't parse the HTTP request!
Relax the test so that it takes into account the opportunity for the
analysers to parse the message.
The lua code must the the appropriate wakeup mechanism for cosockets.
It wakes them up from outside the stream and outside the applet, so it
shouldn't use the applet's wakeup callback which is designed only for
use from within the applet itself. For now it didn't cause any trouble
(yet).
When an action or a fetch modify the channel data, the http request parser
pointer become inconsistent. This patch detects the modification and call
again the parser.
The function lua_settable uses the metatable for acessing data,
so in the C part, we want to index value in the real table and
not throught the meta-methods. It's much faster this way.
This function is a callback made only for calls from the applet handler.
Rename it to remove confusion. It's currently called from the Lua code
but that's not correct, we should call the notify and update functions
instead otherwise it will not enable the applet again.
The current Lua action are not registered. The executed function is
selected according with a function name writed in the HAProxy configuration.
This patch add an action registration function. The configuration mode
described above disappear.
This change make some incompatibilities with existing configuration files for
HAProxy 1.6-dev.
This couple of function executes securely some Lua calls outside of
the lua runtime environment. Each Lua call can return a longjmp
if it encounter a memory error.
Lua documentation extract:
If an error happens outside any protected environment, Lua calls
a panic function (see lua_atpanic) and then calls abort, thus
exiting the host application. Your panic function can avoid this
exit by never returning (e.g., doing a long jump to your own
recovery point outside Lua).
The panic function runs as if it were a message handler (see
2.3); in particular, the error message is at the top of the
stack. However, there is no guarantee about stack space. To push
anything on the stack, the panic function must first check the
available space (see 4.2).
We must check all the Lua entry point. This includes:
- The include/proto/hlua.h exported functions
- the task wrapper function
- The action wrapper function
- The converters wrapper function
- The sample-fetch wrapper functions
It is tolerated that the initilisation function returns an abort.
Before each Lua abort, an error message is writed on stderr.
The macro SET_SAFE_LJMP initialise the longjmp. The Macro
RESET_SAFE_LJMP reset the longjmp. These function must be macro
because they must be exists in the program stack when the longjmp
is called
All the code which emits error log have the same pattern. Its:
Send log with syslog system, and if it is allowed, display error
log on screen.
This patch replace this pattern by a macro. This reduces the number
of lines.
The send_log function needs a final \n.
This bug is repported by Michael Ezzell.
Minor bug: when writing to syslog from Lua scripts, the last character from
each log entry is truncated.
core.Alert("this is truncated");
Sep 7 15:07:56 localhost haproxy[7055]: this is truncate
This issue appears to be related to the fact that send_log() (in src/log.c)
is expecting a newline at the end of the message's format string:
/*
* This function adds a header to the message and sends the syslog message
* using a printf format string. It expects an LF-terminated message.
*/
void send_log(struct proxy *p, int level, const char *format, ...)
I believe the fix would be in in src/hlua.c at line 760
<http://git.haproxy.org/?p=haproxy.git;a=blob;f=src/hlua.c;h=1e4d47c31e66c16c837ff2aa5ef577f6cafdc7e7;hb=316e3196285b89a917c7d84794ced59a6a5b4eba#l760>,
where this...
send_log(px, level, "%s", trash.str);
...should be adding a newline into the format string to accommodate what
the code expects.
send_log(px, level, "%s\n", trash.str);
This change provides what seems to be the correct behavior:
Sep 7 15:08:30 localhost haproxy[7150]: this is truncated
All other uses of send_log() in hlua.c have a trailing dot "." in the
message that is masking the truncation issue because the output message
stops on a clean word boundary. I suspect these would also benefit from
"\n" appended to their format strings as well, since this appears to be the
pattern seen throughout the rest of the code base.
Reported-by: Michael Ezzell <michael@ezzell.net>
See commit id bdc97a8795
Michael Ezzell reported that the following Lua code fails in
dev4 when the TCP is not established immediately (due to a little
bit of latency):
function tricky_socket()
local sock = core.tcp();
sock:settimeout(3);
core.log(core.alert,"calling connect()\n");
local connected, con_err = sock:connect("x.x.x.x",80);
core.log(core.alert,"returned from connect()\n");
if con_err ~= nil then
core.log(core.alert,"connect() failed with error: '" .. con_err .. "'\n");
end
The problem is that the flags who want to wake up the applet are
resetted before each applet call, so the applet must set again the
flags if the connection is not established.
Before this patch, two type of custom actions exists: ACT_ACTION_CONT and
ACT_ACTION_STOP. ACT_ACTION_CONT is a non terminal action and ACT_ACTION_STOP is
a terminal action.
Note that ACT_ACTION_STOP is not used in HAProxy.
This patch remove this behavior. Only type type of custom action exists, and it
is called ACT_CUSTOM. Now, the custion action can return a code indicating the
required behavior. ACT_RET_CONT wants that HAProxy continue the current rule
list evaluation, and ACT_RET_STOP wants that HAPRoxy stops the the current rule
list evaluation.
When called from an http ruleset, txn:done() can still crash the process
because it closes the stream without consuming pending data resulting in
the transaction's buffer representation to differ from the real buffer.
This patch also adjusts the transaction's state to indicate that it's
closed to be consistent with what's already done in redirect rules.
When the Lua execution flow endswith the command done (core.done or txn.done())
an error is detourned, and the stack is no longer usable. This patch juste
reinitilize the stack if this case is detected.
The function txn:close() must be terminal because it demands the session
destruction. This patch renames this function to "done()" to be much
clearer about the fact that it is a final operation.
This patch is inspired by Bowen Ni's proposal and it is based on his first
implementation:
With Lua integration in HAProxy 1.6, one can change the request method,
path, uri, header, response header etc except response line.
I'd like to contribute the following methods to allow modification of the
response line.
[...]
There are two new keywords in 'http-response' that allows you to rewrite
them in the native HAProxy config. There are also two new APIs in Lua that
allows you to do the same rewriting in your Lua script.
Example:
Use it in HAProxy config:
*http-response set-code 404*
Or use it in Lua script:
*txn.http:res_set_reason("Redirect")*
I dont take the full patch because the manipulation of the "reason" is useless.
standard reason are associated with each returned code, and unknown code can
take generic reason.
So, this patch can set the status code, and the reason is automatically adapted.
The function dont remove remaineing analysers and dont update response
channel timeout.
The fix is a copy of the behavior of the functions http_apply_redirect_rule()
and stream_int_retnclose().
Tsvetan Tsvetanov reported that the following Lua code fails in
dev2 and dev3 :
function hello(txn)
local request_msg = txn.req:dup()
local tsm_sock = core.tcp()
tsm_sock:connect("127.0.0.1", 7777)
local res = tsm_sock:send(request_msg)
local response = tsm_sock:receive('*l')
txn.res:send(response)
txn:close()
end
Thierry diagnosed that it was caused by commit 563cc37 ("MAJOR: stream:
use a regular ->update for all stream interfaces"). It broke lua's
ability to establish outgoing connections.
The reason is that the applet used to be notified about established
connections just after the stream analyser loop, and that's not the
case anymore. In peers, this issue didn't happen because peers use
a handshake so after sending data, the response is received and wakes
the applet up again. Here we have to indicate that we want to send or
receive data, this will cause the notification to happen as soon as
the connection is ready. This is similar to pretending that we're
working on a full buffer after all. In theory subscribing for reads
is not needed, but it's added here for completeness.
Reported-By: Tsvetan Tsvetanov <cpi.cecko@gmail.com>
Now the prototype for each action from each section are the same, and
a discriminant for determining for each section we are called are added.
So, this patch removes the wrappers for the action functions called from
more than one section.
This patch removes 132 lines of useless code.
This patch normalize the return code of the configuration parsers. Before
these changes, the tcp action parser returned -1 if fail and 0 for the
succes. The http action returned 0 if fail and 1 if succes.
The normalisation does:
- ACT_RET_PRS_OK for succes
- ACT_RET_PRS_ERR for failure
This patch merges the conguration keyword struct. Each declared configuration
keyword struct are similar with the others. This patch simplify the code.
Action function can return 3 status:
- error if the action encounter fatal error (like out of memory)
- yield if the action must terminate his work later
- continue in other cases
For performances considerations, some actions are not processed by remote
function. They are directly processed by the function. Some of these actions
does the same things but for different processing part (request / response).
This patch give the same name for the same actions, and change the normalization
of the other actions names.
This patch is ONLY a rename, it doesn't modify the code.
This patch group the action name in one file. Some action are called
many times and need an action embedded in the action caller. The main
goal is to have only one header file grouping all definitions.
The (http|tcp)-(request|response) action rules use common
opaque type. For the HAProxy embbedded feature, types are know,
it better to add this types in the action union and use it.
This patch is the first of a serie which merge all the action structs. The
function "tcp-request content", "tcp-response-content", "http-request" and
"http-response" have the same values and the same process for some defined
actions, but the struct and the prototype of the declared function are
different.
This patch try to unify all of these entries.
The union name "data" is a little bit heavy while we read the source
code because we can read "data.data.sint". The rename from "data" to "u"
makes the read easiest like "data.u.sint".
This patch remove the struct information stored both in the struct
sample_data and in the striuct sample. Now, only thestruct sample_data
contains data, and the struct sample use the struct sample_data for storing
his own data.
test conf:
global
tune.lua.session-timeout 0
lua-load lol.lua
debug
maxconn 4096
listen test
bind 0.0.0.0:10010
mode tcp
tcp-request content lua act_test
balance roundrobin
server test 127.0.0.1:3304
lua test:
function act_test(txn)
while true do
core.Alert("TEST")
end
end
The function "act_test()" is not executed because a zero timeout is not
considered as TICK_ETERNITY, but is considered as 0.
This path fix this behavior. This is the same problem than the bugfix
685c014e99.
I've been trying out 1.6 dev3 with lua support, and trying to start
lua tasks seems to not be working.
Using this configuration
global
lua-load /lua/lol.lua
debug
maxconn 4096
backend shard_b
server db01 mysql_shard_b:3306
backend shard_a
server db01 mysql_shard_a:3306
listen mysql-cluster
bind 0.0.0.0:8001
mode tcp
balance roundrobin
use_backend shard_b
And this lua function
core.register_task(function()
while true do
core.Alert("LOLOLOLOLOL")
end
end)
I'd always get a timeout error starting the registered function.
The problem lies as far as I can tell in the fact that is possible for
now_ms to not change (is this maybe a problem on my config/system?)
until the expiration check happens, in the resume function that
actually kickstarts the lua task, making HAProxy think that expiration
time for the task is up, if I understand correctly tasks are meant to
never really timeout.
These ones are considered safe as they have already been reused.
They will be useful in "aggressive" and "always" http-reuse modes
in order to place the first request of a connection with the least
risk.
For now it's not populated but we have the list entry. It will carry
all idle connections that sessions don't want to share. They may be
used later to reclaim connections upon socket shortage for example.
Since we now always call this function with the reuse parameter cleared,
let's simplify the function's logic as it cannot return the existing
connection anymore. The savings on this inline function are appreciable
(240 bytes) :
$ size haproxy.old haproxy.new
text data bss dec hex filename
1020383 40816 36928 1098127 10c18f haproxy.old
1020143 40816 36928 1097887 10c09f haproxy.new
This patch removes the 32 bits unsigned integer and the 32 bit signed
integer. It replaces these types by a unique type 64 bit signed.
This makes easy the usage of integer and clarify signed and unsigned use.
With the previous version, signed and unsigned are used ones in place of
others, and sometimes the converter loose the sign. For example, divisions
are processed with "unsigned", if one entry is negative, the result is
wrong.
Note that the integer pattern matching and dotted version pattern matching
are already working with signed 64 bits integer values.
There is one user-visible change : the "uint()" and "sint()" sample fetch
functions which used to return a constant integer have been replaced with
a new more natural, unified "int()" function. These functions were only
introduced in the latest 1.6-dev2 so there's no impact on regular
deployments.
Dmitry Sivachenko reported the following build warning using Clang,
though it's harmless :
src/hlua.c:1911:13: warning: variable '_socket_info_expanded_form' is not needed
and will not be emitted [-Wunneeded-internal-declaration]
static char _socket_info_expanded_form[] = SOCKET_INFO_EXPANDED_FORM;
^
Indeed, the variable is not used except to compute a sizeof which is
taken from the string it is initialized from. It probably is a leftover
after various code refactorings. Let's get rid of it now since it's not
used anymore.
No backport is needed.
Actually, the registered lua actions with "tcp-request lua" and
"tcp-response lua" are final actions. This patch change the action
type type and permit to continue the evaluation of tcp-* processing
after the evaluation of the lua actions.
First, findproxy() was renamed proxy_find_by_name() so that its explicit
that a name is required for the lookup. Second, we give this function
the ability to search for tables if needed. Third we now provide inline
wrappers to pass the appropriate PR_CAP_* flags and to explicitly look
up a frontend, backend or table.
Options are relative to the sample. Each sample fetched is associated with
fetch options or fetch flags.
This patch adds the 'opt' vaue in the sample struct. This permits to reduce
the sample-fetch function prototype. In other way, the converters will have
more detail about the origin of the sample.
This patch removes the structs "session", "stream" and "proxy" from
the sample-fetches and converters function prototypes.
This permits to remove some weight in the prototype call.
Some sample analyzer (sample-fetch or converters) needs to known the proxy,
session and stream attached to the sampel. The sample-fetches and the converters
function pointers cannot be called without these 3 pointers filled.
This patch permits to reduce the sample-fetch and the converters called
prototypes, and provides a new mean to add information for this type of
functions.
It's much easier to centralize this call into the I/O handler than to
do it everywhere with the risk to miss it. Applets are not allowed to
unregister themselves anyway so their SI is still present and it is
possible to update all the context.
The applet I/O handlers now rely on si_applet_done() which itself decides
to wake up or sleep the appctx. Now it becomes critical that applte handlers
properly call this on every exit path so that the appctx is removed from the
active list after I/O have been handled. One such call was added to the Lua
socket handler. It used to work without it probably because the main task is
woken up by the parent task but now it's needed.
Now that applet's functions only take an appctx in argument, not a
stream interface. This slightly simplifies the code and will be needed
to take the appctx out of the stream interface.
We don't pass sess->origin anymore but the pointer to the previous step. Now
it should be much easier to chain elements together once applets are moved out
of streams. Indeed, the session is only used for configuration and not for the
dynamic chaining anymore.
This patch cretes a new Map class that permits to do some lookup in
HAProxy maps. This Map class is integration in the HAProxy update
system, so we can modify the map throught the socket.
the functions (req|res)_get_headers() return only the last entry
for each header with the same name. This patch fix this behavior.
Each header name contain an array of values.
