Commit 9f4f6b038 ("OPTIM: hpack-huff: reduce the cache footprint of the
huffman decoder") replaced the large tables with more space efficient
byte arrays, but one table, rht_bit15_11_11_4, has a 64 bytes hole in it
that wasn't materialized by filling it with zeroes to make the offsets
match, nor by adjusting the offset from the caller. This resulted in
some control chars not properly being decoded and being seen as byte 0,
and the associated messages to be rejected, as can be seen in issue #1971.
This commit fixes it by adjusting the offset used for the higher part of
the table so that we don't need to store 64 zeroes that will never be
accessed.
This needs to be backported to 2.7.
Thanks to Christopher for spotting the bug, and to Juanga Covas for
providing precious traces showing the problem.
Some tables are currently used to decode bit blocks and lengths. We do
see such lookups in perf top. We have 4 512-byte tables and one 64-byte
one. Looking closer, the second half of the table (length) has so few
variations that most of the time it will be computed in a single "if",
and never more than 3. This alone allows to cut the tables in half. In
addition, one table (bits 15-11) is only 32-element long, while another
one (bits 11-4) starts at 0x60, so we can merge the two as they do not
overlap, and further save size. We're now down to 4 256-entries tables.
This is visible in h3 and h2 where the max request rate is slightly higher
(e.g. +1.6% for h2). The huff_dec() function got slightly larger but the
overall code size shrunk:
$ nm --size haproxy-before | grep huff_dec
000000000000029e T huff_dec
$ nm --size haproxy-after | grep huff_dec
0000000000000345 T huff_dec
$ size haproxy-before haproxy-after
text data bss dec hex filename
7591126 569268 2761348 10921742 a6a70e haproxy-before
7591082 568180 2761348 10920610 a6a2a2 haproxy-after
As suggested in the comment, it's possible to read 32 bits at once in
big-endian order, and now we have the functions to do this, so let's
do it. This reduces the code on the fast path by 31 bytes on x86, and
more importantly performs single-operation 32-bit reads instead of
playing with shifts and additions.
As reported in issue #1635, there's a subtle sign change when shifting
a uint8_t value to the left because integer promotion first turns any
smaller type to signed int *even if it was unsigned*. A warning was
reported about uint8_t shifted left 24 bits that couldn't fit in int
due to this.
It was verified that the emitted code didn't change, as expected, but
at least this allows to silence the code checkers. There's no need to
backport this.
The following directories were moved from contrib/ to dev/ to make their
use case a bit clearer. In short, only developers are expected to ever
go there. The makefile was updated to build and clean from these ones.
base64/ flags/ hpack/ plug_qdisc/ poll/ tcploop/ trace/
The various hpack files are self-contained, but hpack-tbl was one of
those showing difficulties when pools were added because that began
to add quite some dependencies. Now when built in standalone mode,
it still uses the bare minimum pool definitions and doesn't require
to know the prototypes anymore when only the structures are needed.
Thus the files were moved verbatim except for hpack-tbl which was
split between types and prototypes.
All files that were including one of the following include files have
been updated to only include haproxy/api.h or haproxy/api-t.h once instead:
- common/config.h
- common/compat.h
- common/compiler.h
- common/defaults.h
- common/initcall.h
- common/tools.h
The choice is simple: if the file only requires type definitions, it includes
api-t.h, otherwise it includes the full api.h.
In addition, in these files, explicit includes for inttypes.h and limits.h
were dropped since these are now covered by api.h and api-t.h.
No other change was performed, given that this patch is large and
affects 201 files. At least one (tools.h) was already freestanding and
didn't get the new one added.
I found on an (old) AIX 5.1 machine that stdint.h didn't exist while
inttypes.h which is expected to include it does exist and provides the
desired functionalities.
As explained here, stdint being just a subset of inttypes for use in
freestanding environments, it's probably always OK to switch to inttypes
instead:
https://pubs.opengroup.org/onlinepubs/009696799/basedefs/stdint.h.html
Also it's even clearer here in the autoconf doc :
https://www.gnu.org/software/autoconf/manual/autoconf-2.61/html_node/Header-Portability.html
"The C99 standard says that inttypes.h includes stdint.h, so there's
no need to include stdint.h separately in a standard environment.
Some implementations have inttypes.h but not stdint.h (e.g., Solaris
7), but we don't know of any implementation that has stdint.h but not
inttypes.h"
h2spec reported that we didn't check that no more than 7 bits of padding
were left after decoding an huffman-encoded literal. This is harmless but
better fix it now.
To backport to 1.8.
The code was borrowed from the HPACK experimental implementations
available here :
https://github.com/wtarreau/http2-exp
It contains the Huffman table as specified in RFC7541 Appendix B, and a
set of reverse tables used to decode a Huffman byte stream, and produced
by contrib/h2/gen-rht. The encoder is not finalized, it doesn't emit the
byte stream but this is not needed for now.
