It appears that db3 files are not limited to the ASCII character
encoding that they were designed with, so let's clue pgloader about
that.
This commit build
770cbe3526
and the pgloader Makefile has been updated to momentarily fetch cl-db3
from github rather than Quicklisp so that it's possible to enjoy the new
feature immediately.
Given the slashdot effect and some bad luck, the binary artefacts of the
3.2.0 release are not currently available, and anyway contain known bugs
that have been fixed meanwhile thanks to early adopters who did open
issues on github.
So we hastily publish the current master's branch version as a github
release with binary files.
The bug is related to the processing of empty-lines in the middle of
quoted text by cl-csv, which state machine has gotten quite complex to
be able to handle all the crazy different csv variants out there.
Testing shows the bug is fixed in pgloader by just updating cl-csv.
We got unsync again with the qmynd driver (see #124) and the WIP branch
of esrap has been in the works for a long time now. We now manually
fetch the "proper" version of those.
Now it's possible to parse a command to load data from MS SQL. The
parser was until now parsing all database URI within the same common
rule and that isn't possible anymore if we want to distinguish in
between source database right from the parser, which we actually want to
do.
This patch also implement in-passing fixes all over the place, including
the transformation function float-to-string that only happened to work
on double-float data.
Turns out that debian has mawk by default, which is not behaving the
same in our very simple use case already. In passing, add gawk as a
build dependency of the debian package, because the packaging is meant
to exercize the test cases.
As seen in #64 it's no longer necessary anymore to use a local clone of
qmynd to be able to compile pgloader: simplify the Makefile accordingly.
In passing, add source level dependencies so that if you edit any source
lisp file the binary will get automatically rebuilt by `make`.
MacOSX users will be at home when using the usual packaging installer.
The binary file is installed into /usr/local/bin/pgloader and the man
page is installed too.
We need a different buildapp binary file for SBCL and for CCL, so make
it appear that way in the Makefile, and have both
./build/bin/buildapp.sbcl and ./build/bin/buildapp.ccl.
That avoid really confusing error messages when trying to build pgloader
with CCL and using the SBCL-compiled buildapp binary...
For the generated binary to be really portable, we need to be able to
open openssl 1.0.1 even when we've been built against openssl 1.0.0.
A way to achieve that with SBCL is by forcing the unloading of the lib
at image saving time and register a hook to load it again at image init
time. Using the proper API, CFFI will happily load the available file
for the lib rather than insisting on loading the exact same one than
found on the build machine.
Try harder not to mix Common Lisp Developers profiles with main pgloader
expected audience who will not have a Quicklisp system ready, by doing
our magic Quicklisp calls in a pgloader specific directory.
That might impact #45 badly as the homebrew formula should need to now
install its Quicklisp distributions elsewhere, but it should be easy
enough to fix.
Also, systematically `git pull` from the local-projects dependencies we
need to work with to avoid some bad surprises to our users.
This message has the line number where the erroneous data was found on the
server, and given the pre-processing we already done at that point, it's
easy to convert that number into an index into the current batch, an array.
To do do, we need Postmodern to expose the CONTEXT error message and we need
to parse it. The following pull request cares about the Postmodern side of
things:
https://github.com/marijnh/Postmodern/pull/46
The parsing is done as simply as possible, only assuming that the error
message is using comma separators and having the line number in second
position. The parsing as done here should still work with localized message
strings.
CONTEXT: COPY errors, line 3, column b: "2006-13-11"
This change should significantly reduce the cost of error processing.
