The previous patch didn't take into account the need to retain the case
of the PostgreSQL column names when using double-quotes in the load
command, which is now properly forwarded down in the COPY command.
With this patch it's now possible to "quote" column names and to use
_ (underscore) as the leading character in a field or column name. Of
course it is still unnecessary to quote the column names, but handling
the PostgreSQL quoting rules can't be bad here.
The backport takes a long time, and with the output redirected away the
Travis build system kills the build with the following message:
No output has been received in the last 10 minutes, this potentially
indicates a stalled build or something wrong with the build itself.
The build has been terminated.
Generalize what the previous patch did only for the MySQL case, this
time affecting the internal parser API expectations to "flag" all
clauses return values with a per-clause keyword so that an ordinary
lambda list keyword can then be destructured.
Only the MySQL command is addressed in this patch, because the code
level approach is not safisfying me completely. It might be easier to
just bite the bullet and review all the optional clauses return values
rather than add a layer as this patch does.
The feature still is available for MySQL given this patch, so let's push
it, get feedback, then see about how to make the approach scale and
revise all the other commands.
Given than redirecting a tty such as *terminal-io* isn't easy enough,
let's provide a way to copy the summary output to a file. Another way to
solve it would have been to output the summary to the main logs, but
that could have made the logs parsing more difficult that necessary.
Let's see how users like it...
In SQLite it's possible to define columns using type names such as
"smallint unsigned" or "short integer", without any changes to the way
those data types are handled, given its "dynamic typing" features.
Improve the pgloader casting machinery for SQLite to handle those cases.
Turns out that debian has mawk by default, which is not behaving the
same in our very simple use case already. In passing, add gawk as a
build dependency of the debian package, because the packaging is meant
to exercize the test cases.
Those tests currently only work when a single table is the target of the
load, and when this target is explicit in the INTO target clause. More
work needs to be done to cover interesting cases like MySQL and SQLite
where we want to diff a full database rather than a single table.
The census test didn't pass anymore because I broke the archive filename
matching in b17383fa90, where the special
variable *csv-path-root* stoped being authoritative in the archive case.
To fix, initialize that variable to nil and give its value priority as
soon as it's non-nil, such as the archive case.
When adding the CONTEXT message parsing I totally forgot that PostgreSQL
provides a nice error message translation capability. The code now copes
better with the situation, using a more advanced regular expression.
We could inline the known translations in the matching, but that would
be tedious to maintain, so we just use loose matching rules here.
Should help with issue #67 by allowing --client-min-messages to
effectively control entering the debugger in case of unhandled
conditions, etc.
Contrary to the discussion, in this patch --log-min-messages has no
impact on the behavior of the console and interactive behaviors.
In case of PostgreSQL schema preparation error, and when some
materialized views where given with their SQL command, they were left
over by pgloader. The next run would then fail because the view already
exists at CREATE VIEW time.
Fix that by cleaning up materialized views we just created in handling
any condition signaled when preparing the PostgreSQL schema.
When in :data logging mode we log the whole data set as we read then
write it, which is quite a lot of data. Our current logging system works
by filling up a queue that the cl-log lib is then fed from, and sending
lots of data in that queue is way expensive, stop doing that.
Hopefully we don't need to revisit the logs more than that, the other
messages should be few enough not to count much when doing a full load.
With this the user is now able to have a way about where the files are
going to be read and matched against the regular expression. It used not
to be necessary in the archive expansion mode, but is required now that
the feature is exposed in more cases.
When using LOAD CSV it's possible to load from filename matching a
regular expression, but for that to work the *csv-path-root* needs to be
properly setup at run-time.
As seen in #64 it's no longer necessary anymore to use a local clone of
qmynd to be able to compile pgloader: simplify the Makefile accordingly.
In passing, add source level dependencies so that if you edit any source
lisp file the binary will get automatically rebuilt by `make`.
The code forgot completely that MySQL column name references in foreign
key definitions have to follow the identifier case rules, this patch fix
that.
To be able to do that, we need to parse the GROUP_CONCAT() result that
lists the FK columns, as there's apparently no arrays in MySQL. The
problem here is that about any character is allowed in column names when
`quoted`, so using a comma here might reveal to be fragile later.
