We used to parse qualified table names as a simple string, which then
breaks attempts to be smart about how to quote idenfifiers. Some sources
are known to accept dots in quoted table names and we need to be able to
process that properly without tripping on qualified table names too
late.
Current code might not be the best approach as it's just using either a
cons or a string for table names internally, rather than defining a
proper data structure with a schema and a name slot.
Well, that's for a later cleanup patch, I happen to be lazy tonight.
Define a bunch of OS return codes and use them wisely, or at least in a
better way than just doing (uiop:quit) whenever there's something wrong,
without any difference whatsover to the caller.
Now we return a non-zero error code when we know something wrong did
happen. Which is more useful.
Per gripe from Marcos, who argues that for a human readable format
breaking when table names are wider than expected at compile time is
quite a strange position to defend.
See test/parse/hans.goeuro.load for an example usage of the new option.
In passing, any error when creating indexes is now properly reported and
logged, which was missing previously. Oops.
This option is dangerous and allows to skip ALL triggers when loading
data against PostgreSQL. This includes foreign key constraints
definitions and will allow loading data out of order.
When using both the options "create no table" and "disable triggers" it
will be possible to load data into a schema prepared by your favorite
external tool, at the cost of not validating FK constraints. Use with
care.
Fix#167.
The default for MS SQL float types is to only have a precision defined,
as described in https://msdn.microsoft.com/en-us/library/ms173773.aspx,
but the pgloader code didn't know what to do with a float without scale.
It appears that db3 files are not limited to the ASCII character
encoding that they were designed with, so let's clue pgloader about
that.
This commit build
770cbe3526
and the pgloader Makefile has been updated to momentarily fetch cl-db3
from github rather than Quicklisp so that it's possible to enjoy the new
feature immediately.
The command parser for DBF sources wasn't (anymore?) considering the
value of the "table name = " option, only considering the option from
the connection string.
Make it so that the option now takes preference over the DSN tablename
option when both are set.
This is a blind patch given that I couldn't CREATE TABLE as per the bug
report to try and see by myself what's happening. Better have some tests
going on though.
PostgreSQL requires that an idenfitier begin with letters or underscore
only, so an identifier that begins with a digit must be quoted. In the
current coding pgloader will unecessarily quote some identifiers that
begin with a unicode accentuated letter, but that's only cosmetic and
isn't worth worrying about (famous last words).
That should help fixing #158 where MS SQL uses the following name for
one of its fkey: fk_dbo.track_dbo.artist_artistid. PostgreSQL refuses fk
names with dots in it.
The option currently only works within the same build environment where
the image was first build, as noted in #133. This is an attempt at
convincing ASDF not to load systems that pgloader depends on in order to
be able to load only the new pgloader definition.
While it looks sound in principle, I failed to have it work in the lab.
Given that previous to this patch nothing works at all, it's not a
regression, let's push it as is makes the code saner.
Also, it looks like asdf::*immutable-systems* is what we want here, but
that's asdf 3.1.x and we're not there yet.
Loading external libs at application startup time is not convenient as
it forces users to install freetds everywhere even when they don't need
it. This patch makes it so that freetds is only loaded when pgloader is
asked to load from a MS SQL database source.
Note that we could have done the same for SSL if it wasn't possibly used
to connect to PostgreSQL, which isn't optional in current pgloader
implementation.
It's now possible to have pgloader print out its summary in one of
several formats: human-readable (default), csv, copy or json. The
choice of format is made depending on the extension of the summary
filename picked on the command line with the option --summary.
Fix bugs related to parsing the new COPY type, and make it so that we
know how to parse options (and fields, and other type dependant things)
even when --type is missing, in care the source URL has the information.
PostgreSQL COPY format is not really CSV but something way easier to
parse. Funnily enough, parsing it as CSV is not that easy, so we add
here a special simple parser for the COPY format.
It should be quite useful too try loading again reject data files from
pgloader after manual fixing, too. It's still missing some documentation
without any good excuse for that, will add soon.
Also augment the documentation with examples of bare stdin reading and
of advantages of the unix pipes to stream even remove archived content
down to PostgreSQL.
In passing also allow --field to specify the whole field list, there's
no point in forcing the user to have as many --field switches on the
command line as they have columns in their data source file.
