The current problem with Travis build is that it's using the Quicklisp
version of the cl-db3 librairy rather than the current master's branch
version, which current sources of pgloader depend on.
In the Dockerfiles we use the alternate build method for preparing the lisp
image, and invoke the `make clones` target explicitely in order to make sure
that we have the right set of dependencies at build time.
In this patch we apply the same approach to Travis, hoping to solve the
current build issues there.
Before that it was necessary to install a function in the lisp environment
either in the source itself in src/utils/transforms.lisp, or in a lisp file
loaded with --load-lisp-file (or -l for shorts).
While this could be good enough, sometimes a very simple combination of
existing features is required to transform a function and so doing some
level of lisp coding directly in the load command is a nice to have.
Fixes#961.
Fixed file formats might contain an header line with column names and a hint
of the size of each column. While it might be a long shot that we can
acutally use that as a proper fixed-format specification, this patch
implements a guess mode that also outputs the parsed header.
In case when the parsing is wrong in some level of details, it might
actually be a good start to copy/paste from the command output and go from
there.
Fixes#958.
The URL of the test case source has changed. Use the new one. Also set the
encoding properly, the client_encoding trick has been deprecated for awhile
now, as pgloader only talks to Postgres in UTF-8.
It seems that the current open-coding of the materialized views data
structure is still causing some confusion, so use the same code as in the
MySQL support in the MS SQL parts of the code when trying to fix#950.
See #950.
This is a blind fix: it looks like we forgot to take care of identifier case
when adding columns to foreign key definitions in MS SQL catalogs.
Fixes#953.
We migrate bit(xx) to the same PostgreSQL datatype bit(xx) where in Postgres
we can use bitstring as documented at the following URL. In particular the
COPY syntax accepts the notation Xabcd for the values, which is quite nice
when MySQL sends the data to us a a byte vector:
https://www.postgresql.org/docs/current/datatype-bit.htmlFixes#943.
That building script does not rely on buildapp, which might be an advantage
in docker environments: first it's one less dependency, second, we have seen
weird error cases when using buildapp.
Stop installing sbcl in the CCL image, that's not needed, and also make sure
we use a git clone of some of the libs rather than the current QL version of
them.
Fixes#926.
The previous fix left to be desired in that it didn't update the code
expectations about what a view-name looks like in fetch-metadata. In
particular we would use (cons NIL "table-name") in the only-table facility,
which expects non qualified names as strings.
Switch to using the :including filters facility instead, as we do in MS SQL.
Later, we might want to deprecate our internal :only-tables facility.
Fix#932. Again.
It turns out that current Emacs settings aren't agreeing with the file
indenting, let's clean that up before modifying the file, so that later
reviews are easier.
The casting support for DB3 was hand-crafted and didn't get upgraded to
using the current CAST grammar and facilities, for no other reasons than
lack of time and interest. It so happens what implementing it now fixes two
bug reports.
Bug #938 is about conversion defaulting to "not null" column, and that's due
to the usage of the internal pgloader catalogs where the target column's
nullable field is NIL by default, which doesn't make much sense. With
support for user-defined casting rules, the default is nullable columns, so
that's kind of a free fix.
Fixes#927.
Fixes#938.
When adding support for Mat Views to MS SQL, we added support for the view
names to be fully qualified (with their schema), using a cons to host
the (schema . name) data.
Well, turns out the MySQL side of things didn't get the memo.
Blind attempt at fixing #932, see also #918.
In some cases when migrating from MySQL we want to transform data from
binary representation to an hexadecimal number. One such case is going from
MySQL binary(16) to PostgreSQL UUID data type.
Fixes#904.
When using interactive recompiling of the code in Emacs/SLIME, extra closing
parens are just ignored in Emacs before sending the current form to the CL
compiler. When compiling from the source files, of course, that doesn't
work.
See #910.
It turns out that MS SQL Server is using its own representation for GUID
with a mixed endianness. In this blind patch we attempt to parse the binary
vector in the right way when building our internal representation of an
UUID, before making a string out of it for Postgres, which doesn't use the
same mixed-endianness format.
Fixes#910. Maybe?
When calling the create_distributed_table() function, the column name is
given as a literal parameter to the function and should be quoted that way,
with single quotes. In particular, if our column-name is already
double-quoted, we need to get rid of those extra quotes.
Also, the source-table-name might be a cons object when qualified, or a
plain string when not schema-qualified. Adjust the citus-find-table code to
take that into account.
pgloarder parses the COPY error messages to find out the line number where
we have a problem in the batch, allowing for a quite efficient recovery
mechanism where it's easy enough to just skip the known faulty input.
Now, some error messages do not contain a COPY line number, such as fkey
violation messages:
Database error 23503: insert or update on table "produtos" violates
foreign key constraint "produtos_categorias_produtos_fk"
In that case rather than failing the whole batch at once (thanks to the
previous commit, we used to just badly fail before that), we can retry the
batch one row at a time until we find our culprit, and then continue one
input row at a time.
Fixes#836.
We don't know how to parse the PostgreSQL condition sent when there is a
fkey error... and the message would not contain the row number where that
error happened anyway.
At the moment it means that the retry-batch facility errors out for failing
to realize that NIL isn't a number on which we can do arithmetic, which in
itself in a little sad.
In this patch we install a condition handler that knows how to deal with
retry-batch failing, so that pgloader may try and continue rather than
appear locked to the user, when I suspect that the debugger is waiting for
input.
See #836, where that's the first half of the fix. The real fix is to handle
foreign key errors correctly of course.
In some cases we have to quote column names and it's not been done yet, for
instance when dealing with PostgreSQL as a source database.
Patch mostly from @m0n5t3r, only cosmetic changes applied. Thanks!
Fixes#905.
When migrating from PostgreSQL, pgloader takes the index and foreign key
definitions from the server directly, using pg_get_indexdef() and other
catalog functions. That's very useful in that it embeds all the necessary
quoting of the objects, and schema-qualify them.
Of course we can't use the SQL definition as on the source system when we
target a schema name that is different from the source system, which the
code didn't realize before this patch. Here we simply invalidate the
pre-computed SQL statement and resort to using the classic machinery to
build the statement from pieces again.
Fixes#903.
When using a CSV header, we might find fields in a different order than the
target table columns, and maybe not all of the fields are going to be read.
Take account of the header we read rather than expecting the header to look
like the target table definition.
Fix#888.
Killing tasks in the error handling must be done carefully, and given this
testing session it seems better to refrain from doing it when erroring out
at COPY init time (missing column is an example of that). The approach
around that is still very much ad-hoc rather than systematic.
In passing improve the `make save` option to producing a binary image: have
the make recipe respect the CL variable. The command line options
differences were already accounted for.
This allows creating tables in any target tablespace rather than the default
one, and is supported for the various sources having support for the ALTER
TABLE clause already.
