956 Commits

Author SHA1 Message Date
Dimitri Fontaine
e2418891a4
Fix looping over sbcl *external-formats*.
The internal represtation of the SBCL *external-formats* has changed to a
new structure which is not an hash-table anymore.
2022-06-23 17:15:47 +02:00
Scott Thomas
7e1f7c51c8 Allow underscores in cast type names
Underscores might exist in cast type names, for example if the type being cast is an enum (which can have arbitrary names).

Fixes #1378
2022-06-23 13:43:21 +02:00
Athos Ribeiro
248c2f709a Force libcrypto reload in src/hooks.lisp
cl+ssl::libcrypto is also read at startup. If not properly closed in the
hooks, together with libssl, libcrypto will be loaded at startup and if
the first cl+ssl file alternative fails, the debugger is invoked.

* Fixes #1370
2022-06-23 13:28:35 +02:00
Dmitry Shvetsov
2e728f5754 Fix typo in the log message 2021-12-21 10:10:38 +01:00
lukesilvia
48d8ed0613
fix: ranged load does not load last record. (#1203) 2020-08-31 20:12:32 +02:00
Aleksi Kinnunen
9788cc64ee
Add MySQL unsigned int casting rules (#1200)
Fixes dimitri/pgloader#1186
2020-08-28 11:24:36 +02:00
Dimitri Fontaine
e388909f0c Implement a retry loop when SQLite database is "BUSY".
It turns our that you can't do some operations on SQLite from several
concurrent connections, such as a "pgrama encoding" query.

Fixes #1193.
2020-07-27 21:12:55 +02:00
Dimitri Fontaine
f5139cbf29 Add support for DATETIME() function call as a SQLite default values.
Fixes #1177.
2020-07-03 18:39:37 +02:00
Dimitri Fontaine
689dd4a806 Simplify Postgres version string parsing further. 2020-06-06 17:07:06 +02:00
Dimitri Fontaine
cc8975bb88 Improve Postgres version string parsing.
Turns out we have way more variety in the field than what I anticipated...

Should fix #992.
2020-06-05 18:47:32 +02:00
Dimitri Fontaine
2189acfb63 Attempt at fixing #1060. 2020-06-05 18:22:47 +02:00
Dimitri Fontaine
1bdc0ee5f4 Allow SQLite type names such as "double precision".
The parsing of the type names from the SQLite catalogs needs to allow for
names with spaces in them, because SQLite allows that too.

Fixes #921.
2020-05-09 23:55:59 +02:00
Semen Miroshnichenko
11d926126e
Keep name casing for foreign keys on mssql (#1145) 2020-05-09 17:49:04 +02:00
Dimitri Fontaine
d5314a6640 Add support for type "signed long" in SQLite.
Fixes #1126.
2020-04-11 18:33:21 +02:00
Ian L
cb989e1155
Fix MySQL query to compute number of rows per table (#1128)
The CAST should target `unsigned` rather than `integer`, so that we are compatible with MySQL 5.7. Also empty tables might have NULL entries, which we transform to zero entries here, as expected by the Lisp code.

Fixes #1127.
2020-04-11 18:22:02 +02:00
Dimitri Fontaine
86b6a5cb80 We COPY the MS SQL data in the MS SQL column ordering.
Fix #1124.
2020-04-05 15:12:46 +02:00
Dimitri Fontaine
11970bbca8
Implement tables row count ordering for MySQL. (#1120)
This should help optimise the duration of migrating databases with very big
tables and lots of smaller ones. It might be a little too naive as far as
the optimisation goes, while still being an improvement on the default
alphabetical one.

Fixes #1099.
2020-04-04 16:40:53 +02:00
Dimitri Fontaine
14fb15bfbd Force the summary file to be opened in UTF-8.
After all the default summary output contains the “✓” character and that
won't fit in the ascii external format.

Fixes #1103.
2020-04-03 23:49:20 +02:00
Dimitri Fontaine
64643bff83 Fix MS SQL bigint casting to numeric.
Fixes #937.
2020-03-28 00:00:21 +01:00
Rudi Bruchez
c2b9f79413
handling mssql datetimeoffset (#1113) 2020-03-27 23:50:35 +01:00
Dimitri Fontaine
b3cd5f28d6 Set the MS SQL port in the environment.
As our API to connect to MS SQL does not provide a facility to set the
target port number, place it in the TDSPORT environment variable, which is
reported to have the expected impact.

Should fix #1094.
2020-03-27 23:24:16 +01:00
Svante von Erichsen
8c59f8c9f9
fix typo that chooses wrong option rule (#1107) 2020-03-27 23:14:58 +01:00
Michał "phoe" Herda
6b111ba483
Begin v3.6.3 development 2020-03-22 22:38:46 +01:00
Michał "phoe" Herda
484d3e1dd4
Release v3.6.2 2020-03-22 22:26:18 +01:00
Dimitri Fontaine
c899d3b5c4 Fix zero-dates-to-null.
Fixes #1098.
2020-03-22 15:56:57 +01:00
Nicolas Delperdange
cc2dc8d671
fix(mysql): use new st_astext function (#1100) 2020-03-22 15:47:57 +01:00
Dimitri Fontaine
0daace9d70 Allow source type names to be doube-quoted in CAST rules.
Fixes #1015.
2020-03-21 17:37:54 +01:00
Dimitri Fontaine
30376b2cfe Add support for "datetime2" data type in MS SQL.
That's a blind fix, or almost blind: we're given some information but I'm
not in a position to test the fix myself. Hope it works.

Fixes #1036.
Fixes #1018.
2020-02-29 22:23:58 +01:00
Dimitri Fontaine
df94340396 Avoid parser look-ahead when not necessary.
Fixes #1082 where letters from the URI are doubled because of the
look-ahead in the previous parser's implementation.

Thanks @svantevonerichsen6906 for the fix!
2020-02-29 22:04:06 +01:00
Dimitri Fontaine
3b5c29b030 Attempt to fix foreign-key creation to tables that have been filtered out.
See #1016 where we try to build the DDL for a foreign key that references
tables that are not found in our catalogs. We should probably just ignore
those foreign keys, as we might have a partial load to implement.
2020-02-12 21:50:18 +01:00
Dimitri Fontaine
bbcce92418 Allow underscores in SQLite type names.
Fixes #1049.
2020-02-12 00:07:58 +01:00
Dimitri Fontaine
d4da90648e Implement proper hostname parsing, following labels specifications.
Bug report #1053 was the occasion to wander in the specification for DNS
hostnames and their label components, and the syntactic rules for those. It
turns out that my implementation was nothing like the specs:

  https://en.wikipedia.org/wiki/Domain_Name_System#Domain_name_syntax

Fixes #1053.
2020-02-12 00:01:07 +01:00
Dimitri Fontaine
e551099463 Attempt at fixing a MS SQL bug.
That's another blind fix, but it looks like this should be doing it.
Hopefully. Chances to break pgloader even more are quite slim, so let's try
it in the master branch.

Fixes #1069.
2020-02-11 22:43:02 +01:00
Dimitri Fontaine
2fef253d28 Implement AFTER CREATE SCHEMA DO for more sources.
It was only implemented for Postgres sources even though the implementation
is generic enough to be shared. It's only a matter of instructing our parser
about the new facility, which this patch does.

Fixes #1062.
2020-02-11 22:20:08 +01:00
Dimitri Fontaine
8a13c02561 Review our Postgres catalog usage for default values.
The pg_catalog.pg_attribute column adsrc is populated only once, and then
not maintained by the server. The function pg_get_expr(adbin, adrelid) is
what to use and has been here since release 8.0, so that's what pgloader
should be using.

Fixed #1934.
2020-02-11 21:46:10 +01:00
Dimitri Fontaine
26cc9ca79f Implement support for DBF deleted records.
Skip over deleted records rather than blindly importing them. Requires an
update of the underlying DBF library.

Fixes #985.
2019-06-10 22:58:12 +02:00
Dimitri Fontaine
d8b0bd5145 Allow casting rules to guard on signed data types.
It used to be that our casting rules mechanism would allow for matching
unsigned data types only, and we sometimes have a need to do special
behavior on signed data types.

In particular, a signed bigint(20) in MySQL has the same values range as a
PostgreSQL bigint, so we don't need to target a numeric in that case. It's
only when the bigint is unsigned that we need to target a numeric.

In passing update some of the default casting rules documentation to match
the code.

Fix #982.
2019-06-04 15:22:25 +02:00
Dimitri Fontaine
b8da7dd2e9
Generic Function API for Materialized Views support. (#970)
Implement a generic-function API to discover the source database schema and
populate pgloader internal version of the catalogs. Cut down three copies of about
the same code-path down to a single shared one, thanks to applying some amount
of OOP to the code.
2019-05-20 19:28:38 +02:00
Dimitri Fontaine
12e788094b Improve sexp parser and standard symbols support.
Also add split-sequence to the list of special cases that we can use and is
not found in the pgloader.transforms package.

Fixes #965.
2019-05-14 15:49:24 +02:00
Dimitri Fontaine
501cbed745 Quote database name in ALTER DATABASE "..." SET search_path TO
Fixes #933.
2019-05-12 00:50:59 +02:00
Dimitri Fontaine
350cffffad Improve DBF support.
The cl-db3 lib just got improvements for new dbase file types and field
types, reflect those in pgloader.

Also, cl-db3 now can read the encoding of the file (language driver)
directly in the header, meaning we can rely on that metadata by default, and
only override it when the users tells us to.

See #961.
2019-05-11 20:50:52 +02:00
Dimitri Fontaine
a51819f874 Have db3-numeric-to-pgsql-integer accept integer values.
The cl-db3 driver is now parsing type I fields and emit them as native
integers. We need to deal with that.

See #929.
2019-05-10 22:09:52 +02:00
Dimitri Fontaine
954eca02d0 Optimize Travis-CI workload.
Before this patch Travis would build the pgloader image twice, one when
doing the `make clones save` command and another time when doing the main
command `make check`, that depends on Make targets that `make save` does not
build, such as buildapp.

Also, fix `make check-saved` to stop requiring that we save pgloader again.

Finally, use uiop:featurep to clean-up the code in the src/save.lisp file.
2019-05-09 10:52:09 +02:00
Dimitri Fontaine
351ce3faaf Rename pgloader's own DB3 field structure.
That's to avoid a name conflict with the cl-db3 package. In SBCL 1.5.2 in
Travis this conflict is an hard error and provoque failure to build the
pgloader binary.
2019-05-09 10:41:40 +02:00
Dimitri Fontaine
ca92cdbf20 Introduce core-compression discovery in make save to fix Travis builds. 2019-05-09 10:27:20 +02:00
Dimitri Fontaine
2b6fb3e6c2 Add a casting rule for the I data type in DBF files.
The I data type is mapped to a PostgreSQL integer.

See #929.
2019-05-08 23:36:17 +02:00
Dimitri Fontaine
7d2e5ae941 Allow lambda expression as cast rules using functions.
Before that it was necessary to install a function in the lisp environment
either in the source itself in src/utils/transforms.lisp, or in a lisp file
loaded with --load-lisp-file (or -l for shorts).

While this could be good enough, sometimes a very simple combination of
existing features is required to transform a function and so doing some
level of lisp coding directly in the load command is a nice to have.

Fixes #961.
2019-05-08 19:02:18 +02:00
Dimitri Fontaine
b1d55e07d6 Implement header parsing for fixed file formats.
Fixed file formats might contain an header line with column names and a hint
of the size of each column. While it might be a long shot that we can
acutally use that as a proper fixed-format specification, this patch
implements a guess mode that also outputs the parsed header.

In case when the parsing is wrong in some level of details, it might
actually be a good start to copy/paste from the command output and go from
there.

Fixes #958.
2019-05-07 22:57:55 +02:00
Dimitri Fontaine
27b1a83b9f Allow quoted table and column names in user-defined CASTing rules.
Fixes #955.
2019-04-30 16:55:26 +02:00
Dimitri Fontaine
d4369cc605 Move DB3 transformations function in pglaoder.transforms package.
That makes them available for user defined cast rules.

Fix #929.
2019-04-29 16:52:24 +02:00