Commit Graph

35 Commits

Author SHA1 Message Date
Tuan Le
14cf7c0955 Initialize MSSQL library when used 2024-01-16 18:17:01 +01:00
Dimitri Fontaine
2fef253d28 Implement AFTER CREATE SCHEMA DO for more sources.
It was only implemented for Postgres sources even though the implementation
is generic enough to be shared. It's only a matter of instructing our parser
about the new facility, which this patch does.

Fixes #1062.
2020-02-11 22:20:08 +01:00
ChristophKaser
36fbadded6 Allow "preserve index names" for MSSQL (#902)
With this change, preserve index names is also supported for MSSQL-connections.
2019-02-05 11:08:23 +03:00
Dimitri Fontaine
c019c16113 Implement MATERIALIZE VIEWS support for MS SQL, and distribute.
The latter is not tested yet, but should have no impact if not used. Given
how rare it is that I get a chance to play around with a MS SQL instance
anyway, it might be better to push blind changes for it when it doesn't
impact existing features…
2018-12-19 01:25:27 +01:00
Dimitri Fontaine
5c10f12a07 Fix duplicate package names.
In a previous commit we re-used the package name pgloader.copy for the now
separated implementation of the COPY protocol, but this package was already
in use for the implementation of the COPY file format as a pgloader source.

Oops.

And CCL was happily doing its magic anyway, so that I've been blind to the
problem.

To fix, rename the new package pgloader.pgcopy, and to avoid having to deal
with other problems of the same kind in the future, rename every source
package pgloader.source.<format>, so that we now have pgloader.source.copy
and pgloader.pgcopy, two visibily different packages to deal with.

This light refactoring came with a challenge tho. The split in between the
pgloader.sources API and the rest of the code involved some circular
depencendies in the namespaces. CL is pretty flexible here because it can
reload code definitions at runtime, but it was still a mess. To untangle it,
implement a new namespace, the pgloader.load package, where we can use the
pgloader.sources API and the pgloader.connection and pgloader.pgsql APIs
too.

A little problem gave birth to quite a massive patch. As it happens when
refactoring and cleaning-up the dirt in any large enough project, right?

See #748.
2018-02-24 19:24:22 +01:00
Dimitri Fontaine
f86371970f Review the pgloader COPY implementation further.
Refactor file organisation further to allow for adding a “direct stream”
option when the on-error-stop behavior has been selected. This happens
currently by default for databases sources.

Introduce the new WITH option “on error resume next” which forces the
classic behavior of pgloader. The option “on error stop” already existed,
its implementation is new.

When this new behavior is activated, the data is sent to PostgreSQL
directly, without intermediate batches being built. It means that the whole
operation fails at the first error, and we don't have any information in
memory to try replaying any COPY of the data. It's gone.

This behavior should be fine for database migrations as you don't usually
want to fix the data manually in intermediate files, you want to fix the
problem at the source database and do the whole dance all-over again, up
until your casting rules are perfect.

This patch might also incurr some performance benenits in terms of both
timing and memory usage, though the local testing didn't show much of
anything for the moment.
2018-01-24 22:45:23 +01:00
Dimitri Fontaine
01e5c23763 Add support for explicit TARGET TABLE clause in load commands.
It used to be that you would give the target table name as an option to the
PostgreSQL connection string, which is untasteful:

   load ... into pgsql://user@host/dbname?tablename=foo.bar ...

Or even, for backwards compatibility:

   load ... into pgsql://user@host/dbname?foo.bar ...

The new syntax makes provision for a separate clause for the target table
name, possibly schema-qualified:

   load ... into pgsql://user@host/dbname target table foo.bar ...

Which is much better, in particular when used together with the target
columns clause.

Implementing this seemingly quite small feature had impact on many parsing
related features of pgloader, such as the regression testing facility. So
much so that some extra refactoring got into its way here, around the
lisp-code-for-loading-from-<source> functions and their usage in
`load-data'.

While at it, this patch simplifies a lot the `load-data' function by making
a good use of &allow-other-keys and :allow-other-keys t.

Finally, this patch splits main.lisp into main.lisp and api.lisp, with the
latter intended to contain functions for Common Lisp programs wanting to use
pgloader as a library. The API itself is still the same as before this
patch, tho. Just in another file for clarity.
2017-08-25 01:57:54 +02:00
Dimitri Fontaine
f719d2976d Implement a template system for pgloader commands.
This feature has been asked several times, and I can't see any way to fix
the GETENV parsing mess that we have. In this patch the GETENV support is
retired and replaced with a templating system, using the Mustache syntax.

To get back the GETENV feature, our implementation of the Mustache template
system adds support for fetching the template variable values from the OS
environment.

Fixes #555, Fixes #609.
See #500, #477, #278.
2017-08-16 01:33:11 +02:00
Dimitri Fontaine
20a85055f4 Implement support for MS SQL set parameters.
It is sometimes needed to tweak MS SQL server parameters, such as the
textsize parameters which allows fetching the whole set of bytes of a text
of binary column (not kidding).

Now it's possible to add such a line in the load file:

  set mssql parameters textsize to '104857600'

Fixes #603.
2017-08-12 23:43:22 +02:00
Dimitri Fontaine
7a371529be Implement "drop indexes" option for MySQL and MSSQL too.
It was only offered for SQLite without good reason really, and tests show
that it works as well with MySQL of course. Offer the option there too.

See 3eab88b144 for details.
2017-07-06 10:06:03 +02:00
Dimitri Fontaine
6d66280fa5 Review parallelism and memory behavior.
The previous patch made format-vector-row allocate its memory in one go
rather than byte after byte with vector-push-extend. In this patch we review
our usage of batches and parallelism.

Now the reader pushes each row directly to the lparallel queue and writers
concurrently consume from it, cook batches in COPY format, and then send
that chunk of data down to PostgreSQL. When looking at runtime profiles, the
time spent writing in PostgreSQL is a fraction of the time spent reading
from MySQL, so we consider that the writing thread has enough time to do the
data mungling without slowing us down.

The most interesting factor here is the memory behavor of pgloader, which
seems more stable than before, and easier to cope with for SBCL's GC.

Note that batch concurrency is no more, replaced by prefetch rows: the
reader thread no longer build batches and the count of items in the reader
queue is now a number a rows, not of batches of them.

Anyway, with this patch in I can't reproduce the following issues:

Fixes #337, Fixes #420.
2017-06-27 23:10:33 +02:00
Dimitri Fontaine
c6b634caad Provide "on error stop" as a WITH option.
As seen in #546 it would be easier to be able to specify the option in the
load command directly rather than only at the command line. Here we go!
2017-06-01 16:43:09 +02:00
Dimitri Fontaine
42e9e521e0 Add option "max parallel create index".
By default, pgloader will start as many parallel CREATE INDEX commands
as the maximum number of indexes you have on any single table that takes
part in the load.

As this number might be so great as to exhaust the target PostgreSQL
server (e.g. maintenance_work_mem), we add an option to limit that to
something reasonnable when the source schema isn't.

Fix #386 in which 150 indexes are found on a single source table.
2016-04-11 17:40:52 +02:00
Dimitri Fontaine
35155654df Allow to ALTER TABLE ... IN SCHEMA.
That brings the ALTER TABLE feature to MS SQL source.
2016-03-26 20:50:05 +01:00
Dimitri Fontaine
fcc6e8f813 Implement ALTER SCHEMA ... RENAME TO...
That's only available for MS SQL as of now, as it's the only source
database we have where the notion of a schema makes sense. Fix #224.
2016-03-26 20:25:03 +01:00
Dimitri Fontaine
f1fe9ab702 Assorted fixes to MS SQL support.
Having been given a test instance of a MS SQL database allows to quickly
fix a series of assorted bugs related to schema handling of MS SQL
databases. As it's the only source with a proper notion of schema that
pgloader supports currently, it's not a surprise we had them.

Fix #343. Fix #349. Fix #354.
2016-03-16 21:43:04 +01:00
Dimitri Fontaine
eb45bf0338 Expose concurrency settings to the end users.
Add the workers and concurrency settings to the LOAD commands for
database sources so that users can tweak them now, and add mentions of
them in the documentation too.

From the documentation string of the copy-from method as found in
src/sources/common/methods.lisp:

   We allow WORKER-COUNT simultaneous workers to be active at the same time
   in the context of this COPY object. A single unit of work consist of
   several kinds of workers:

     - a reader getting raw data from the COPY source with `map-rows',
     - N transformers preparing raw data for PostgreSQL COPY protocol,
     - N writers sending the data down to PostgreSQL.

   The N here is setup to the CONCURRENCY parameter: with a CONCURRENCY of
   2, we start (+ 1 2 2) = 5 concurrent tasks, with a CONCURRENCY of 4 we
   start (+ 1 4 4) = 9 concurrent tasks, of which only WORKER-COUNT may be
   active simultaneously.

Those options should find their way in the remaining sources, that's for
a follow-up patch tho.
2016-01-15 23:22:32 +01:00
Dimitri Fontaine
fb40a472ab Simplify database WITH option handling.
Share more code by having a common flattening function as a semantic
predicate in the grammar.
2016-01-15 22:34:27 +01:00
Dimitri Fontaine
9e4938cea4 Implement PostgreSQL catalogs data structure.
In order to share more code in between the different source types,
finally have a go at the quite horrible mess of anonymous data
structures floating around.

Having a catalog and schema instances not only allows for code cleanup,
but will also allow to implement some bug fixes and wishlist items such
as mapping tables from a schema to another one.

Also, supporting database sources having a notion of "schema" (in
between "catalog" and "table") should get easier, including getting
on-par with MySQL in the MS SQL support (materialized views has been
asked for already).

See #320, #316, #224 for references and a notion of progress being made.

In passing, also clean up the copy-databases methods for database source
types, so that they all use a fetch-metadata generic function and a
prepare-pgsql-database and a complete-pgsql-database generic function.
Actually, a single method does the job here.

The responsibility of introspecting the source to populate the internal
catalog/schema representation is now held by the fetch-metadata generic
function, which in turn will call the specialized versions of
list-all-columns and friends implementations. Once the catalog has been
fetched, an explicit CAST call is then needed before we can continue.

Finally, the fields/columns/transforms slots in the copy objects are
still being used by the operative code, so the internal catalog
representation is only used up to starting the data copy step, where the
copy class instances are then all that's used.

This might be refactored again in a follow-up patch.
2015-12-30 21:53:01 +01:00
Dimitri Fontaine
e23de0ce9f Improve SQLite table names filtering.
Filter the list of tables we migrate directly from the SQLite query,
avoiding to return useless data. To do that, use the LIKE pattern
matching supported by SQLite, where the REGEX operator is only available
when extra features are loaded apparently.

See #310 where filtering out the view still caused errors in the
loading.
2015-11-22 22:10:26 +01:00
Dimitri Fontaine
96a33de084 Review the stats and reporting code organisation.
In order to later be able to have more worker threads sharing the
load (multiple readers and/or writers, maybe more specialized threads
too), have all the stats be managed centrally by a single thread. We
already have a "monitor" thread that get passed log messages so that the
output buffer is not subject to race conditions, extend its use to also
deal with statistics messages.

In the current code, we send a message each time we read a row. In some
future commits we should probably reduce the messaging here to something
like one message per batch in the common case.

Also, as a nice side effect of the code simplification and refactoring
this fixes #283 wherein the before/after sections of individual CSV
files within an ARCHIVE command where not counted in the reporting.
2015-10-05 01:46:29 +02:00
Dimitri Fontaine
f6aa8210b9 Load libsybdb even in --dry-run mode, fix #295. 2015-09-16 20:39:42 +02:00
Dimitri Fontaine
92d27f4f98 Allow quote/downcase identifiers option for MS SQL.
As seen in #287 the previous decision to force quoting to :none is
wrong, because index names in MS SQL source database might contain
spaces, and then need to be quoted.

Let's see what happens if we do it the usual way for MS SQL too, and
allow users to control the quoting behaviour of pgloader here.
2015-09-03 23:34:25 +02:00
Dimitri Fontaine
ea35eb575d Implement --dry-run option, fix #264.
The dry run option will currently only check database connections, but
as that happens after having correctly parsed the load file, it allows
to also check that the command file is correct for the parser.

Note that the list load-data API isn't subject to the dry-run method.

In passing, we add some more API entry points to the connection objects
and we should actually clean the code base to use the new QUERY generic
all over the place. It's for another patch tho.
2015-08-22 16:23:47 +02:00
Dimitri Fontaine
178210b6f8 Fix 273 by renaming the new create-schemas option properly. 2015-08-18 17:38:04 +02:00
Dimitri Fontaine
6fc40c4844 Implement MS SQL option to skip creating schemas, fix #263.
Allow the user to control whether pgloader should create the same set of
schema as found on the MS SQL database.
2015-08-15 16:10:15 +02:00
zickzackv
5e1ebe0dd8 add missing conditional compilation for sbcl
Added missing #+sbcl for sb-ext:*muffled-warnings*.  prevented succesfully building the application with ccl
2015-01-18 22:48:35 +01:00
Dimitri Fontaine
5f520737e6 Improve loading of external libs, fixes #142.
Loading external libs at application startup time is not convenient as
it forces users to install freetds everywhere even when they don't need
it. This patch makes it so that freetds is only loaded when pgloader is
asked to load from a MS SQL database source.

Note that we could have done the same for SSL if it wasn't possibly used
to connect to PostgreSQL, which isn't optional in current pgloader
implementation.
2015-01-07 22:23:12 +01:00
Dimitri Fontaine
302a7d402b Refactor connection handling, and clean-up many things.
That's the big refactoring patch I've been sitting on for too long.

First, refactor connection handling to use a uniformed "connection"
concept (class and generic functions API) everywhere, so that the COPY
derived objects just use that in their :source-db and :target-db slots.

Given that, we don't need no messing around with *pgconn* and *myconn-*
and other special variables at all anywhere in the tree.

Second, clean up some oddities accumulated over time, where some parts
of the code didn't get the memo when new API got into place.

Third, fix any other oddity or missing part found while doing those
first two activities, it was long overdue anyway...
2014-12-26 21:50:29 +01:00
Dimitri Fontaine
65c2043694 Improve pgloader usage from the command line.
Make it so that the following command line usages are accepted when
using pgloader without a command file:

 ./build/bin/pgloader ./test/sqlite/sqlite.db postgresql:///pgloader

 ./build/bin/pgloader --set "search_path='sakila'"  \
                      mysql://root@localhost/sakila \
                 postgresql:///sakila

 ./build/bin/pgloader --type csv                             \
                      --field id --field field               \
                      --with truncate                        \
                      --with "fields terminated by ','"      \
                      ./test/data/matching-1.csv             \
                      postgres:///pgloader?matching

It's now possible in most cases to just use command-line options, which
should make the entry bar to pgloader much lower.
2014-12-23 02:40:13 +01:00
Dimitri Fontaine
34b9a908b0 Fix parsing of include and exclude patterns.
By reusing the same symbols in parsers/command-m[sy]sql.lisp the parser
was broken, keep using unique symbol names for different concepts...
2014-12-22 23:00:23 +01:00
Dimitri Fontaine
46be4a58b7 Implement INCLUDING and EXCLUDING ... IN SCHEMA for MS SQL. 2014-12-22 17:18:29 +01:00
Dimitri Fontaine
38acce57b5 Fix parsing MSSQL options, which are actually MySQL options. 2014-11-24 23:41:20 +01:00
Dimitri Fontaine
5b87b1a85e Refactor *identifier-case* option into a dynamic binding.
That makes it much easier to use from about anywhere in the code, which
is what is needed. In passing, fix #129.
2014-11-21 23:32:02 +01:00
Dimitri Fontaine
87e157bee2 Add a new database source type in the parser.
Now it's possible to parse a command to load data from MS SQL. The
parser was until now parsing all database URI within the same common
rule and that isn't possible anymore if we want to distinguish in
between source database right from the parser, which we actually want to
do.

This patch also implement in-passing fixes all over the place, including
the transformation function float-to-string that only happened to work
on double-float data.
2014-11-17 00:23:06 +01:00