Dimitri Fontaine
114d2fedbc
Another try at fixing #40 .
...
The babel character-decoding-error condition is exposing both its
internal BUFFER and the current OCTETS, and it seems we should refer to
the BUFFER in our error reporting...
2014-03-04 15:54:31 +01:00
Dimitri Fontaine
654b3f5531
Fix the condition handler fix for #40 .
...
Refrain from trying to display the character where we found a decoding
error when the error actually happens at end-of-input-in-character...
2014-03-04 14:20:27 +01:00
Dimitri Fontaine
56f3da28ed
Fix #20 by skipping table and view missing from the catalogs.
2014-03-04 14:01:04 +01:00
Dimitri Fontaine
4d6def8105
Move some MySQL old import/export functions apart...
2014-03-04 13:52:48 +01:00
Dimitri Fontaine
46fd6632f2
Fix #40 by providing a per-table forced-encoding option.
...
This patch takes benefits from the recent patch
62fc85a1cf
2014-03-03 23:39:22 +01:00
Dimitri Fontaine
1461cda1c0
Improve MySQL encoding errors handling.
...
When it's not possible to decode a MySQL value in the proper given
encoding, automatically replace the value with nil and be quite verbose
about it by logging an error.
2014-03-02 22:44:06 +01:00
Dimitri Fontaine
42635c70bd
Refrain from controling the encoding in pgloader, qmynd now handles it.
2014-03-02 01:27:02 +01:00
Dimitri Fontaine
db947e1467
Rework reader and writer data exchange.
...
With this patch, the whole data massaging and final formating into the
PostgreSQL COPY TEXT format is done by the reader thread, which publishes a
batch at a time in the communication channel: a lparallel.queue object.
Before that, the raw vectors where pushed directly in the queue, offering
more flexibility to adjust to the reader and writer IO rates and
capabilities, but impeding the ability of the Garbage Collector: data still
in the queue was not collected even if not needed anymore.
The new model also uses less memory, and allows a better control over what
amount of data stays in memory. The new *concurrent-batches* parameter
should be key to being able to process huge rows.
The intent is to offering a way for the users to tune *concurrent-batches*
down to 1 for sources with massive per-row memory footprint. Even better
would be to find a way to automatically adjust the setting without spending
too much time counting the bytes we're batching.
Preliminary tests show no sensible impact on performances from this patch,
even some improvements in cases.
2014-01-25 23:54:49 +01:00
Dimitri Fontaine
c56bbab0c4
Fix #24 by allowing cast rules adding only transformation functions.
2014-01-20 16:00:09 +01:00
Dimitri Fontaine
8dda84b0e8
Some more refactoring, MySQL copy-database function is still too big.
2013-12-20 17:37:25 +01:00
Dimitri Fontaine
fe302af221
Refactor the dbname API to feed from the connection string directly.
2013-12-20 17:24:02 +01:00
Dimitri Fontaine
4748104a38
Refactor the too big copy-database function for MySQL.
2013-12-20 15:06:40 +01:00
Dimitri Fontaine
aa934f1a5c
Review, fix some comments and indenting.
2013-12-20 11:08:26 +01:00
Dimitri Fontaine
49f2567b0d
Improve stats on meta-data and before sections.
2013-12-20 10:50:27 +01:00
Dimitri Fontaine
bb561a0c4d
Fetch all MySQL meta-data from a single connection.
2013-12-20 10:21:31 +01:00
Dimitri Fontaine
3a5d618cc8
Switch to using vectors for representing rows, minimize consing.
2013-12-18 23:30:11 +01:00
Dimitri Fontaine
fe2fb517ac
Typo fix.
2013-12-18 18:26:31 +01:00
Gleb Golubitsky
50e8fca96b
Fixing "quote identifiers" option for MySQL conversions
...
Signed-off-by: Gleb Golubitsky <sectoid@gnolltech.org>
2013-12-18 15:06:22 +02:00
Dimitri Fontaine
1ef024c5c7
Fix handling of reserved table names in SELECT and COPY commands.
2013-12-18 12:06:41 +01:00
Dimitri Fontaine
477a52a6d3
Thanks to Qmynd, no need for the IS NULL query dance anymore.
2013-12-09 11:23:00 +01:00
Dimitri Fontaine
f02eb641b4
Switch from cl-mysql to qmynd, an all-lisp driver for MySQL.
2013-12-03 22:05:39 +01:00
Dimitri Fontaine
718ac80560
Implement a Materiaze Views option for MySQL.
2013-11-16 21:41:07 +01:00
Dimitri Fontaine
6fb3de7043
Improve error management in MySQL copy-database preparatory code.
2013-11-13 22:46:17 +01:00
Dimitri Fontaine
2a65a1d39f
Fix setting of PostgreSQL GUCs and transaction usage for DDL.
2013-11-12 11:34:31 +01:00
Dimitri Fontaine
8c9af591e3
Cleanup where to add the table oids for MySQL indexes migration.
2013-11-08 21:09:50 +01:00
Dimitri Fontaine
2e3edde3ad
Implement a generic API to deal with indexes, use it in MySQL and SQLite sources.
2013-11-07 22:07:00 +01:00
Dimitri Fontaine
2fb176f879
Fix new debug message.
2013-11-07 20:51:21 +01:00
Dimitri Fontaine
a227943012
Fix the logging system, we now have a proper logfile.
2013-11-07 20:46:47 +01:00
Dimitri Fontaine
cd3c9b42ce
Some attemps at catching spurious MySQL Errors.
...
MySQL error: "Lost connection to MySQL server during query" (errno = 2013).
2013-11-07 17:46:28 +01:00
Dimitri Fontaine
885a6da5a0
Fix a bug in MySQL command execution when no BEFORE LOAD DO section is used.
2013-11-07 15:48:45 +01:00
Dimitri Fontaine
fe75f3df54
Improve the set of options for loading from databases, assorted fixes.
2013-11-05 23:07:03 +01:00
Dimitri Fontaine
a9dd0aafa3
Fix ALTER TABLE ... DROP CONSTRAINT not to happen against non existing tables or constraints.
2013-11-05 21:24:32 +01:00
Dimitri Fontaine
450c06b6fb
Add a "with foreign keys" option to the MySQL Command.
2013-11-05 19:16:12 +01:00
Dimitri Fontaine
e53e613a82
Implement support for MySQL Foreign Keys.
2013-11-05 18:48:54 +01:00
Dimitri Fontaine
6593995832
Add support for BEFORE LOAD DO and FINALLY DO blocks in the MySQL command.
2013-11-05 17:07:13 +01:00
Dimitri Fontaine
2477b68e15
Implement filtering against the list of tables to migrate, in MySQL.
2013-11-05 14:48:05 +01:00
Dimitri Fontaine
fa70dbd978
Fix filtered loading of MySQL databases the same way as SQLite databases.
2013-11-04 23:24:43 +01:00
Dimitri Fontaine
6a75187b7d
Refactor MySQL to use the new API.
2013-11-04 19:16:08 +01:00
Dimitri Fontaine
0a38195853
Refactoring the API with a real definition of it, and reorg the source tree.
2013-11-04 13:21:45 +01:00
