mirrors/pgloader
1
0
Fork 0
mirror of https://github.com/dimitri/pgloader.git synced 2026-01-25 09:01:07 +01:00
Code Issues Projects Releases Wiki Activity
383 Commits 7 Branches 39 Tags
Commit Graph

136 Commits

Author SHA1 Message Date
Dimitri Fontaine
ef358c0b7d Take benefits of PostgreSQL COPY error CONTEXT. 
This message has the line number where the erroneous data was found on the
server, and given the pre-processing we already done at that point, it's
easy to convert that number into an index into the current batch, an array.

To do do, we need Postmodern to expose the CONTEXT error message and we need
to parse it. The following pull request cares about the Postmodern side of
things:

  https://github.com/marijnh/Postmodern/pull/46

The parsing is done as simply as possible, only assuming that the error
message is using comma separators and having the line number in second
position. The parsing as done here should still work with localized message
strings.

  CONTEXT: COPY errors, line 3, column b: "2006-13-11"

This change should significantly reduce the cost of error processing.
 2013-12-25 21:43:22 +01:00
Dimitri Fontaine
36bfd923fa Fix monitor logging with no argument is given to the format string. 2013-12-25 21:25:20 +01:00
Dimitri Fontaine
cd1d5ad597 Fix missing *pg-dbname* in CSV processing, resulting in broken logging. 2013-12-25 16:50:14 +01:00
Dimitri Fontaine
7d94d4ff62 Small code cleanup. 2013-12-25 16:20:37 +01:00
Dimitri Fontaine
f2bec5fcd1 Fix the main command line "driver" to use the new with-monitor API. 2013-12-24 19:54:22 +01:00
Dimitri Fontaine
d6edc54272 Ensure ordering of sections (table-names) in the summary. 2013-12-24 19:31:27 +01:00
Dimitri Fontaine
b2c9e0d2dc Refactor the whole logging infrastructure not to depend on threads sharing streams. 2013-12-24 19:08:55 +01:00
Dimitri Fontaine
e31c5cbe83 Set the default batch size (in rows) back to 25000. 2013-12-23 13:17:04 +01:00
Dimitri Fontaine
8bf4bffb2c Fix SQLite stats, wasn't counting read rows. 2013-12-23 13:16:04 +01:00
Dimitri Fontaine
8dda84b0e8 Some more refactoring, MySQL copy-database function is still too big. 2013-12-20 17:37:25 +01:00
Dimitri Fontaine
fe302af221 Refactor the dbname API to feed from the connection string directly. 2013-12-20 17:24:02 +01:00
Dimitri Fontaine
5a464a0fae Remove some dead code (was intended for later use). 2013-12-20 17:07:43 +01:00
Dimitri Fontaine
4748104a38 Refactor the too big copy-database function for MySQL. 2013-12-20 15:06:40 +01:00
Dimitri Fontaine
aa934f1a5c Review, fix some comments and indenting. 2013-12-20 11:08:26 +01:00
Dimitri Fontaine
49f2567b0d Improve stats on meta-data and before sections. 2013-12-20 10:50:27 +01:00
Dimitri Fontaine
ce14542a15 Respect the Materialize Views ordering given in the command when loading data. 2013-12-20 10:35:00 +01:00
Dimitri Fontaine
bb561a0c4d Fetch all MySQL meta-data from a single connection. 2013-12-20 10:21:31 +01:00
Dimitri Fontaine
e09eb3fbb2 There is always a need for a before state when migrating from MySQL... 2013-12-20 10:19:36 +01:00
Dimitri Fontaine
aca04b1514 Fix problem found when trying to load the code with CCL. 2013-12-19 23:08:02 +01:00
Dimitri Fontaine
3a5d618cc8 Switch to using vectors for representing rows, minimize consing. 2013-12-18 23:30:11 +01:00
Dimitri Fontaine
0a6608061c Due to the bug fixed and the new batch implementation, version is now 3.0.95. 2013-12-18 18:56:12 +01:00
Dimitri Fontaine
82c4bc9e9e Switch the pgsql batch implementation to using arrays to reduce consing. 2013-12-18 18:31:06 +01:00
Dimitri Fontaine
fe2fb517ac Typo fix. 2013-12-18 18:26:31 +01:00
Gleb Golubitsky
50e8fca96b Fixing "quote identifiers" option for MySQL conversions 
Signed-off-by: Gleb Golubitsky <sectoid@gnolltech.org>
 2013-12-18 15:06:22 +02:00
Dimitri Fontaine
1ef024c5c7 Fix handling of reserved table names in SELECT and COPY commands. 2013-12-18 12:06:41 +01:00
Dimitri Fontaine
633b6ae64f Export with-mysql-connection facility for REPL testing. 2013-12-18 12:05:56 +01:00
Dimitri Fontaine
f58b5960cd Prepare a debian package, and make it pgloader 3.0.94. 2013-12-09 13:04:29 +01:00
Dimitri Fontaine
477a52a6d3 Thanks to Qmynd, no need for the IS NULL query dance anymore. 2013-12-09 11:23:00 +01:00
Dimitri Fontaine
f02eb641b4 Switch from cl-mysql to qmynd, an all-lisp driver for MySQL. 2013-12-03 22:05:39 +01:00
Dimitri Fontaine
157587476b Call it version 3.0.93 now that we did fix many bugs. 2013-11-26 16:59:44 +01:00
Dimitri Fontaine
2019b918f0 Implement support for matching several files in a single archive clause. 2013-11-26 16:47:37 +01:00
Dimitri Fontaine
98a7eb12a5 Error out when asked to load a non-existing file. 2013-11-26 11:56:34 +01:00
Dimitri Fontaine
de1afc7416 Fix CSV and FIXED parsing to run the BEFORE section before doing PostgreSQL introspection. 2013-11-26 11:51:42 +01:00
Dimitri Fontaine
32d91d7054 Return a non-zero error code to the OS when something unexpected did happen. 2013-11-25 11:25:57 +01:00
Dimitri Fontaine
e4b50d2315 An archive always need a before state to be able to report about its extract step. 2013-11-24 17:09:56 +01:00
Dimitri Fontaine
451861cfd5 Don't query PostgreSQL for the OIDs of an empty list of tables. 2013-11-24 16:43:59 +01:00
Dimitri Fontaine
ea4e74e545 Improve parsing of list separating commas wrt to whitespacing. 2013-11-23 21:49:08 +01:00
Dimitri Fontaine
c482015248 Arrange to error out when given non-existing file names in the command line. 2013-11-23 21:36:22 +01:00
Dimitri Fontaine
4bc4cc53a7 Add support for connecting to PostgreSQL with Unix Domain Sockets, fixing #15. 2013-11-22 17:19:48 +01:00
Dimitri Fontaine
fdfcb8dde9 Fix the byte-vector-to-bytea transform function (handle nil, better type declaration). 2013-11-22 17:18:09 +01:00
Dimitri Fontaine
2a344ab7ce Add a usage line in the --help output. 2013-11-22 11:23:15 +01:00
Dimitri Fontaine
78b42cc4ae Properly handle blob formats, converting then to PostgreSQL bytea input. 2013-11-22 11:01:20 +01:00
Dimitri Fontaine
74cabeb6f5 Change MySQL list-tables function not to include views. 2013-11-22 10:55:38 +01:00
Dimitri Fontaine
6a6684bd8b Desultory improve the SQLite importer. 2013-11-21 21:34:02 +01:00
Dimitri Fontaine
1419a1f65d Default --logfile to pgloader.log within --root-dir. 2013-11-21 14:28:11 +01:00
Dimitri Fontaine
516bb28ec0 Fix using TMPDIR from the environment when running from the binary image. 2013-11-21 13:06:16 +01:00
Dimitri Fontaine
d70ed07b12 If --lofgile is relative, expand it into --root-dir. 2013-11-21 12:56:54 +01:00
Dimitri Fontaine
febbc03459 Review some log messages and levels. 2013-11-21 11:37:47 +01:00
Dimitri Fontaine
d52240a95e Review the --root-dir patches for better default management. 
Also ensure the directory we're given actually exists on disk, creating it
if necessary, and bail out early in case for whatever reason it's not
possible to create the directory.
 2013-11-21 11:27:54 +01:00
Cédric Villemain
17ef2f1933 A word was missing in comment 2013-11-21 11:00:02 +01:00