mirrors/pgloader
1
0
Fork 0
mirror of https://github.com/dimitri/pgloader.git synced 2025-08-10 00:07:00 +02:00
Code Issues Projects Releases Wiki Activity
790 Commits 7 Branches 39 Tags 23 MiB
087d4d28cb
Commit Graph

112 Commits

Author SHA1 Message Date
Dimitri Fontaine
efd11ab759 Add user options to control pgloader batch behaviour. 
The new WITH options allows the user to set values for the dynamic
variables *copy-batch-rows*, *copy-batch-size* and *concurrent-batches*.
That's needed in case like in issue #16 even with the batch size
defaulting to what looks like a proper setup.

In a longer term a review of the pgloader memory usage should be done
seriously, the numbers being way higher than the batch sizes we do setup
here.
 2014-04-27 22:37:17 +02:00
Dimitri Fontaine
789d854799 Fix issue #49 where data could be considered as a format string. 2014-04-23 17:03:35 +02:00
Dimitri Fontaine
56f3da28ed Fix #20 by skipping table and view missing from the catalogs. 2014-03-04 14:01:04 +01:00
Dimitri Fontaine
46fd6632f2 Fix #40 by providing a per-table forced-encoding option. 
This patch takes benefits from the recent patch
62fc85a1cf
so that you will need to freshen your local Qmynd copy if you want to
test from sources.
 2014-03-03 23:39:22 +01:00
Dimitri Fontaine
b033aed88b Some more testing. 2014-03-02 01:27:02 +01:00
Dimitri Fontaine
7befe27807 Add encoding errors to the csv-error.load test. 2014-03-02 01:27:02 +01:00
Alexander Pánek
97109153f4 Removed tinyint cast rule 
This rule has overridden the default rule for `tinyint(1)` and instead of placing `boolean`, it kept the typemod and placed `boolean(1)` into the resulting query.
 2014-02-17 16:14:27 +01:00
Dimitri Fontaine
643875a266 Improve CSV error handling, thanks to cl-csv continue restart. 2014-02-08 17:51:15 +01:00
Dimitri Fontaine
dbfd8cf06c Implement new CSV option "lines terminated by", fixes #23. 2014-02-04 20:58:46 +01:00
Dimitri Fontaine
a8b0f91f37 Allow optional control of batch memory footprint, see #16 and #22. 
With the new internal setting *copy-batch-size* it's now possible to
instruct pgloader to close batches early (before *copy-batch-rows* limit)
when crossing the byte count threshold.

When set to 20 MB it allows the new test case (exhausted) to pass under SBCL
and CCL, and there's no measurable cost when *copy-batch-size* is set to
nil (its default value) in the testing done.

This patch is published without any way to tune the values from the command
language yet, that's the next step once its been proven effective.
 2014-01-26 23:22:18 +01:00
Dimitri Fontaine
8ac2cc4930 Skip empty lines when reading from files. 2014-01-24 15:11:15 +01:00
Dimitri Fontaine
3f61c66a79 Also handle extra columns in CSV parsing. 2014-01-23 15:15:42 +01:00
Dimitri Fontaine
516ef08c37 Allow loading ragged CSV files. 2014-01-23 15:07:05 +01:00
Dimitri Fontaine
7c238f45f2 Fix batch retry handling, broken in previous refactoring. Fixes #22. 2014-01-22 22:52:18 +01:00
Dimitri Fontaine
a5c661dd4a Cleanup error recovery logging. 2014-01-22 17:57:56 +01:00
Dimitri Fontaine
c13d7bbae7 Bug fix when processing plain filenames. 2014-01-22 11:00:12 +01:00
Dimitri Fontaine
c56bbab0c4 Fix #24 by allowing cast rules adding only transformation functions. 2014-01-20 16:00:09 +01:00
Dimitri Fontaine
80b6c46aae Version 3.0.97. 2014-01-15 22:53:43 +01:00
Dimitri Fontaine
07c614c170 Switch to the newer cl-csv API. 
Thanks to the work at https://github.com/AccelerationNet/cl-csv/pull/12 we
can now use the main branch of cl-csv again.
 2014-01-11 18:28:24 +01:00
Dimitri Fontaine
ef358c0b7d Take benefits of PostgreSQL COPY error CONTEXT. 
This message has the line number where the erroneous data was found on the
server, and given the pre-processing we already done at that point, it's
easy to convert that number into an index into the current batch, an array.

To do do, we need Postmodern to expose the CONTEXT error message and we need
to parse it. The following pull request cares about the Postmodern side of
things:

  https://github.com/marijnh/Postmodern/pull/46

The parsing is done as simply as possible, only assuming that the error
message is using comma separators and having the line number in second
position. The parsing as done here should still work with localized message
strings.

  CONTEXT: COPY errors, line 3, column b: "2006-13-11"

This change should significantly reduce the cost of error processing.
 2013-12-25 21:43:22 +01:00
Dimitri Fontaine
43fc540660 Pick a CL encoding name found both in SBCL and CCL for the census test. 2013-12-24 19:37:15 +01:00
Dimitri Fontaine
b84541367d Add MySQL (MariaDB) support in tests. 2013-12-03 22:06:10 +01:00
Dimitri Fontaine
3ddcc959ac Improve tests and add test cases. 2013-11-26 16:48:45 +01:00
Dimitri Fontaine
ea4e74e545 Improve parsing of list separating commas wrt to whitespacing. 2013-11-23 21:49:08 +01:00
Dimitri Fontaine
4bc4cc53a7 Add support for connecting to PostgreSQL with Unix Domain Sockets, fixing #15. 2013-11-22 17:19:48 +01:00
Dimitri Fontaine
6a6684bd8b Desultory improve the SQLite importer. 2013-11-21 21:34:02 +01:00
Dimitri Fontaine
941747e36a Cleanup in the census places test. 2013-11-21 10:54:57 +01:00
Dimitri Fontaine
ee857e7523 While at it, provide a proper right-trim transforms function. 2013-11-18 11:55:26 +01:00
Dimitri Fontaine
8d8e0b041a Allow filename matching expressions in the FIXED command, and support it in archives. 2013-11-18 11:38:50 +01:00
Dimitri Fontaine
fb54d19357 Review some default casting rules, in particular add tinyint(1) to boolean. 2013-11-18 10:28:38 +01:00
Dimitri Fontaine
b31ccded6f Implement per-column MySQL CAST rules. 2013-11-18 10:12:43 +01:00
Dimitri Fontaine
8fa7a86013 Implement typemod and default guards in the MySQL CAST clause. 2013-11-17 22:33:06 +01:00
Dimitri Fontaine
718ac80560 Implement a Materiaze Views option for MySQL. 2013-11-16 21:41:07 +01:00
Dimitri Fontaine
d99b859c3f Improve and fix COPY error handling, transactions, connections, and GUCs. 2013-11-13 23:54:41 +01:00
Dimitri Fontaine
00d3886ba6 Fix the goeuro local only test case. 2013-11-13 22:35:45 +01:00
Dimitri Fontaine
9d5dad7e3e Implement support for FIXED COLS input files, reaching release candidate status. 2013-11-07 15:39:28 +01:00
Dimitri Fontaine
fe75f3df54 Improve the set of options for loading from databases, assorted fixes. 2013-11-05 23:07:03 +01:00
Dimitri Fontaine
450c06b6fb Add a "with foreign keys" option to the MySQL Command. 2013-11-05 19:16:12 +01:00
Dimitri Fontaine
6593995832 Add support for BEFORE LOAD DO and FINALLY DO blocks in the MySQL command. 2013-11-05 17:07:13 +01:00
Dimitri Fontaine
2477b68e15 Implement filtering against the list of tables to migrate, in MySQL. 2013-11-05 14:48:05 +01:00
Dimitri Fontaine
ff9d0fd6f5 Fix function exporting bug, and index counts that caused infinite waiting. 2013-11-04 23:13:46 +01:00
Dimitri Fontaine
24f47640e2 Add grammar support for loading a SQLite database. 2013-11-04 21:54:51 +01:00
Dimitri Fontaine
561eb29175 Add a test case for the sakila database. 2013-11-04 21:24:23 +01:00
Dimitri Fontaine
63005168fd MySQL SET datatype support: convert to ENUM Arrays. 2013-11-01 11:34:43 +00:00
Dimitri Fontaine
2f6c7677fc Fix "reset sequences" option to only target just loaded tables. 2013-10-21 23:15:24 +02:00
Dimitri Fontaine
9fd0bbabe2 Yet another round of fixes for the test setup. 
Including some Makefile hacks where test doesn't depend on the main pgloader
binary anymore because I coulnd't stop the binary to get being built again
even if it's been done already...
 2013-10-20 23:27:11 +02:00
Dimitri Fontaine
208d10c919 Review and clean the vagrant setup, add tests. 2013-10-20 22:51:06 +02:00
Dimitri Fontaine
f619054c79 Improve database connection parsing to use sensible defaults. 2013-10-19 22:39:09 +02:00
Dimitri Fontaine
f768f74cbb Add support for comments (-- and /* */ style) in the command language. 2013-10-18 11:19:24 +02:00
Dimitri Fontaine
e1d2bd1318 Improve LOAD DBF command (support for http and zip). 2013-10-13 22:48:24 +02:00
Dimitri Fontaine
89aaabd179 Port tests from pgloader 2.x, implement trailing-sep. 2013-10-13 22:10:07 +02:00
Dimitri Fontaine
7d1b34477e Implement CSV option keep|trim unquoted blanks. 2013-10-12 14:49:52 +02:00
Dimitri Fontaine
4ff7d9ae69 Improve documentation and empty strings test cases, now that cl-csv is fixed. 2013-10-12 14:36:35 +02:00
Dimitri Fontaine
1407d379f1 Add support for the SET (gucs) option in the CSV commands. 2013-10-11 13:50:23 +02:00
Dimitri Fontaine
b762905953 Use our HTTP available very small test case for the archive tests. 2013-10-11 12:07:55 +02:00
Dimitri Fontaine
1cf15b0f6f Add a NULL test case and fix some areas of the code to make it pass. 2013-10-11 11:20:35 +02:00
Dimitri Fontaine
3221ce6cc3 Add support for loading INLINE data in CSV commands. 2013-10-11 01:00:52 +02:00
Dimitri Fontaine
f22eeec123 Allow LOAD CSV command to accept BEFORE LOAD DO and AFTER LOAD DO sections. 2013-10-10 23:37:49 +02:00
Dimitri Fontaine
65035b1bc6 Allow for loading CSV files not using any field quoting. 2013-10-08 22:16:52 +02:00
Dimitri Fontaine
bab53d1cd0 Improve testing, load commands from files. 2013-10-08 21:48:43 +02:00
Dimitri Fontaine
165f74b197 Add some test files. 2013-10-04 17:12:50 +02:00
Dimitri Fontaine
0af0ac315c Remove useless block of example code. 2013-08-21 17:26:36 +02:00