mirrors/pgloader
1
0
Fork 0
mirror of https://github.com/dimitri/pgloader.git synced 2025-08-08 07:16:58 +02:00
Code Issues Projects Releases Wiki Activity
1,568 Commits 7 Branches 39 Tags 23 MiB
fix/order-mysql-tables-by-decreasing-row-count
Commit Graph

245 Commits

Author SHA1 Message Date
Dimitri Fontaine
80b6c46aae Version 3.0.97. 2014-01-15 22:53:43 +01:00
Dimitri Fontaine
07c614c170 Switch to the newer cl-csv API. 
Thanks to the work at https://github.com/AccelerationNet/cl-csv/pull/12 we
can now use the main branch of cl-csv again.
 2014-01-11 18:28:24 +01:00
Dimitri Fontaine
ef358c0b7d Take benefits of PostgreSQL COPY error CONTEXT. 
This message has the line number where the erroneous data was found on the
server, and given the pre-processing we already done at that point, it's
easy to convert that number into an index into the current batch, an array.

To do do, we need Postmodern to expose the CONTEXT error message and we need
to parse it. The following pull request cares about the Postmodern side of
things:

  https://github.com/marijnh/Postmodern/pull/46

The parsing is done as simply as possible, only assuming that the error
message is using comma separators and having the line number in second
position. The parsing as done here should still work with localized message
strings.

  CONTEXT: COPY errors, line 3, column b: "2006-13-11"

This change should significantly reduce the cost of error processing.
 2013-12-25 21:43:22 +01:00
Dimitri Fontaine
43fc540660 Pick a CL encoding name found both in SBCL and CCL for the census test. 2013-12-24 19:37:15 +01:00
Dimitri Fontaine
b84541367d Add MySQL (MariaDB) support in tests. 2013-12-03 22:06:10 +01:00
Dimitri Fontaine
3ddcc959ac Improve tests and add test cases. 2013-11-26 16:48:45 +01:00
Dimitri Fontaine
ea4e74e545 Improve parsing of list separating commas wrt to whitespacing. 2013-11-23 21:49:08 +01:00
Dimitri Fontaine
4bc4cc53a7 Add support for connecting to PostgreSQL with Unix Domain Sockets, fixing #15. 2013-11-22 17:19:48 +01:00
Dimitri Fontaine
6a6684bd8b Desultory improve the SQLite importer. 2013-11-21 21:34:02 +01:00
Dimitri Fontaine
941747e36a Cleanup in the census places test. 2013-11-21 10:54:57 +01:00
Dimitri Fontaine
ee857e7523 While at it, provide a proper right-trim transforms function. 2013-11-18 11:55:26 +01:00
Dimitri Fontaine
8d8e0b041a Allow filename matching expressions in the FIXED command, and support it in archives. 2013-11-18 11:38:50 +01:00
Dimitri Fontaine
fb54d19357 Review some default casting rules, in particular add tinyint(1) to boolean. 2013-11-18 10:28:38 +01:00
Dimitri Fontaine
b31ccded6f Implement per-column MySQL CAST rules. 2013-11-18 10:12:43 +01:00
Dimitri Fontaine
8fa7a86013 Implement typemod and default guards in the MySQL CAST clause. 2013-11-17 22:33:06 +01:00
Dimitri Fontaine
718ac80560 Implement a Materiaze Views option for MySQL. 2013-11-16 21:41:07 +01:00
Dimitri Fontaine
d99b859c3f Improve and fix COPY error handling, transactions, connections, and GUCs. 2013-11-13 23:54:41 +01:00
Dimitri Fontaine
00d3886ba6 Fix the goeuro local only test case. 2013-11-13 22:35:45 +01:00
Dimitri Fontaine
9d5dad7e3e Implement support for FIXED COLS input files, reaching release candidate status. 2013-11-07 15:39:28 +01:00
Dimitri Fontaine
fe75f3df54 Improve the set of options for loading from databases, assorted fixes. 2013-11-05 23:07:03 +01:00
Dimitri Fontaine
450c06b6fb Add a "with foreign keys" option to the MySQL Command. 2013-11-05 19:16:12 +01:00
Dimitri Fontaine
6593995832 Add support for BEFORE LOAD DO and FINALLY DO blocks in the MySQL command. 2013-11-05 17:07:13 +01:00
Dimitri Fontaine
2477b68e15 Implement filtering against the list of tables to migrate, in MySQL. 2013-11-05 14:48:05 +01:00
Dimitri Fontaine
ff9d0fd6f5 Fix function exporting bug, and index counts that caused infinite waiting. 2013-11-04 23:13:46 +01:00
Dimitri Fontaine
24f47640e2 Add grammar support for loading a SQLite database. 2013-11-04 21:54:51 +01:00
Dimitri Fontaine
561eb29175 Add a test case for the sakila database. 2013-11-04 21:24:23 +01:00
Dimitri Fontaine
63005168fd MySQL SET datatype support: convert to ENUM Arrays. 2013-11-01 11:34:43 +00:00
Dimitri Fontaine
2f6c7677fc Fix "reset sequences" option to only target just loaded tables. 2013-10-21 23:15:24 +02:00
Dimitri Fontaine
9fd0bbabe2 Yet another round of fixes for the test setup. 
Including some Makefile hacks where test doesn't depend on the main pgloader
binary anymore because I coulnd't stop the binary to get being built again
even if it's been done already...
 2013-10-20 23:27:11 +02:00
Dimitri Fontaine
208d10c919 Review and clean the vagrant setup, add tests. 2013-10-20 22:51:06 +02:00
Dimitri Fontaine
f619054c79 Improve database connection parsing to use sensible defaults. 2013-10-19 22:39:09 +02:00
Dimitri Fontaine
f768f74cbb Add support for comments (-- and /* */ style) in the command language. 2013-10-18 11:19:24 +02:00
Dimitri Fontaine
e1d2bd1318 Improve LOAD DBF command (support for http and zip). 2013-10-13 22:48:24 +02:00
Dimitri Fontaine
89aaabd179 Port tests from pgloader 2.x, implement trailing-sep. 2013-10-13 22:10:07 +02:00
Dimitri Fontaine
7d1b34477e Implement CSV option keep|trim unquoted blanks. 2013-10-12 14:49:52 +02:00
Dimitri Fontaine
4ff7d9ae69 Improve documentation and empty strings test cases, now that cl-csv is fixed. 2013-10-12 14:36:35 +02:00
Dimitri Fontaine
1407d379f1 Add support for the SET (gucs) option in the CSV commands. 2013-10-11 13:50:23 +02:00
Dimitri Fontaine
b762905953 Use our HTTP available very small test case for the archive tests. 2013-10-11 12:07:55 +02:00
Dimitri Fontaine
1cf15b0f6f Add a NULL test case and fix some areas of the code to make it pass. 2013-10-11 11:20:35 +02:00
Dimitri Fontaine
3221ce6cc3 Add support for loading INLINE data in CSV commands. 2013-10-11 01:00:52 +02:00
Dimitri Fontaine
f22eeec123 Allow LOAD CSV command to accept BEFORE LOAD DO and AFTER LOAD DO sections. 2013-10-10 23:37:49 +02:00
Dimitri Fontaine
65035b1bc6 Allow for loading CSV files not using any field quoting. 2013-10-08 22:16:52 +02:00
Dimitri Fontaine
bab53d1cd0 Improve testing, load commands from files. 2013-10-08 21:48:43 +02:00
Dimitri Fontaine
165f74b197 Add some test files. 2013-10-04 17:12:50 +02:00
Dimitri Fontaine
0af0ac315c Remove useless block of example code. 2013-08-21 17:26:36 +02:00