mirrors/pgloader
1
0
Fork 0
mirror of https://github.com/dimitri/pgloader.git synced 2025-08-08 07:16:58 +02:00
Code Issues Projects Releases Wiki Activity
137 Commits 7 Branches 39 Tags 23 MiB
ef38322c97
Commit Graph

46 Commits

Author SHA1 Message Date
Dimitri Fontaine
ef38322c97 Merge remote branch 'refs/remotes/dpanech/tempdir' 2010-11-09 14:07:33 +01:00
Davlet Panech
b78da36220 use os.path.join for constructing file names 2010-09-27 11:18:20 -04:00
Davlet Panech
f1bcdbee3a use gettempdir() rather than hard-coded "/tmp" 2010-09-21 01:52:00 -04:00
Davlet Panech
9393180c0c Better error handling in loader threads. 
The main script currently hangs if any loader fails to connect to
the database. This happens because connection exceptions from
PGLoader._postinit are not handled, and the main script keeps
waiting for "finished" events that never happen.

This patch ensures that exceptions in the loader threads are noted
by the main script, and that an appropriate exit status is returned
to the OS.

Changes in pgloader/pgloader.py:
 - Handle all exceptions in PGLoader.run
 - Add a boolean "success" flag to PGLoader class that determines
   whether any errors occured within that object's main thread
 - Whenever an exception happens within a thread, set the "success"
   flag to false and call "terminate"

Changes in pgloader.py:
 - check "success" flag on all threads after they finish
 - return 0 on success and 1 on any error
 - the script no longer returns the result of "print_summary",
   just 0 or 1.
 2010-09-20 15:07:31 -04:00
Dimitri Fontaine
a062db184d Avoid calling self.fd.tell() when we're reading from stdin. 
That means the options --load-from-stdin and --boundaries are not compatible.
 2010-04-29 13:20:44 +02:00
Dimitri Fontaine
cbd167c0e8 Implement --load-from-stdin 2010-04-15 16:55:10 +02:00
Dimitri Fontaine
38c8a079f2 Bugfix: MAX_PARALLEL_SECTIONS should not be None, even if not given on command line. 2010-04-07 14:11:10 +02:00
Dimitri Fontaine
411a9a53d7 Add support for PG options as arguments and in the config file, with assorted fixes 2010-04-05 21:17:22 +02:00
Dimitri Fontaine
201209259a Implement --reject-log and --reject-data options. 2010-04-05 01:02:30 +02:00
Dimitri Fontaine
ac0a16f9b2 Add support for --max-parallel-sections and --section-threads from command line. 2010-04-05 00:32:46 +02:00
Dimitri Fontaine
b7b0bbc62d Support for processing a filename rather than a section, and add --field-separator 2010-04-04 20:42:40 +02:00
dim
090de905b2 Implement skip_head_lines in configuration file, some more bugfixes 2009-03-10 17:56:16 +00:00
dim
848595f49b Some more python2.3 support changes 2008-09-18 12:58:12 +00:00
dim
5a63f4f609 Protect the database connection establishing into the parralel sections semaphore, allow field_sep to be configured to \t, and some fixes 2008-05-29 09:16:13 +00:00
dim
b7c7e6a7c1 Support for fixed format datafile 2008-05-21 10:33:06 +00:00
dim
e15ac609a8 Allow user to force psycopg version to use, as psycopg2 is not always the best choice 2008-02-27 12:30:13 +00:00
dim
45176adec7 Don't wait for non started threads (PGLoader init could error out) 2008-02-25 14:11:33 +00:00
dim
a27a801f62 Synchronize threads starts and ends with an Event 2008-02-15 21:14:45 +00:00
dim
192da034c3 Handle more nicely connection errors - from python stacktrace to ERROR message 2008-02-15 09:53:06 +00:00
dim
a905ac7521 FIX check_path calls, from user bug report 2008-02-14 20:15:30 +00:00
dim
a068a5ea6a FIX just-introduced bug: no reader init meant self.db.copy_sep = None 2008-02-14 14:57:47 +00:00
dim
e641b18fb3 Support for 'rrqueue_size' per-section configuration parameter, defaults to copy_every 2008-02-13 09:42:26 +00:00
dim
a2195ec2e7 Better handling of split_file_reading case, Still In Progress 2008-02-12 12:24:15 +00:00
dim
b534962192 Using semaphores to control how many max threads run at any given time 2008-02-12 11:27:18 +00:00
dim
fc8adf1831 First implementation of parallel loading, each thread loading a part of the input file (stat, seek, etc) 2008-02-11 17:38:52 +00:00
dim
9e17976ef8 First attempt at a multi-threaded pgloader, seems to be ok though not configurable yet 2008-02-05 13:41:45 +00:00
dim
526836deca database object is now initialized from PGLoader init 2008-02-05 11:45:30 +00:00
dim
60ffb4ba3f pgloader no longer has to run VACUUM after having loaded all the tables 2008-02-05 11:20:07 +00:00
dim
8605b14084 * pgloader -V now VACUUM each table separately, no more vacuumdb issued 
* New option -D to DISABLE Triggers while loading (ENABLE them one done)
 2008-02-01 10:24:39 +00:00
dim
70b3def565 New logging mecanism init fix, when nothing is configured 2007-12-10 10:35:59 +00:00
dim
a530dedd66 * Rework DatabaseError and consider dichotomy in this case too. 
* New options lc_messages and log_file
 2007-12-07 21:13:35 +00:00
dim
efecde9f5b new logging options documentation 2007-12-07 14:37:28 +00:00
dim
5da357c262 Beginning of real logging implementation for pgloader messages 2007-12-04 23:08:30 +00:00
dim
d9f328ce4d debian packaging 2007-11-29 21:30:25 +00:00
dim
0426dcb4f0 Fix default reformat_path checking; Fix template configuration usage 
with user-defined columns (defining copy_columns from template).
 2007-11-27 21:52:57 +00:00
dim
0c1ad56106 FIX: reformat_path could be not set after configuration reading 2007-11-27 14:28:17 +00:00
dim
1e7a16b869 Configuration now knows about templates, and command line about a reformat 
path option, --reformat_path= or -R.
 2007-11-26 21:33:24 +00:00
dim
3a8ac261c8 Support for template sections, WIP 2007-11-26 17:36:49 +00:00
dim
86b33defef DEBUG and documentation of reformat option, plus other fixes for having all 
the test cases pass again.
 2007-11-20 15:11:49 +00:00
dim
e9b884cb4e Global input_encoding configuration, per-section datestyle configuration, fixes. 2007-11-19 16:44:20 +00:00
dim
5cbd3531aa Adding the 'input_encoding' option in the configuration file, per 
section.
 2007-11-18 23:04:48 +00:00
dim
dbaeef0ad2 pgloader version 2.2.3, with User Defined Columns 
* User Defined Columns
  * Temporary files with copy data content now suffixed .pgloader
  * New option --version
  * Fix TextReader newline_escapes configuration option reading
  * Fix Reader reject initialisation
  * Skip database related settings when in DRY_RUN mode (-n)
  * List all command line options from man page synopsis
 2007-11-14 21:09:36 +00:00
dim
57ea0f443e Improve summary output 2007-11-05 14:17:01 +00:00
dim
d32472bb3e Add support for --quiet and --summary options, in order to provide some 
way to only see errors on output.
 2007-10-20 14:28:50 +00:00
dim
921db51d65 * Support for datestyle setting 
* Support for omiting column numbering
  * Change documentation source format from SGML to asciidoc
  * New version 2.2.1
 2007-08-23 10:38:09 +00:00
dim
8ed1e0ff2c New version of pgloader (2.0.2): 
* new developpers
* python code replaces tcl code
* one config file for many tables (one command line for loading them all)
* manpage
* debian package
* supports multi-line input file (without quotes)
* Informix large objects support (loading to TEXT or BYTEA)
* configurable amount of rows per COPY instruction
* trailing slash optionnal removal (support informix UNLOAD file format)
* begin processing at any line in the file, by number or row id
* dry-run option, to validate input reading without connecting to database
* pedantic option, to stop processing on warning
 2006-11-19 21:18:42 +00:00