mirrors/pgloader
1
0
Fork 0
mirror of https://github.com/dimitri/pgloader.git synced 2026-01-21 15:11:01 +01:00
Code Issues Projects Releases Wiki Activity
152 Commits 7 Branches 39 Tags
Commit Graph

152 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Dimitri Fontaine
c603221ae0 repamming? 2012-08-31 09:57:15 +02:00
Dimitri Fontaine
4f102d69c7 Add a README with some details. 2012-08-31 09:55:43 +02:00
Guillaume Lelarge
8c03a124aa Add an example of the truncate option. 2011-02-25 11:51:24 +01:00
Guillaume Lelarge
220f055d12 Add a truncate option for sections. 2011-02-24 10:15:15 +01:00
Dimitri Fontaine
723baaefa0 Fix lintian warnings, and update copyright to match PostgreSQL licence text 2010-11-09 16:13:15 +01:00
Dimitri Fontaine
1711ef97ff prepare for 2.3.3~dev3 release, some testers need that 2010-11-09 14:39:37 +01:00
Dimitri Fontaine
d81e23210e Review the dsn patch, little typo fix 2010-11-09 14:32:17 +01:00
Dimitri Fontaine
f97c09cc3b Merge remote branch 'refs/remotes/dpanech/dsn' 2010-11-09 14:25:20 +01:00
Dimitri Fontaine
553f3e80c9 Merge remote branch 'refs/remotes/dpanech/sysexit' 2010-11-09 14:18:39 +01:00
Dimitri Fontaine
76f8dd0686 Merge sigint from dpanech 2010-11-09 14:15:55 +01:00
Dimitri Fontaine
ef38322c97 Merge remote branch 'refs/remotes/dpanech/tempdir' 2010-11-09 14:07:33 +01:00
Dimitri Fontaine
6eaea7193b Merge remote branch 'refs/remotes/dpanech/exit_status' 2010-11-09 14:04:28 +01:00
Dimitri Fontaine
b8160e821a Add an example with null = 2010-11-09 12:31:26 +01:00
Davlet Panech
a206b18f8b better import placement 2010-09-27 11:32:12 -04:00
Davlet Panech
b78da36220 use os.path.join for constructing file names 2010-09-27 11:18:20 -04:00
Davlet Panech
f1bcdbee3a use gettempdir() rather than hard-coded "/tmp" 2010-09-21 01:52:00 -04:00
Davlet Panech
bc4ff93d8f fixed formatting 2010-09-20 17:51:42 -04:00
Davlet Panech
d07bddc153 Disable KeyboardInterrupt, just exit on SIGINT. 
The KeyboardInterrupt exception doesn't get raised if the script
is stuck in a blocking call, i.e., inside the check_event function.
So pressing CTRL+C has no effect.

This patch disables the exception, reverting to the OS' default
behaviour for the SIGINT signal (i.e., terminate the program).
 2010-09-20 17:44:07 -04:00
Davlet Panech
82b2667ef7 Handle SystemExit separately from Exception (python <= 2.4) 2010-09-20 17:15:43 -04:00
Davlet Panech
441ce481a1 Make all postgres connection parameters optional 2010-09-20 17:09:55 -04:00
Davlet Panech
256d1f6daa updated manual page 2010-09-20 15:26:02 -04:00
Davlet Panech
9393180c0c Better error handling in loader threads. 
The main script currently hangs if any loader fails to connect to
the database. This happens because connection exceptions from
PGLoader._postinit are not handled, and the main script keeps
waiting for "finished" events that never happen.

This patch ensures that exceptions in the loader threads are noted
by the main script, and that an appropriate exit status is returned
to the OS.

Changes in pgloader/pgloader.py:
 - Handle all exceptions in PGLoader.run
 - Add a boolean "success" flag to PGLoader class that determines
   whether any errors occured within that object's main thread
 - Whenever an exception happens within a thread, set the "success"
   flag to false and call "terminate"

Changes in pgloader.py:
 - check "success" flag on all threads after they finish
 - return 0 on success and 1 on any error
 - the script no longer returns the result of "print_summary",
   just 0 or 1.
 2010-09-20 15:07:31 -04:00
Dimitri Fontaine
b323952e63 Fix work in progress line encoding in reader.py to not touch read data if self.encoding is None 2010-08-25 18:00:54 +02:00
Dimitri Fontaine
d9c923590e Fix reader.py logging when using -C 2010-08-25 18:00:21 +02:00
Dimitri Fontaine
4e5fec95e3 Fix connection management bug when columns = *. 
Instead of opening a connection per section and leaving it open until the
end for the run, open a connection just for getting this section columns
then close it. Per gripes from a user dealing with 90000 sections.
 2010-04-29 17:14:44 +02:00
Dimitri Fontaine
a062db184d Avoid calling self.fd.tell() when we're reading from stdin. 
That means the options --load-from-stdin and --boundaries are not compatible.
 2010-04-29 13:20:44 +02:00
Dimitri Fontaine
f154d9bf83 Implement --boundaries to load a part of a file only (given in bytes). 2010-04-15 16:59:06 +02:00
Dimitri Fontaine
1738aaa6aa Add forgotten exemple, fixed input cols. 2010-04-15 16:55:37 +02:00
Dimitri Fontaine
cbd167c0e8 Implement --load-from-stdin 2010-04-15 16:55:10 +02:00
Dimitri Fontaine
38c8a079f2 Bugfix: MAX_PARALLEL_SECTIONS should not be None, even if not given on command line. 2010-04-07 14:11:10 +02:00
Dimitri Fontaine
efa4b6b103 Package version 2.3.3~dev1 for some testing. Won't get released yet. 2010-04-06 16:45:46 +02:00
Dimitri Fontaine
1b6c0f4735 Implement support for the CVS field_size_limit, new in python 2.5. 2010-04-06 16:35:39 +02:00
Dimitri Fontaine
411a9a53d7 Add support for PG options as arguments and in the config file, with assorted fixes 2010-04-05 21:17:22 +02:00
Dimitri Fontaine
d63b21c3f9 Update the debian changelog for 2.3.3 2010-04-05 01:07:40 +02:00
Dimitri Fontaine
201209259a Implement --reject-log and --reject-data options. 2010-04-05 01:02:30 +02:00
Dimitri Fontaine
ac0a16f9b2 Add support for --max-parallel-sections and --section-threads from command line. 2010-04-05 00:32:46 +02:00
Dimitri Fontaine
b7b0bbc62d Support for processing a filename rather than a section, and add --field-separator 2010-04-04 20:42:40 +02:00
Dimitri Fontaine
99df251963 Adding missed file, parallel data exemple 2010-04-04 19:32:48 +02:00
dim
8af4355994 release 2.3.2 2009-06-29 10:30:46 +00:00
dim
43c95d67c2 python-support is needed for pgloader to run, not only to build 2009-04-28 13:56:37 +00:00
dim
090de905b2 Implement skip_head_lines in configuration file, some more bugfixes 2009-03-10 17:56:16 +00:00
dim
848595f49b Some more python2.3 support changes 2008-09-18 12:58:12 +00:00
dim
2bacf1a854 FIX RRR bug wrt offsets, allow python 2.3 to run pgloader when it does not need collections.deque (no Round Robin Reader), and prepare packaging for 2.3.2 2008-09-17 16:55:30 +00:00
dim
ea03303e27 Use psycopg cursor.copy_expert() when avaiable (> 2.0.6), which should fix problems tied to too small copy buffer, per Marko's comment on Skytools project 2008-09-05 12:52:18 +00:00
dim
5f335eda28 FIX fixed reader: it now know about -C 2008-09-05 12:50:43 +00:00
dim
094313d7ff Preliminary support for real line numbers usage in logs, ok for simple cases --- no split_file_reading, multi-line input broken 2008-08-07 22:49:22 +00:00
dim
0e42971ea0 some more facilities in the TODO list 2008-07-27 18:00:43 +00:00
dim
e9fdd8e0b7 FIXES: add -F and -C supports to CSV format, cvs doublequotes defaults to True, per Andrew Dunstan gripes 2008-07-24 16:22:52 +00:00
dim
23b19d381e debian packaging bug#485067: Missing Build-Depends 2008-06-09 13:25:34 +00:00
dim
85caa89c03 Consider XML support as a TODO item 2008-06-09 13:21:23 +00:00