Commit Graph

790 Commits

Author SHA1 Message Date
Dimitri Fontaine
f5f584fdf1 Fix parsing ccl:describe-character-encodings.
First, despite the documentation mentionning the function writes
to *terminal-io*, in fact it's doing (format t ...) and thus the result
is written to *standard-output*.

Second, CCL has encodings with no aliases.
2014-04-29 14:25:40 +02:00
Dimitri Fontaine
0c371797dc Publish version 3.0.99 on pgloader.io website. 2014-04-29 14:04:40 +02:00
Dimitri Fontaine
a5a29407f0 Release pgloader version 3.0.99. 2014-04-29 13:59:33 +02:00
Dimitri Fontaine
24281fb679 Include the new version of the manpage, pre-compiled. 2014-04-29 11:54:37 +02:00
Dimitri Fontaine
943373d8e3 No patches needed anymore to compile pgloader. 2014-04-29 11:48:09 +02:00
Dimitri Fontaine
c0d9bb4d8f Allows to build pgloader image using CCL.
Too many Makefile commands where hard-coded using SBCL, which prevented
from building successfully against CCL. That's now fixed.
2014-04-29 11:47:22 +02:00
Dimitri Fontaine
40128dbd75 Fix with-monitor support of :start-logger option.
It used to still launch an extra set of threads for monitoring where,
and that would confuse CCL where it's not possible to write into a
stream from more than one thread concurrently.
2014-04-29 11:43:03 +02:00
Dimitri Fontaine
0f62751a3f Improve summary output.
Try at having a deterministic ouput of it, which still apparently is not
always the case when using SBCL, now that it's been switched to using
the explicit *terminal-io* rather than t.

This change is needed for CCL support, though, where you don't get to
write to the same stream from different threads.
2014-04-29 11:42:02 +02:00
Dimitri Fontaine
3abcfeb569 Avoid empty index definitions in SQLite, fixes #52.
I could get down to the problem here, which is that a couple of indexes
where reported to pgloader but without any SQL definition for them, and
then pgloader would wait for non existing tasks.

It seems easier to just skip does indexes, that's what this patch does.
2014-04-28 16:00:34 +02:00
Dimitri Fontaine
9516a90d9d Fix SQLite support for filename parsing.
The code didn't get the memo about the way we now do support source
filenames and all.
2014-04-28 15:20:30 +02:00
Dimitri Fontaine
b758058208 Fix the fix for parsing quoted-filenames. 2014-04-28 15:18:18 +02:00
Dimitri Fontaine
b5c89e750c Quick review of the generic API documentation strings. 2014-04-28 14:36:15 +02:00
Dimitri Fontaine
429232c3de Fix loading data from stdin: fix #53.
The stdin support really was one brick shy of a load, and in particular
with-open-file was used against a stream when using that option.
2014-04-27 23:38:02 +02:00
Dimitri Fontaine
b5dec87915 Allow any non-quote characters in a quoted filename.
In particular, allow for a space to be used in the filename. The only
character that is not permitted anymore is the quote itself ('), it
should be easy enough to allow for escaping it as in the password field
if required.

Should probably fix #54, even though the lack of data currently reported
in that issue makes it a blind guess only.
2014-04-27 22:49:27 +02:00
Dimitri Fontaine
efd11ab759 Add user options to control pgloader batch behaviour.
The new WITH options allows the user to set values for the dynamic
variables *copy-batch-rows*, *copy-batch-size* and *concurrent-batches*.
That's needed in case like in issue #16 even with the batch size
defaulting to what looks like a proper setup.

In a longer term a review of the pgloader memory usage should be done
seriously, the numbers being way higher than the batch sizes we do setup
here.
2014-04-27 22:37:17 +02:00
Dimitri Fontaine
78a988eb47 Oops, forgot to add the new file charsets.lisp. 2014-04-26 18:55:43 +02:00
Dimitri Fontaine
35ca4927e9 Get rid of some lib dependencies.
The charset business isn't worth depending on an AGPL licenced lib which
is part of a huge Quicklisp system.
2014-04-25 17:21:11 +02:00
Dimitri Fontaine
cb20cb7145 The make clean target needs more work. 2014-04-25 17:17:19 +02:00
Dimitri Fontaine
789d854799 Fix issue #49 where data could be considered as a format string. 2014-04-23 17:03:35 +02:00
Dimitri Fontaine
a2109a0cd5 Add build/bin in the git repository, we need that dir now. 2014-04-23 16:13:24 +02:00
Dimitri Fontaine
77f87fb8b3 Copy/past'o fix. 2014-04-22 17:21:44 +02:00
Dimitri Fontaine
71fcd6b748 Add some INSTALL notes, with dependency instructions. 2014-04-22 17:16:51 +02:00
Dimitri Fontaine
4ccf5c5ad0 Build the buildapp binary in build/bin. 2014-04-22 17:14:00 +02:00
Dimitri Fontaine
3a9bc9db0f Switch the default memory watch to on. 2014-04-22 17:13:36 +02:00
Dimitri Fontaine
9fa638e233 Handle NIL values in transform functions.
When declaring types of arguments (mainly done for hinting the Common
Lisp compiler into generating more efficient code), it's important to
account for the possibility of the arguments being NIL, of NULL type.

That's been made clear in the way the projection function is now
generated in src/sources/source.lisp in project-fields function, with
all the arguments now being &optional so that we are able to cope with
ragged CSV files.

The only expected change from this patch is missing warnings in some
test cases, such as test/reformat.load, test/fixed.load and
test/archive.load.
2014-04-18 22:51:30 +02:00
Dimitri Fontaine
1af517323c Attempt to fix the OpenSSL loading situation.
For the generated binary to be really portable, we need to be able to
open openssl 1.0.1 even when we've been built against openssl 1.0.0.

A way to achieve that with SBCL is by forcing the unloading of the lib
at image saving time and register a hook to load it again at image init
time. Using the proper API, CFFI will happily load the available file
for the lib rather than insisting on loading the exact same one than
found on the build machine.
2014-04-18 22:24:11 +02:00
Dimitri Fontaine
a39de0012e Simplify using the Makefile for non-CL-hackers.
Try harder not to mix Common Lisp Developers profiles with main pgloader
expected audience who will not have a Quicklisp system ready, by doing
our magic Quicklisp calls in a pgloader specific directory.

That might impact #45 badly as the homebrew formula should need to now
install its Quicklisp distributions elsewhere, but it should be easy
enough to fix.

Also, systematically `git pull` from the local-projects dependencies we
need to work with to avoid some bad surprises to our users.
2014-04-14 15:36:19 +02:00
Dimitri Fontaine
b1333c936c Merge pull request #45 from benesch/homebrew
Homebrew support
2014-04-07 12:29:38 +02:00
Nikhil Benesch
73299237fb Makefile: add standalone target that doesn't install quicklisp 2014-04-06 19:59:34 -04:00
Nikhil Benesch
7ec2ca30e8 pgloader.1: include compiled man page for packaging convenience
Allows packaging systems like Homebrew to install manual pages
automatically for users without introducing a dependency on the manual
page build system.
2014-04-06 19:59:31 -04:00
Nikhil Benesch
56fd8e3368 Makefile: use ronn to generate docs 2014-04-06 19:59:02 -04:00
Nikhil Benesch
f6b63ade0a pgloader.1.md: remove trailing whitespace 2014-04-06 19:59:02 -04:00
Nikhil Benesch
80f28f2477 pgloader.1.md: update formatting for ronn compatibility
Ronn is an alternative to pandoc for generating Groff man pages from
Markdown, and seems to produce better-looking man pages.
2014-04-06 19:58:59 -04:00
Dimitri Fontaine
e7d828e696 Get rid of special versions of postmodern and cl-csv, fix #44.
The patches we made against those systems made it to Quicklisp. Next on
the list is Qmynd, which should be ok to go at the next Quicklisp release.
2014-03-31 18:07:55 -04:00
Dimitri Fontaine
7ed5d4e777 Document --debug --verbose (client-min-messages data).
That should help fixing #48 where this level of information is required.
2014-03-28 09:39:38 +01:00
Dimitri Fontaine
9e3c6130ff Implement Google Analytics on pgloader.tapoueh.org 2014-03-25 21:19:00 +01:00
Dimitri Fontaine
114d2fedbc Another try at fixing #40.
The babel character-decoding-error condition is exposing both its
internal BUFFER and the current OCTETS, and it seems we should refer to
the BUFFER in our error reporting...
2014-03-04 15:54:31 +01:00
Dimitri Fontaine
654b3f5531 Fix the condition handler fix for #40.
Refrain from trying to display the character where we found a decoding
error when the error actually happens at end-of-input-in-character...
2014-03-04 14:20:27 +01:00
Dimitri Fontaine
56f3da28ed Fix #20 by skipping table and view missing from the catalogs. 2014-03-04 14:01:04 +01:00
Dimitri Fontaine
4d6def8105 Move some MySQL old import/export functions apart... 2014-03-04 13:52:48 +01:00
Dimitri Fontaine
46fd6632f2 Fix #40 by providing a per-table forced-encoding option.
This patch takes benefits from the recent patch
62fc85a1cf
so that you will need to freshen your local Qmynd copy if you want to
test from sources.
2014-03-03 23:39:22 +01:00
Dimitri Fontaine
1461cda1c0 Improve MySQL encoding errors handling.
When it's not possible to decode a MySQL value in the proper given
encoding, automatically replace the value with nil and be quite verbose
about it by logging an error.
2014-03-02 22:44:06 +01:00
Dimitri Fontaine
b033aed88b Some more testing. 2014-03-02 01:27:02 +01:00
Dimitri Fontaine
7befe27807 Add encoding errors to the csv-error.load test. 2014-03-02 01:27:02 +01:00
Dimitri Fontaine
42635c70bd Refrain from controling the encoding in pgloader, qmynd now handles it. 2014-03-02 01:27:02 +01:00
Dimitri Fontaine
f92817191e Merge pull request #41 from JimPanic/patch-2
Updated shebang line
2014-02-24 17:27:11 +01:00
Dimitri Fontaine
7fa95c1135 Fix bug #39 wherein unix domain sockets didn't make it properly to cl-postgres. 2014-02-24 17:23:17 +01:00
Alexander Pánek
cc2c21ae42 Updated shebang line
Not everyone has SBCL installed in /usr/bin/sbcl and sometimes one wants to test other versions of SBCL or just not install the official Debian/Ubuntu packages. I have tested this on Ubuntu precise and it worked for me.

I got the info this would work from here:
https://bugs.launchpad.net/sbcl/+bug/1284148
2014-02-24 16:34:19 +01:00
Dimitri Fontaine
14a588cb9f Merge pull request #36 from chanmix51/master
Updated README with build informations
2014-02-17 16:41:23 +01:00
Dimitri Fontaine
241e6e4da1 Merge pull request #37 from JimPanic/patch-1
Removed tinyint cast rule
2014-02-17 16:36:49 +01:00