598c860cf5
To be able to use "t" (or "nil") as a column name, pgloader needs to be able to generate lisp code where those symbols are available. It's simple enough in that a Common Lisp package that doesn't :use :cl fullfills the condition, so intern user symbols in a specially crafted package that doesn't :use :cl. Now, we still need to be able to run transformation code that is using the :cl package symbols and the pgloader.transforms functions too. In this commit we introduce a heuristic to pick symbols either as functions from pgloader.transforms or anything else in pgloader.user-symbols. And so that user code may use NIL too, we provide an override mechanism to the intern-symbol heuristic and use it only when parsing user code, not when producing Common Lisp code from the parsed load command. |
||
|---|---|---|
| build | ||
| conf | ||
| debian | ||
| src | ||
| test | ||
| web | ||
| .gitignore | ||
| .travis.yml | ||
| bootstrap-centos7.sh | ||
| bootstrap-centos.sh | ||
| bootstrap-debian.sh | ||
| INSTALL.md | ||
| Makefile | ||
| pgloader.1 | ||
| pgloader.1.md | ||
| pgloader.asd | ||
| pgloader.lisp | ||
| pgloader.spec | ||
| README.md | ||
| TODO.md | ||
| Vagrantfile | ||
PGLoader
pgloader is a data loading tool for PostgreSQL, using the COPY command.
Its main advantage over just using COPY or \copy, and over using a
Foreign Data Wrapper, is its transaction behaviour, where pgloader
will keep a separate file of rejected data, but continue trying to
copy good data in your database.
The default PostgreSQL behaviour is transactional, which means that any erroneous line in the input data (file or remote database) will stop the entire bulk load for the table.
pgloader also implements data reformatting, a typical example of that
being the transformation of MySQL datestamps 0000-00-00 and
0000-00-00 00:00:00 to PostgreSQL NULL value (because our calendar
never had a year zero).
Versioning
pgloader version 1.x is quite old and was developed in TCL.
When faced with maintaining that code, the new emerging development
team (hi!) picked python instead because that made sense at the
time. So pgloader version 2.x was written in python.
The current version of pgloader is the 3.x series, which is written in Common Lisp for better development flexibility, runtime performance, and support of real threading.
The versioning is now following the Emacs model, where any X.0 release
number means you're using a development version (alpha, beta, or release
candidate). The next stable versions are going to be 3.1 then 3.2 etc.
When using a development snapshot rather than a released version the version number includes the git hash (in its abbreviated form):
-
pgloader version "3.0.99"Release candidate 9 for pgloader version 3.1, with a git tag named
v3.0.99so that it's easy to checkout the same sources as the released code. -
pgloader version "3.0.fecae2c"Development snapshot again git hash
fecae2c. It's possible to have the same sources on another setup with using the git commandgit checkout fecae2c. -
pgloader version "3.1.0"Stable release.
LICENCE
pgloader is available under The PostgreSQL Licence.
INSTALL
pgloader is now a Common Lisp program, tested using the SBCL (>= 1.1.14) and Clozure CL implementations with Quicklisp.
$ apt-get install sbcl unzip libsqlite3-dev make curl gawk freetds-dev libzip-dev
$ cd /path/to/pgloader
$ make pgloader
$ ./build/bin/pgloader --help
When building from sources, you should always build from the current git HEAD as it's basically the only source that is managed in a way to ensure it builds aginst current set of dependencies versions.
The pgloader.lisp script
Now you can use the #! script or build a self-contained binary executable
file, as shown below.
./pgloader.lisp --help
Each time you run the pgloader command line, it will check that all its
dependencies are installed and compiled and if that's not the case fetch
them from the internet and prepare them (thanks to Quicklisp). So please
be patient while that happens and make sure we can actually connect and
download the dependencies.
Build Self-Contained binary file
The Makefile target pgloader knows how to produce a Self Contained
Binary file for pgloader, named pgloader.exe:
$ make pgloader
By default, the Makefile uses SBCL to compile your
binary image, though it's possible to also build using
CCL.
$ make CL=ccl pgloader
Note that the Makefile uses the --compress-core option when using SBCL,
that should be enabled in your local copy of SBCL. If that's not the case,
it's probably because you did compile and install SBCL yourself, so that
you have a decently recent version to use. Then you need to compile it with
the --with-sb-core-compression option.
You can also remove the --compress-core option that way:
$ make COMPRESS_CORE=no pgloader
The --compress-core is unique to SBCL, so not used when CC is different
from the sbcl value.
The make pgloader command when successful outputs a ./build/bin/pgloader
file for you to use.
Usage
Give as many command files that you need to pgloader:
$ ./build/bin/pgloader --help
$ ./build/bin/pgloader <file.load>
See the documentation file pgloader.1.md for details. You can compile that
file into a manual page or an HTML page thanks to the ronn application:
$ apt-get install ruby-ronn
$ make docs