diff --git a/README.md b/README.md index a6b8c21..b2af55d 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,6 @@ [![Build Status](https://travis-ci.org/dimitri/pgloader.svg?branch=master)](https://travis-ci.org/dimitri/pgloader) [![Join the chat at https://gitter.im/dimitri/pgloader](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/dimitri/pgloader?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) -[![Docker Build Status](https://img.shields.io/docker/cloud/build/dimitri/pgloader.svg)](https://cloud.docker.com/repository/docker/dimitri/pgloader) [![Read The Docs Status](https://readthedocs.org/projects/pgloader/badge/?version=latest&style=plastic)](http://pgloader.readthedocs.io/en/latest/) pgloader is a data loading tool for PostgreSQL, using the `COPY` command. @@ -21,147 +20,43 @@ being the transformation of MySQL datestamps `0000-00-00` and `0000-00-00 00:00:00` to PostgreSQL `NULL` value (because our calendar never had a *year zero*). -## Versioning +## Documentation -pgloader version 1.x is quite old and was developed in `TCL`. -When faced with maintaining that code, the new emerging development -team (hi!) picked `python` instead because that made sense at the -time. So pgloader version 2.x was written in python. +Full documentation is available online, including manual pages of all the +pgcopydb sub-commands. Check out +[https://pgcopydb.readthedocs.io/](https://pgcopydb.readthedocs.io/en/latest/). -The current version of pgloader is the 3.x series, which is written in -[Common Lisp](http://cliki.net/) for better development flexibility, -runtime performance, and support of real threading. - -The versioning is now following the Emacs model, where any X.0 release -number means you're using a development version (alpha, beta, or release -candidate). The next stable versions are going to be `3.1` then `3.2` etc. - -When using a development snapshot rather than a released version the version -number includes the git hash (in its abbreviated form): - - - `pgloader version "3.0.99"` - - Release candidate 9 for pgloader version 3.1, with a *git tag* named - `v3.0.99` so that it's easy to checkout the same sources as the - released code. - - - `pgloader version "3.0.fecae2c"` - - Development snapshot again *git hash* `fecae2c`. It's possible to have - the same sources on another setup with using the git command `git - checkout fecae2c`. - - - `pgloader version "3.1.0"` - - Stable release. - -## LICENCE - -pgloader is available under [The PostgreSQL Licence](http://www.postgresql.org/about/licence/). - -## INSTALL - -You can install pgloader directly from -[apt.postgresql.org](https://wiki.postgresql.org/wiki/Apt) and from official -debian repositories, see -[packages.debian.org/pgloader](https://packages.debian.org/search?keywords=pgloader). - - $ apt-get install pgloader - -You can also use a **docker** image for pgloader at -: - - $ docker pull dimitri/pgloader - $ docker run --rm --name pgloader dimitri/pgloader:latest pgloader --version - $ docker run --rm --name pgloader dimitri/pgloader:latest pgloader --help - -## Build from sources - -pgloader is now a Common Lisp program, tested using the -[SBCL](http://sbcl.org/) (>= 1.2.5) and -[Clozure CL](http://ccl.clozure.com/) implementations with -[Quicklisp](http://www.quicklisp.org/beta/). - -When building from sources, you should always build from the current git -`HEAD` as it's basically the only source that is managed in a way to ensure -it builds aginst current set of dependencies versions. - -### Building from sources on debian - - $ apt-get install sbcl unzip libsqlite3-dev make curl gawk freetds-dev libzip-dev - $ cd /path/to/pgloader - $ make pgloader - $ ./build/bin/pgloader --help - -### Building from sources on RedHat/CentOS - -See "Redhat / CentOS" in [INSTALL.md](INSTALL.md#redhat--centos) - -### Building from sources on macOS - -When using [brew](https://brew.sh), it should be a simple `brew install ---HEAD pgloader`. - -When using [macports](https://www.macports.org), then we have a situation to -deal with with shared objects pgloader depends on, as reported in issue #161 -at : - -> I was able to get a clean build without having to disable compression after -> symlinking /usr/local/lib to /opt/local/lib. Note that I did not have -> anything installed to /usr/local/lib so I didn't lose anything here. - -### Building from sources on Windows - -Building pgloader on Windows is supported, thanks to Common Lisp -implementations being available on that platform, and to the Common Lisp -Standard for making it easy to write actually portable code. - -It is recommended to have a look at the issues labelled with *Windows -support* if you run into trouble when building -pgloader: - - - -### Building Docker image from sources - -You can build a Docker image from source using SBCL by default: - - $ docker build . - -Or Clozure CL (CCL): - - $ docker build -f Dockerfile.ccl . - -## More options when building from source - -The `Makefile` target `pgloader` knows how to produce a Self Contained -Binary file for pgloader, found at `./build/bin/pgloader`: - - $ make pgloader - -By default, the `Makefile` uses [SBCL](http://sbcl.org/) to compile your -binary image, though it's possible to build using -[CCL](http://ccl.clozure.com/). - - $ make CL=ccl pgloader - -If using `SBCL` and it supports core compression, the make process will -use it to generate a smaller binary. To force disabling core -compression, you may use: - - $ make COMPRESS_CORE=no pgloader - -The `--compress-core` is unique to SBCL, so not used when `CC` is different -from the `sbcl` value. - -You can also tweak the default amount of memory that the `pgloader` image -will allow itself using when running through your data (don't ask for more -than your current RAM tho): - - $ make DYNSIZE=8192 pgloader - -The `make pgloader` command when successful outputs a `./build/bin/pgloader` -file for you to use. +``` +$ pgloader --help +pgloader [ option ... ] SOURCE TARGET + --help -h boolean Show usage and exit. + --version -V boolean Displays pgloader version and exit. + --quiet -q boolean Be quiet + --verbose -v boolean Be verbose + --debug -d boolean Display debug level information. + --client-min-messages string Filter logs seen at the console (default: "warning") + --log-min-messages string Filter logs seen in the logfile (default: "notice") + --summary -S string Filename where to copy the summary + --root-dir -D string Output root directory. (default: #P"/tmp/pgloader/") + --upgrade-config -U boolean Output the command(s) corresponding to .conf file for v2.x + --list-encodings -E boolean List pgloader known encodings and exit. + --logfile -L string Filename where to send the logs. + --load-lisp-file -l string Read user code from files + --dry-run boolean Only check database connections, don't load anything. + --on-error-stop boolean Refrain from handling errors properly. + --no-ssl-cert-verification boolean Instruct OpenSSL to bypass verifying certificates. + --context -C string Command Context Variables + --with string Load options + --set string PostgreSQL options + --field string Source file fields specification + --cast string Specific cast rules + --type string Force input source type + --encoding string Source expected encoding + --before string SQL script to run before loading the data + --after string SQL script to run after loading the data + --self-upgrade string Path to pgloader newer sources + --regress boolean Drive regression testing +``` ## Usage @@ -183,3 +78,13 @@ indexes, foreign keys, comments) and parallel loading of the corrected data: $ createdb pagila $ pgloader mysql://user@localhost/sakila postgresql:///pagila + +## LICENCE + +pgloader is available under [The PostgreSQL +Licence](http://www.postgresql.org/about/licence/). + +## INSTALL + +Please see full documentation at +[https://pgloader.readthedocs.io/](https://pgloader.readthedocs.io/en/latest/install.html). diff --git a/docs/conf.py b/docs/conf.py index feacb41..b2cac9f 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -55,9 +55,9 @@ author = 'Dimitri Fontaine' # built documents. # # The short X.Y version. -version = '3.4' +version = '3.6' # The full version, including alpha/beta/rc tags. -release = '3.4.1' +release = '3.6.3' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/docs/index.rst b/docs/index.rst index 569a86e..c8a26a2 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -244,6 +244,7 @@ implement the third step in a fully automated way. That's pgloader. intro quickstart tutorial/tutorial + install pgloader ref/csv ref/fixed diff --git a/docs/install.rst b/docs/install.rst new file mode 100644 index 0000000..b59b211 --- /dev/null +++ b/docs/install.rst @@ -0,0 +1,213 @@ +Installing pgloader +=================== + +Several distributions are available for pgcopydb. + +debian packages +--------------- + +You can install pgloader directly from `apt.postgresql.org`__ and from +official debian repositories, see `packages.debian.org/pgloader`__. + +:: + + $ apt-get install pgloader + +__ https://wiki.postgresql.org/wiki/Apt +__ https://packages.debian.org/search?keywords=pgloader + +Docker Images +------------- + +Docker images are maintained for each tagged release at dockerhub, and also +built from the CI/CD integration on GitHub at each commit to the `main` +branch. + +The DockerHub `dimitri/pgloader`__ repository is where the tagged releases +are made available. The image uses the Postgres version currently in debian +stable. + +__ https://hub.docker.com/r/dimitri/pgloader + +To use the ``dimitri/pgloader`` docker image:: + + $ docker run --rm -it dimitri/pgloader:latest pgloader --version + +Or you can use the CI/CD integration that publishes packages from the main +branch to the GitHub docker repository:: + + $ docker pull ghcr.io/dimitri/pgloader:latest + $ docker run --rm -it ghcr.io/dimitri/pgloader:latest pgloader --version + $ docker run --rm -it ghcr.io/dimitri/pgloader:latest pgloader --help + +Build from sources +------------------ + +pgloader is a Common Lisp program, tested using the `SBCL`__ (>= 1.2.5) and +`Clozure CL`__ implementations and with `Quicklisp`__ to fetch build +dependencies. + +__ http://sbcl.org/ +__ http://ccl.clozure.com/ +__ http://www.quicklisp.org/beta/ + +When building from sources, you should always build from the current git +HEAD as it's basically the only source that is managed in a way to ensure it +builds aginst current set of dependencies versions. + +The build system for pgloader uses a Makefile and the Quicklisp Common Lisp +packages distribution system. + +The modern build system for pgloader is entirely written in Common Lisp, +where the historical name for our operation is `save-lisp-and-die` and can +be used that way: + +:: + + $ make save + +The legacy build system also uses Buildapp and can be used that way: + +:: + + $ make pgloader + +Building from sources on debian +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Install the build dependencies first, then use the Makefile:: + + $ apt-get install sbcl unzip libsqlite3-dev make curl gawk freetds-dev libzip-dev + $ cd /path/to/pgloader + + $ make save + $ ./build/bin/pgloader --help + +Building from sources on RedHat/CentOS +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To build and install pgloader the Steel Bank Common Lisp package (sbcl) from +EPEL, and the freetds packages are required. + +It is recommended to build the RPM yourself, see below, to ensure that all +installed files are properly tracked and that you can safely update to newer +versions of pgloader as they're released. + +To do an adhoc build and install run ``boostrap-centos.sh`` for CentOS 6 or +``bootstrap-centos7.sh`` for CentOS 7 to install the required dependencies. + +Building a pgloader RPM from sources +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The spec file in the root of the pgloader repository can be used to build your +own RPM. For production deployments it is recommended that you build this RPM on +a dedicated build box and then copy the RPM to your production environment for +use; it is considered bad practice to have compilers and build tools present in +production environments. + +1. Install the [EPEL repo](https://fedoraproject.org/wiki/EPEL#Quickstart). + +2. Install rpmbuild dependencies:: + + sudo yum -y install yum-utils rpmdevtools @"Development Tools" + +3. Install pgloader build dependencies:: + + sudo yum-builddep pgloader.spec + +4. Download pgloader source:: + + spectool -g -R pgloader.spec + +5. Build the source and binary RPMs (see `rpmbuild --help` for other build + options):: + + rpmbuild -ba pgloader.spec + +Building from sources on macOS +------------------------------ + +We suppose you already have ``git`` and ``make`` available, if that's not +the case now is the time to install those tools. The SQLite lib that comes +in MacOSX is fine, no need for extra software here. + +You will need to install either SBCL or CCL separately, and when using +[brew](http://brew.sh/) it's as simple as: + +:: + + $ brew install sbcl + $ brew install clozure-cl + +NOTE: Make sure you installed the universal binaries of Freetds, so that +they can be loaded correctly. + +:: + + $ brew install freetds --universal --build-from-source + +Then use the normal build system for pgloader: + +:: + + $ make save + $ ./build/bin/pgloader --version + +Building from sources on Windows +-------------------------------- + +Building pgloader on Windows is supported (in theory), thanks to Common Lisp +implementations being available on that platform, and to the Common Lisp +Standard for making it easy to write actually portable code. + +It is recommended to have a look at the `issues labelled with Windows +support`__ if you run into trouble when building pgloader, because the +development team is lacking windows user and in practice we can't maintain +the support for that Operating System: + +__ https://github.com/dimitri/pgloader/issues?utf8=✓&q=label%3A%22Windows%20support%22%20> + +If you need ``pgloader.exe`` on windows please condider contributing fixes +for that environment and maybe longer term support then. Specifically, a CI +integration with a windows build host would allow ensuring that we continue +to support that target. + +Building Docker image from sources +---------------------------------- + +You can build a Docker image from source using SBCL by default:: + + $ docker build . + +Or Clozure CL (CCL):: + + $ docker build -f Dockerfile.ccl . + +More options when building from source +-------------------------------------- + +The ``Makefile`` target ``save`` knows how to produce a Self Contained +Binary file for pgloader, found at ``./build/bin/pgloader``:: + + $ make save + +By default, the ``Makefile`` uses `SBCL`__ to compile your binary image, +though it's possible to build using `Clozure-CL`__. + +__ http://sbcl.org/ +__ http://ccl.clozure.com/ + +:: + + $ make CL=ccl64 save + +It is possible to to tweak the default amount of memory that the pgloader +image will allow itself using when running through your data (don't ask for +more than your current RAM tho). At the moment only the legacy build system +includes support for this custom build:: + + $ make DYNSIZE=8192 pgloader + +The ``make pgloader`` command when successful outputs a +`./build/bin/pgloader` file for you to use. +