mirror of
https://github.com/dimitri/pgloader.git
synced 2026-05-04 10:31:02 +02:00
Improve README and add proper INSTALL docs. (#1417)
This commit is contained in:
Dimitri Fontaine
GitHub
parent
fac03a68d4
commit
ff33ec5e2e
185
README.md
185
README.md
@ -2,7 +2,6 @@
|
||||
|
||||
[](https://travis-ci.org/dimitri/pgloader)
|
||||
[](https://gitter.im/dimitri/pgloader?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
|
||||
[](https://cloud.docker.com/repository/docker/dimitri/pgloader)
|
||||
[](http://pgloader.readthedocs.io/en/latest/)
|
||||
|
||||
pgloader is a data loading tool for PostgreSQL, using the `COPY` command.
|
||||
@ -21,147 +20,43 @@ being the transformation of MySQL datestamps `0000-00-00` and
|
||||
`0000-00-00 00:00:00` to PostgreSQL `NULL` value (because our calendar
|
||||
never had a *year zero*).
|
||||
|
||||
## Versioning
|
||||
## Documentation
|
||||
|
||||
pgloader version 1.x is quite old and was developed in `TCL`.
|
||||
When faced with maintaining that code, the new emerging development
|
||||
team (hi!) picked `python` instead because that made sense at the
|
||||
time. So pgloader version 2.x was written in python.
|
||||
Full documentation is available online, including manual pages of all the
|
||||
pgcopydb sub-commands. Check out
|
||||
[https://pgcopydb.readthedocs.io/](https://pgcopydb.readthedocs.io/en/latest/).
|
||||
|
||||
The current version of pgloader is the 3.x series, which is written in
|
||||
[Common Lisp](http://cliki.net/) for better development flexibility,
|
||||
runtime performance, and support of real threading.
|
||||
|
||||
The versioning is now following the Emacs model, where any X.0 release
|
||||
number means you're using a development version (alpha, beta, or release
|
||||
candidate). The next stable versions are going to be `3.1` then `3.2` etc.
|
||||
|
||||
When using a development snapshot rather than a released version the version
|
||||
number includes the git hash (in its abbreviated form):
|
||||
|
||||
- `pgloader version "3.0.99"`
|
||||
|
||||
Release candidate 9 for pgloader version 3.1, with a *git tag* named
|
||||
`v3.0.99` so that it's easy to checkout the same sources as the
|
||||
released code.
|
||||
|
||||
- `pgloader version "3.0.fecae2c"`
|
||||
|
||||
Development snapshot again *git hash* `fecae2c`. It's possible to have
|
||||
the same sources on another setup with using the git command `git
|
||||
checkout fecae2c`.
|
||||
|
||||
- `pgloader version "3.1.0"`
|
||||
|
||||
Stable release.
|
||||
|
||||
## LICENCE
|
||||
|
||||
pgloader is available under [The PostgreSQL Licence](http://www.postgresql.org/about/licence/).
|
||||
|
||||
## INSTALL
|
||||
|
||||
You can install pgloader directly from
|
||||
[apt.postgresql.org](https://wiki.postgresql.org/wiki/Apt) and from official
|
||||
debian repositories, see
|
||||
[packages.debian.org/pgloader](https://packages.debian.org/search?keywords=pgloader).
|
||||
|
||||
$ apt-get install pgloader
|
||||
|
||||
You can also use a **docker** image for pgloader at
|
||||
<https://hub.docker.com/r/dimitri/pgloader/>:
|
||||
|
||||
$ docker pull dimitri/pgloader
|
||||
$ docker run --rm --name pgloader dimitri/pgloader:latest pgloader --version
|
||||
$ docker run --rm --name pgloader dimitri/pgloader:latest pgloader --help
|
||||
|
||||
## Build from sources
|
||||
|
||||
pgloader is now a Common Lisp program, tested using the
|
||||
[SBCL](http://sbcl.org/) (>= 1.2.5) and
|
||||
[Clozure CL](http://ccl.clozure.com/) implementations with
|
||||
[Quicklisp](http://www.quicklisp.org/beta/).
|
||||
|
||||
When building from sources, you should always build from the current git
|
||||
`HEAD` as it's basically the only source that is managed in a way to ensure
|
||||
it builds aginst current set of dependencies versions.
|
||||
|
||||
### Building from sources on debian
|
||||
|
||||
$ apt-get install sbcl unzip libsqlite3-dev make curl gawk freetds-dev libzip-dev
|
||||
$ cd /path/to/pgloader
|
||||
$ make pgloader
|
||||
$ ./build/bin/pgloader --help
|
||||
|
||||
### Building from sources on RedHat/CentOS
|
||||
|
||||
See "Redhat / CentOS" in [INSTALL.md](INSTALL.md#redhat--centos)
|
||||
|
||||
### Building from sources on macOS
|
||||
|
||||
When using [brew](https://brew.sh), it should be a simple `brew install
|
||||
--HEAD pgloader`.
|
||||
|
||||
When using [macports](https://www.macports.org), then we have a situation to
|
||||
deal with with shared objects pgloader depends on, as reported in issue #161
|
||||
at <https://github.com/dimitri/pgloader/issues/161#issuecomment-201162647>:
|
||||
|
||||
> I was able to get a clean build without having to disable compression after
|
||||
> symlinking /usr/local/lib to /opt/local/lib. Note that I did not have
|
||||
> anything installed to /usr/local/lib so I didn't lose anything here.
|
||||
|
||||
### Building from sources on Windows
|
||||
|
||||
Building pgloader on Windows is supported, thanks to Common Lisp
|
||||
implementations being available on that platform, and to the Common Lisp
|
||||
Standard for making it easy to write actually portable code.
|
||||
|
||||
It is recommended to have a look at the issues labelled with *Windows
|
||||
support* if you run into trouble when building
|
||||
pgloader:
|
||||
|
||||
<https://github.com/dimitri/pgloader/issues?utf8=✓&q=label%3A%22Windows%20support%22%20>
|
||||
|
||||
### Building Docker image from sources
|
||||
|
||||
You can build a Docker image from source using SBCL by default:
|
||||
|
||||
$ docker build .
|
||||
|
||||
Or Clozure CL (CCL):
|
||||
|
||||
$ docker build -f Dockerfile.ccl .
|
||||
|
||||
## More options when building from source
|
||||
|
||||
The `Makefile` target `pgloader` knows how to produce a Self Contained
|
||||
Binary file for pgloader, found at `./build/bin/pgloader`:
|
||||
|
||||
$ make pgloader
|
||||
|
||||
By default, the `Makefile` uses [SBCL](http://sbcl.org/) to compile your
|
||||
binary image, though it's possible to build using
|
||||
[CCL](http://ccl.clozure.com/).
|
||||
|
||||
$ make CL=ccl pgloader
|
||||
|
||||
If using `SBCL` and it supports core compression, the make process will
|
||||
use it to generate a smaller binary. To force disabling core
|
||||
compression, you may use:
|
||||
|
||||
$ make COMPRESS_CORE=no pgloader
|
||||
|
||||
The `--compress-core` is unique to SBCL, so not used when `CC` is different
|
||||
from the `sbcl` value.
|
||||
|
||||
You can also tweak the default amount of memory that the `pgloader` image
|
||||
will allow itself using when running through your data (don't ask for more
|
||||
than your current RAM tho):
|
||||
|
||||
$ make DYNSIZE=8192 pgloader
|
||||
|
||||
The `make pgloader` command when successful outputs a `./build/bin/pgloader`
|
||||
file for you to use.
|
||||
```
|
||||
$ pgloader --help
|
||||
pgloader [ option ... ] SOURCE TARGET
|
||||
--help -h boolean Show usage and exit.
|
||||
--version -V boolean Displays pgloader version and exit.
|
||||
--quiet -q boolean Be quiet
|
||||
--verbose -v boolean Be verbose
|
||||
--debug -d boolean Display debug level information.
|
||||
--client-min-messages string Filter logs seen at the console (default: "warning")
|
||||
--log-min-messages string Filter logs seen in the logfile (default: "notice")
|
||||
--summary -S string Filename where to copy the summary
|
||||
--root-dir -D string Output root directory. (default: #P"/tmp/pgloader/")
|
||||
--upgrade-config -U boolean Output the command(s) corresponding to .conf file for v2.x
|
||||
--list-encodings -E boolean List pgloader known encodings and exit.
|
||||
--logfile -L string Filename where to send the logs.
|
||||
--load-lisp-file -l string Read user code from files
|
||||
--dry-run boolean Only check database connections, don't load anything.
|
||||
--on-error-stop boolean Refrain from handling errors properly.
|
||||
--no-ssl-cert-verification boolean Instruct OpenSSL to bypass verifying certificates.
|
||||
--context -C string Command Context Variables
|
||||
--with string Load options
|
||||
--set string PostgreSQL options
|
||||
--field string Source file fields specification
|
||||
--cast string Specific cast rules
|
||||
--type string Force input source type
|
||||
--encoding string Source expected encoding
|
||||
--before string SQL script to run before loading the data
|
||||
--after string SQL script to run after loading the data
|
||||
--self-upgrade string Path to pgloader newer sources
|
||||
--regress boolean Drive regression testing
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
@ -183,3 +78,13 @@ indexes, foreign keys, comments) and parallel loading of the corrected data:
|
||||
|
||||
$ createdb pagila
|
||||
$ pgloader mysql://user@localhost/sakila postgresql:///pagila
|
||||
|
||||
## LICENCE
|
||||
|
||||
pgloader is available under [The PostgreSQL
|
||||
Licence](http://www.postgresql.org/about/licence/).
|
||||
|
||||
## INSTALL
|
||||
|
||||
Please see full documentation at
|
||||
[https://pgloader.readthedocs.io/](https://pgloader.readthedocs.io/en/latest/install.html).
|
||||
|
||||
@ -55,9 +55,9 @@ author = 'Dimitri Fontaine'
|
||||
# built documents.
|
||||
#
|
||||
# The short X.Y version.
|
||||
version = '3.4'
|
||||
version = '3.6'
|
||||
# The full version, including alpha/beta/rc tags.
|
||||
release = '3.4.1'
|
||||
release = '3.6.3'
|
||||
|
||||
# The language for content autogenerated by Sphinx. Refer to documentation
|
||||
# for a list of supported languages.
|
||||
|
||||
@ -244,6 +244,7 @@ implement the third step in a fully automated way. That's pgloader.
|
||||
intro
|
||||
quickstart
|
||||
tutorial/tutorial
|
||||
install
|
||||
pgloader
|
||||
ref/csv
|
||||
ref/fixed
|
||||
|
||||
213
docs/install.rst
Normal file
213
docs/install.rst
Normal file
@ -0,0 +1,213 @@
|
||||
Installing pgloader
|
||||
===================
|
||||
|
||||
Several distributions are available for pgcopydb.
|
||||
|
||||
debian packages
|
||||
---------------
|
||||
|
||||
You can install pgloader directly from `apt.postgresql.org`__ and from
|
||||
official debian repositories, see `packages.debian.org/pgloader`__.
|
||||
|
||||
::
|
||||
|
||||
$ apt-get install pgloader
|
||||
|
||||
__ https://wiki.postgresql.org/wiki/Apt
|
||||
__ https://packages.debian.org/search?keywords=pgloader
|
||||
|
||||
Docker Images
|
||||
-------------
|
||||
|
||||
Docker images are maintained for each tagged release at dockerhub, and also
|
||||
built from the CI/CD integration on GitHub at each commit to the `main`
|
||||
branch.
|
||||
|
||||
The DockerHub `dimitri/pgloader`__ repository is where the tagged releases
|
||||
are made available. The image uses the Postgres version currently in debian
|
||||
stable.
|
||||
|
||||
__ https://hub.docker.com/r/dimitri/pgloader
|
||||
|
||||
To use the ``dimitri/pgloader`` docker image::
|
||||
|
||||
$ docker run --rm -it dimitri/pgloader:latest pgloader --version
|
||||
|
||||
Or you can use the CI/CD integration that publishes packages from the main
|
||||
branch to the GitHub docker repository::
|
||||
|
||||
$ docker pull ghcr.io/dimitri/pgloader:latest
|
||||
$ docker run --rm -it ghcr.io/dimitri/pgloader:latest pgloader --version
|
||||
$ docker run --rm -it ghcr.io/dimitri/pgloader:latest pgloader --help
|
||||
|
||||
Build from sources
|
||||
------------------
|
||||
|
||||
pgloader is a Common Lisp program, tested using the `SBCL`__ (>= 1.2.5) and
|
||||
`Clozure CL`__ implementations and with `Quicklisp`__ to fetch build
|
||||
dependencies.
|
||||
|
||||
__ http://sbcl.org/
|
||||
__ http://ccl.clozure.com/
|
||||
__ http://www.quicklisp.org/beta/
|
||||
|
||||
When building from sources, you should always build from the current git
|
||||
HEAD as it's basically the only source that is managed in a way to ensure it
|
||||
builds aginst current set of dependencies versions.
|
||||
|
||||
The build system for pgloader uses a Makefile and the Quicklisp Common Lisp
|
||||
packages distribution system.
|
||||
|
||||
The modern build system for pgloader is entirely written in Common Lisp,
|
||||
where the historical name for our operation is `save-lisp-and-die` and can
|
||||
be used that way:
|
||||
|
||||
::
|
||||
|
||||
$ make save
|
||||
|
||||
The legacy build system also uses Buildapp and can be used that way:
|
||||
|
||||
::
|
||||
|
||||
$ make pgloader
|
||||
|
||||
Building from sources on debian
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Install the build dependencies first, then use the Makefile::
|
||||
|
||||
$ apt-get install sbcl unzip libsqlite3-dev make curl gawk freetds-dev libzip-dev
|
||||
$ cd /path/to/pgloader
|
||||
|
||||
$ make save
|
||||
$ ./build/bin/pgloader --help
|
||||
|
||||
Building from sources on RedHat/CentOS
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
To build and install pgloader the Steel Bank Common Lisp package (sbcl) from
|
||||
EPEL, and the freetds packages are required.
|
||||
|
||||
It is recommended to build the RPM yourself, see below, to ensure that all
|
||||
installed files are properly tracked and that you can safely update to newer
|
||||
versions of pgloader as they're released.
|
||||
|
||||
To do an adhoc build and install run ``boostrap-centos.sh`` for CentOS 6 or
|
||||
``bootstrap-centos7.sh`` for CentOS 7 to install the required dependencies.
|
||||
|
||||
Building a pgloader RPM from sources
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The spec file in the root of the pgloader repository can be used to build your
|
||||
own RPM. For production deployments it is recommended that you build this RPM on
|
||||
a dedicated build box and then copy the RPM to your production environment for
|
||||
use; it is considered bad practice to have compilers and build tools present in
|
||||
production environments.
|
||||
|
||||
1. Install the [EPEL repo](https://fedoraproject.org/wiki/EPEL#Quickstart).
|
||||
|
||||
2. Install rpmbuild dependencies::
|
||||
|
||||
sudo yum -y install yum-utils rpmdevtools @"Development Tools"
|
||||
|
||||
3. Install pgloader build dependencies::
|
||||
|
||||
sudo yum-builddep pgloader.spec
|
||||
|
||||
4. Download pgloader source::
|
||||
|
||||
spectool -g -R pgloader.spec
|
||||
|
||||
5. Build the source and binary RPMs (see `rpmbuild --help` for other build
|
||||
options)::
|
||||
|
||||
rpmbuild -ba pgloader.spec
|
||||
|
||||
Building from sources on macOS
|
||||
------------------------------
|
||||
|
||||
We suppose you already have ``git`` and ``make`` available, if that's not
|
||||
the case now is the time to install those tools. The SQLite lib that comes
|
||||
in MacOSX is fine, no need for extra software here.
|
||||
|
||||
You will need to install either SBCL or CCL separately, and when using
|
||||
[brew](http://brew.sh/) it's as simple as:
|
||||
|
||||
::
|
||||
|
||||
$ brew install sbcl
|
||||
$ brew install clozure-cl
|
||||
|
||||
NOTE: Make sure you installed the universal binaries of Freetds, so that
|
||||
they can be loaded correctly.
|
||||
|
||||
::
|
||||
|
||||
$ brew install freetds --universal --build-from-source
|
||||
|
||||
Then use the normal build system for pgloader:
|
||||
|
||||
::
|
||||
|
||||
$ make save
|
||||
$ ./build/bin/pgloader --version
|
||||
|
||||
Building from sources on Windows
|
||||
--------------------------------
|
||||
|
||||
Building pgloader on Windows is supported (in theory), thanks to Common Lisp
|
||||
implementations being available on that platform, and to the Common Lisp
|
||||
Standard for making it easy to write actually portable code.
|
||||
|
||||
It is recommended to have a look at the `issues labelled with Windows
|
||||
support`__ if you run into trouble when building pgloader, because the
|
||||
development team is lacking windows user and in practice we can't maintain
|
||||
the support for that Operating System:
|
||||
|
||||
__ https://github.com/dimitri/pgloader/issues?utf8=✓&q=label%3A%22Windows%20support%22%20>
|
||||
|
||||
If you need ``pgloader.exe`` on windows please condider contributing fixes
|
||||
for that environment and maybe longer term support then. Specifically, a CI
|
||||
integration with a windows build host would allow ensuring that we continue
|
||||
to support that target.
|
||||
|
||||
Building Docker image from sources
|
||||
----------------------------------
|
||||
|
||||
You can build a Docker image from source using SBCL by default::
|
||||
|
||||
$ docker build .
|
||||
|
||||
Or Clozure CL (CCL)::
|
||||
|
||||
$ docker build -f Dockerfile.ccl .
|
||||
|
||||
More options when building from source
|
||||
--------------------------------------
|
||||
|
||||
The ``Makefile`` target ``save`` knows how to produce a Self Contained
|
||||
Binary file for pgloader, found at ``./build/bin/pgloader``::
|
||||
|
||||
$ make save
|
||||
|
||||
By default, the ``Makefile`` uses `SBCL`__ to compile your binary image,
|
||||
though it's possible to build using `Clozure-CL`__.
|
||||
|
||||
__ http://sbcl.org/
|
||||
__ http://ccl.clozure.com/
|
||||
|
||||
::
|
||||
|
||||
$ make CL=ccl64 save
|
||||
|
||||
It is possible to to tweak the default amount of memory that the pgloader
|
||||
image will allow itself using when running through your data (don't ask for
|
||||
more than your current RAM tho). At the moment only the legacy build system
|
||||
includes support for this custom build::
|
||||
|
||||
$ make DYNSIZE=8192 pgloader
|
||||
|
||||
The ``make pgloader`` command when successful outputs a
|
||||
`./build/bin/pgloader` file for you to use.
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user