mirror of
https://github.com/dimitri/pgloader.git
synced 2025-08-07 23:07:00 +02:00
2147a1d07b
This allows creating tables in any target tablespace rather than the default one, and is supported for the various sources having support for the ALTER TABLE clause already.
409 lines
14 KiB
ReStructuredText
409 lines
14 KiB
ReStructuredText
Migrating a PostgreSQL Database to PostgreSQL
|
|
=============================================
|
|
|
|
This command instructs pgloader to load data from a database connection.
|
|
Automatic discovery of the schema is supported, including build of the
|
|
indexes, primary and foreign keys constraints. A default set of casting
|
|
rules are provided and might be overloaded and appended to by the command.
|
|
|
|
Here's a short example of migrating a database from a PostgreSQL server to
|
|
another:
|
|
|
|
::
|
|
|
|
load database
|
|
from pgsql://localhost/pgloader
|
|
into pgsql://localhost/copy
|
|
|
|
including only table names matching 'bits', ~/utilisateur/ in schema 'mysql'
|
|
including only table names matching ~/geolocations/ in schema 'public'
|
|
;
|
|
|
|
PostgreSQL Database Source Specification: FROM
|
|
----------------------------------------------
|
|
|
|
Must be a connection URL pointing to a PostgreSQL database.
|
|
|
|
See the `SOURCE CONNECTION STRING` section above for details on how to write
|
|
the connection string.
|
|
|
|
::
|
|
|
|
pgsql://[user[:password]@][netloc][:port][/dbname][?option=value&...]
|
|
|
|
|
|
PostgreSQL Database Migration Options: WITH
|
|
-------------------------------------------
|
|
|
|
When loading from a `PostgreSQL` database, the following options are
|
|
supported, and the default *WITH* clause is: *no truncate*, *create schema*,
|
|
*create tables*, *include drop*, *create indexes*, *reset sequences*,
|
|
*foreign keys*, *downcase identifiers*, *uniquify index names*, *reindex*.
|
|
|
|
- *include drop*
|
|
|
|
When this option is listed, pgloader drops all the tables in the target
|
|
PostgreSQL database whose names appear in the MySQL database. This
|
|
option allows for using the same command several times in a row until
|
|
you figure out all the options, starting automatically from a clean
|
|
environment. Please note that `CASCADE` is used to ensure that tables
|
|
are dropped even if there are foreign keys pointing to them. This is
|
|
precisely what `include drop` is intended to do: drop all target tables
|
|
and recreate them.
|
|
|
|
Great care needs to be taken when using `include drop`, as it will
|
|
cascade to *all* objects referencing the target tables, possibly
|
|
including other tables that are not being loaded from the source DB.
|
|
|
|
- *include no drop*
|
|
|
|
When this option is listed, pgloader will not include any `DROP`
|
|
statement when loading the data.
|
|
|
|
- *truncate*
|
|
|
|
When this option is listed, pgloader issue the `TRUNCATE` command
|
|
against each PostgreSQL table just before loading data into it.
|
|
|
|
- *no truncate*
|
|
|
|
When this option is listed, pgloader issues no `TRUNCATE` command.
|
|
|
|
- *disable triggers*
|
|
|
|
When this option is listed, pgloader issues an `ALTER TABLE ... DISABLE
|
|
TRIGGER ALL` command against the PostgreSQL target table before copying
|
|
the data, then the command `ALTER TABLE ... ENABLE TRIGGER ALL` once the
|
|
`COPY` is done.
|
|
|
|
This option allows loading data into a pre-existing table ignoring the
|
|
*foreign key constraints* and user defined triggers and may result in
|
|
invalid *foreign key constraints* once the data is loaded. Use with
|
|
care.
|
|
|
|
- *create tables*
|
|
|
|
When this option is listed, pgloader creates the table using the meta
|
|
data found in the `MySQL` file, which must contain a list of fields with
|
|
their data type. A standard data type conversion from DBF to PostgreSQL
|
|
is done.
|
|
|
|
- *create no tables*
|
|
|
|
When this option is listed, pgloader skips the creation of table before
|
|
loading data, target tables must then already exist.
|
|
|
|
Also, when using *create no tables* pgloader fetches the metadata from
|
|
the current target database and checks type casting, then will remove
|
|
constraints and indexes prior to loading the data and install them back
|
|
again once the loading is done.
|
|
|
|
- *create indexes*
|
|
|
|
When this option is listed, pgloader gets the definitions of all the
|
|
indexes found in the MySQL database and create the same set of index
|
|
definitions against the PostgreSQL database.
|
|
|
|
- *create no indexes*
|
|
|
|
When this option is listed, pgloader skips the creating indexes.
|
|
|
|
- *drop indexes*
|
|
|
|
When this option is listed, pgloader drops the indexes in the target
|
|
database before loading the data, and creates them again at the end
|
|
of the data copy.
|
|
|
|
- *reindex*
|
|
|
|
When this option is used, pgloader does both *drop indexes* before
|
|
loading the data and *create indexes* once data is loaded.
|
|
|
|
- *drop schema*
|
|
|
|
When this option is listed, pgloader drops the target schema in the
|
|
target PostgreSQL database before creating it again and all the objects
|
|
it contains. The default behavior doesn't drop the target schemas.
|
|
|
|
- *foreign keys*
|
|
|
|
When this option is listed, pgloader gets the definitions of all the
|
|
foreign keys found in the MySQL database and create the same set of
|
|
foreign key definitions against the PostgreSQL database.
|
|
|
|
- *no foreign keys*
|
|
|
|
When this option is listed, pgloader skips creating foreign keys.
|
|
|
|
- *reset sequences*
|
|
|
|
When this option is listed, at the end of the data loading and after the
|
|
indexes have all been created, pgloader resets all the PostgreSQL
|
|
sequences created to the current maximum value of the column they are
|
|
attached to.
|
|
|
|
The options *schema only* and *data only* have no effects on this
|
|
option.
|
|
|
|
- *reset no sequences*
|
|
|
|
When this option is listed, pgloader skips resetting sequences after the
|
|
load.
|
|
|
|
The options *schema only* and *data only* have no effects on this
|
|
option.
|
|
|
|
- *downcase identifiers*
|
|
|
|
When this option is listed, pgloader converts all MySQL identifiers
|
|
(table names, index names, column names) to *downcase*, except for
|
|
PostgreSQL *reserved* keywords.
|
|
|
|
The PostgreSQL *reserved* keywords are determined dynamically by using
|
|
the system function `pg_get_keywords()`.
|
|
|
|
- *quote identifiers*
|
|
|
|
When this option is listed, pgloader quotes all MySQL identifiers so
|
|
that their case is respected. Note that you will then have to do the
|
|
same thing in your application code queries.
|
|
|
|
- *schema only*
|
|
|
|
When this option is listed pgloader refrains from migrating the data
|
|
over. Note that the schema in this context includes the indexes when the
|
|
option *create indexes* has been listed.
|
|
|
|
- *data only*
|
|
|
|
When this option is listed pgloader only issues the `COPY` statements,
|
|
without doing any other processing.
|
|
|
|
- *rows per range*
|
|
|
|
How many rows are fetched per `SELECT` query when using *multiple
|
|
readers per thread*, see above for details.
|
|
|
|
PostgreSQL Database Casting Rules
|
|
---------------------------------
|
|
|
|
The command *CAST* introduces user-defined casting rules.
|
|
|
|
The cast clause allows to specify custom casting rules, either to overload
|
|
the default casting rules or to amend them with special cases.
|
|
|
|
A casting rule is expected to follow one of the forms::
|
|
|
|
type <type-name> [ <guard> ... ] to <pgsql-type-name> [ <option> ... ]
|
|
column <table-name>.<column-name> [ <guards> ] to ...
|
|
|
|
It's possible for a *casting rule* to either match against a PostgreSQL data
|
|
type or against a given *column name* in a given *table name*. So it's
|
|
possible to migrate a table from a PostgreSQL database while changing and
|
|
`int` column to a `bigint` one, automatically.
|
|
|
|
The *casting rules* are applied in order, the first match prevents following
|
|
rules to be applied, and user defined rules are evaluated first.
|
|
|
|
The supported guards are:
|
|
|
|
- *when default 'value'*
|
|
|
|
The casting rule is only applied against MySQL columns of the source
|
|
type that have given *value*, which must be a single-quoted or a
|
|
double-quoted string.
|
|
|
|
- *when typemod expression*
|
|
|
|
The casting rule is only applied against MySQL columns of the source
|
|
type that have a *typemod* value matching the given *typemod
|
|
expression*. The *typemod* is separated into its *precision* and *scale*
|
|
components.
|
|
|
|
Example of a cast rule using a *typemod* guard::
|
|
|
|
type char when (= precision 1) to char keep typemod
|
|
|
|
This expression casts MySQL `char(1)` column to a PostgreSQL column of
|
|
type `char(1)` while allowing for the general case `char(N)` will be
|
|
converted by the default cast rule into a PostgreSQL type `varchar(N)`.
|
|
|
|
- *with extra auto_increment*
|
|
|
|
The casting rule is only applied against PostgreSQL attached to a
|
|
sequence. This can be the result of doing that manually, using a
|
|
`serial` or a `bigserial` data type, or an `identity` column.
|
|
|
|
|
|
The supported casting options are:
|
|
|
|
- *drop default*, *keep default*
|
|
|
|
When the option *drop default* is listed, pgloader drops any
|
|
existing default expression in the MySQL database for columns of the
|
|
source type from the `CREATE TABLE` statement it generates.
|
|
|
|
The spelling *keep default* explicitly prevents that behaviour and
|
|
can be used to overload the default casting rules.
|
|
|
|
- *drop not null*, *keep not null*, *set not null*
|
|
|
|
When the option *drop not null* is listed, pgloader drops any
|
|
existing `NOT NULL` constraint associated with the given source
|
|
MySQL datatype when it creates the tables in the PostgreSQL
|
|
database.
|
|
|
|
The spelling *keep not null* explicitly prevents that behaviour and
|
|
can be used to overload the default casting rules.
|
|
|
|
When the option *set not null* is listed, pgloader sets a `NOT NULL`
|
|
constraint on the target column regardless whether it has been set
|
|
in the source MySQL column.
|
|
|
|
- *drop typemod*, *keep typemod*
|
|
|
|
When the option *drop typemod* is listed, pgloader drops any
|
|
existing *typemod* definition (e.g. *precision* and *scale*) from
|
|
the datatype definition found in the MySQL columns of the source
|
|
type when it created the tables in the PostgreSQL database.
|
|
|
|
The spelling *keep typemod* explicitly prevents that behaviour and
|
|
can be used to overload the default casting rules.
|
|
|
|
- *using*
|
|
|
|
This option takes as its single argument the name of a function to
|
|
be found in the `pgloader.transforms` Common Lisp package. See above
|
|
for details.
|
|
|
|
It's possible to augment a default cast rule (such as one that
|
|
applies against `ENUM` data type for example) with a *transformation
|
|
function* by omitting entirely the `type` parts of the casting rule,
|
|
as in the following example::
|
|
|
|
column enumerate.foo using empty-string-to-null
|
|
|
|
PostgreSQL Views Support
|
|
------------------------
|
|
|
|
PostgreSQL views support allows pgloader to migrate view as if they were
|
|
base tables. This feature then allows for on-the-fly transformation of the
|
|
source schema, as the view definition is used rather than the base data.
|
|
|
|
MATERIALIZE VIEWS
|
|
^^^^^^^^^^^^^^^^^
|
|
|
|
This clause allows you to implement custom data processing at the data
|
|
source by providing a *view definition* against which pgloader will query
|
|
the data. It's not possible to just allow for plain `SQL` because we want to
|
|
know a lot about the exact data types of each column involved in the query
|
|
output.
|
|
|
|
This clause expect a comma separated list of view definitions, each one
|
|
being either the name of an existing view in your database or the following
|
|
expression::
|
|
|
|
*name* `AS` `$$` *sql query* `$$`
|
|
|
|
The *name* and the *sql query* will be used in a `CREATE VIEW` statement at
|
|
the beginning of the data loading, and the resulting view will then be
|
|
dropped at the end of the data loading.
|
|
|
|
MATERIALIZE ALL VIEWS
|
|
^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Same behaviour as *MATERIALIZE VIEWS* using the dynamic list of views as
|
|
returned by PostgreSQL rather than asking the user to specify the list.
|
|
|
|
PostgreSQL Partial Migration
|
|
----------------------------
|
|
|
|
INCLUDING ONLY TABLE NAMES MATCHING
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Introduce a comma separated list of table names or *regular expression* used
|
|
to limit the tables to migrate to a sublist.
|
|
|
|
Example::
|
|
|
|
including only table names matching ~/film/, 'actor' in schema 'public'
|
|
|
|
EXCLUDING TABLE NAMES MATCHING
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Introduce a comma separated list of table names or *regular expression* used
|
|
to exclude table names from the migration. This filter only applies to the
|
|
result of the *INCLUDING* filter.
|
|
|
|
::
|
|
|
|
excluding table names matching ~<ory> in schema 'public'
|
|
|
|
PostgreSQL Schema Transformations
|
|
---------------------------------
|
|
|
|
ALTER TABLE NAMES MATCHING
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Introduce a comma separated list of table names or *regular expressions*
|
|
that you want to target in the pgloader *ALTER TABLE* command. Available
|
|
actions are *SET SCHEMA*, *RENAME TO*, and *SET*::
|
|
|
|
ALTER TABLE NAMES MATCHING ~/_list$/, 'sales_by_store', ~/sales_by/
|
|
IN SCHEMA 'public'
|
|
SET SCHEMA 'mv'
|
|
|
|
ALTER TABLE NAMES MATCHING 'film' IN SCHEMA 'public' RENAME TO 'films'
|
|
|
|
ALTER TABLE NAMES MATCHING ~/./ IN SCHEMA 'public' SET (fillfactor='40')
|
|
|
|
ALTER TABLE NAMES MATCHING ~/./ IN SCHEMA 'public' SET TABLESPACE 'pg_default'
|
|
|
|
You can use as many such rules as you need. The list of tables to be
|
|
migrated is searched in pgloader memory against the *ALTER TABLE* matching
|
|
rules, and for each command pgloader stops at the first matching criteria
|
|
(regexp or string).
|
|
|
|
No *ALTER TABLE* command is sent to PostgreSQL, the modification happens at
|
|
the level of the pgloader in-memory representation of your source database
|
|
schema. In case of a name change, the mapping is kept and reused in the
|
|
*foreign key* and *index* support.
|
|
|
|
The *SET ()* action takes effect as a *WITH* clause for the `CREATE TABLE`
|
|
command that pgloader will run when it has to create a table.
|
|
|
|
The *SET TABLESPACE* action takes effect as a *TABLESPACE* clause for the
|
|
`CREATE TABLE` command that pgloader will run when it has to create a table.
|
|
|
|
PostgreSQL Migration: limitations
|
|
---------------------------------
|
|
|
|
The only PostgreSQL objects supported at this time in pgloader are
|
|
extensions, schema, tables, indexes and constraints. Anything else is ignored.
|
|
|
|
- Views are not migrated,
|
|
|
|
Supporting views might require implementing a full SQL parser for the
|
|
MySQL dialect with a porting engine to rewrite the SQL against
|
|
PostgreSQL, including renaming functions and changing some constructs.
|
|
|
|
While it's not theoretically impossible, don't hold your breath.
|
|
|
|
- Triggers are not migrated
|
|
|
|
The difficulty of doing so is not yet assessed.
|
|
|
|
- Stored Procedures and Functions are not migrated.
|
|
|
|
|
|
Default PostgreSQL Casting Rules
|
|
--------------------------------
|
|
|
|
When migrating from PostgreSQL the following Casting Rules are provided::
|
|
|
|
type int with extra auto_increment to serial
|
|
type bigint with extra auto_increment to bigserial
|
|
type "character varying" to text drop typemod
|
|
|
|
|