mirror of
https://github.com/dimitri/pgloader.git
synced 2025-08-08 07:16:58 +02:00
370038a74e
See #599 again, wherein I missed that the URL error was not a copy-paste'o but rather an error in the documentation itself…
209 lines
14 KiB
HTML
209 lines
14 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
|
<meta name="description" content="">
|
|
<meta name="author" content="">
|
|
<link rel="shortcut icon" href="../../docs-assets/ico/favicon.png">
|
|
|
|
<title>pgloader</title>
|
|
|
|
<!-- Bootstrap core CSS -->
|
|
<link href="../dist/css/bootstrap.css" rel="stylesheet">
|
|
|
|
<!-- Custom styles for this template -->
|
|
<link href="../dist/carousel.css" rel="stylesheet">
|
|
</head>
|
|
<!-- NAVBAR
|
|
================================================== -->
|
|
<body>
|
|
<div class="navbar-wrapper">
|
|
<div class="container">
|
|
|
|
<div class="navbar navbar-inverse navbar-static-top" role="navigation">
|
|
<div class="container">
|
|
<div class="navbar-header">
|
|
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse">
|
|
<span class="sr-only">Toggle navigation</span>
|
|
<span class="icon-bar"></span>
|
|
<span class="icon-bar"></span>
|
|
<span class="icon-bar"></span>
|
|
</button>
|
|
<a class="navbar-brand" href="../index.html">pgloader</a>
|
|
</div>
|
|
<div class="navbar-collapse collapse">
|
|
<ul class="nav navbar-nav">
|
|
<li><a href="../index.html">Home</a></li>
|
|
<li><a href="quickstart.html">Quick Start</a></li>
|
|
<li><a href="pgloader.1.html">Reference documentation</a></li>
|
|
<li class="dropdown active">
|
|
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Advanced HowTos <b class="caret"></b></a>
|
|
<ul class="dropdown-menu">
|
|
<li class="dropdown-header">Plain Files</li>
|
|
<li><a href="csv.html">CSV</a></li>
|
|
<li><a href="fixed.html">Fixed format</a></li>
|
|
<li><a href="geolite.html">Geolite</a></li>
|
|
<li class="divider"></li>
|
|
<li class="dropdown-header">Databases</li>
|
|
<li><a href="dBase.html">dBase</a></li>
|
|
<li><a href="sqlite.html">SQLite</a></li>
|
|
<li><a href="mysql.html">MySQL</a></li>
|
|
</ul>
|
|
</li>
|
|
<li><a href="../download.html">Download</a></li>
|
|
<li><a href="../sponsors.html">Sponsors</a></li>
|
|
<li><a href="../pgloader-moral-license.html">License</a></li>
|
|
</ul>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
|
|
</div>
|
|
</div>
|
|
|
|
<!-- an empty carousel -->
|
|
<div id="myCarousel" class="carousel slide" data-ride="carousel" style="height: 100px">
|
|
<div class="carousel-inner" style="height: 100px">
|
|
<div class="item active" style="height: 100px">
|
|
<img data-src="holder.js/900x100/auto/#777:#7a7a7a" style="height: 100px">
|
|
<!-- <div class="container"> -->
|
|
<!-- <div class="carousel-caption"> -->
|
|
<!-- <h1>Load data into PostgreSQL. Fast.</h1> -->
|
|
<!-- <p></p> -->
|
|
<!-- </div> -->
|
|
<!-- </div> -->
|
|
</div>
|
|
</div>
|
|
</div><!-- /.carousel -->
|
|
|
|
<div class="container">
|
|
<div class="row">
|
|
<div class="col-md-2"> </div>
|
|
<div class="col-md-8">
|
|
<h1>Migrating from MySQL to PostgreSQL</h1><p>If you want to migrate your data over to <a href="http://www.postgresql.org">PostgreSQL</a> from MySQL then pgloader is the tool of choice! </p><p>Most tools around are skipping the main problem with migrating from MySQL, which is to do with the type casting and data sanitizing that needs to be done. pgloader will not leave you alone on those topics. </p><h2>In a Single Command Line</h2><p>As an example, we will use the f1db database from <http://ergast.com/mrd/> which which provides a historical record of motor racing data for non-commercial purposes. You can either use their API or download the whole database at <a href="http://ergast.com/downloads/f1db.sql.gz">http://ergast.com/downloads/f1db.sql.gz</a>. Once you've done that load the database in MySQL: </p><pre><code>$ mysql -u root
|
|
> create database f1db;
|
|
> source f1db.sql </code></pre><p>Now let's migrate this database into PostgreSQL in a single command line: </p><pre><code>$ createdb f1db
|
|
$ pgloader mysql://root@localhost/f1db pgsql:///f1db </code></pre><p>Done! All with schema, table definitions, constraints, indexes, primary keys, <em>auto_increment</em> columns turned into <em>bigserial</em> , foreign keys, comments, and if you had some MySQL default values such as <em>ON UPDATE CURRENT_TIMESTAMP</em> they would have been translated to a <a href="https://www.postgresql.org/docs/current/static/plpgsql-trigger.html">PostgreSQL before update trigger</a> automatically. </p><pre><code>$ pgloader mysql://root@localhost/f1db pgsql:///f1db
|
|
2017-06-16T08:56:14.064000+02:00 LOG Main logs in '/private/tmp/pgloader/pgloader.log'
|
|
2017-06-16T08:56:14.068000+02:00 LOG Data errors in '/private/tmp/pgloader/'
|
|
2017-06-16T08:56:19.542000+02:00 LOG report summary reset
|
|
table name read imported errors total time
|
|
------------------------- --------- --------- --------- --------------
|
|
fetch meta data 33 33 0 0.365s
|
|
Create Schemas 0 0 0 0.007s
|
|
Create SQL Types 0 0 0 0.006s
|
|
Create tables 26 26 0 0.068s
|
|
Set Table OIDs 13 13 0 0.012s
|
|
------------------------- --------- --------- --------- --------------
|
|
f1db.constructorresults 11011 11011 0 0.205s
|
|
f1db.circuits 73 73 0 0.150s
|
|
f1db.constructors 208 208 0 0.059s
|
|
f1db.constructorstandings 11766 11766 0 0.365s
|
|
f1db.drivers 841 841 0 0.268s
|
|
f1db.laptimes 413578 413578 0 2.892s
|
|
f1db.driverstandings 31420 31420 0 0.583s
|
|
f1db.pitstops 5796 5796 0 2.154s
|
|
f1db.races 976 976 0 0.227s
|
|
f1db.qualifying 7257 7257 0 0.228s
|
|
f1db.seasons 68 68 0 0.527s
|
|
f1db.results 23514 23514 0 0.658s
|
|
f1db.status 133 133 0 0.130s
|
|
------------------------- --------- --------- --------- --------------
|
|
COPY Threads Completion 39 39 0 4.303s
|
|
Create Indexes 20 20 0 1.497s
|
|
Index Build Completion 20 20 0 0.214s
|
|
Reset Sequences 0 10 0 0.058s
|
|
Primary Keys 13 13 0 0.012s
|
|
Create Foreign Keys 0 0 0 0.000s
|
|
Create Triggers 0 0 0 0.001s
|
|
Install Comments 0 0 0 0.000s
|
|
------------------------- --------- --------- --------- --------------
|
|
Total import time 506641 506641 0 5.547s </code></pre><p>You may need to have special cases to take care of tho, or views that you want to materialize while doing the migration. In advanced case you can use the pgloader command. </p><h2>The Command</h2><p>To load data with <a href="http://pgloader.tapoueh.org/">pgloader</a> you need to define in a <em>command</em> the operations in some details. Here's our example for loading the <a href="http://dev.mysql.com/doc/sakila/en/">MySQL Sakila Sample Database</a>: </p><p>Here's our command: </p><pre><code>load database
|
|
from mysql://root@localhost/sakila
|
|
into postgresql:///sakila
|
|
|
|
WITH include drop, create tables, no truncate,
|
|
create indexes, reset sequences, foreign keys
|
|
|
|
SET maintenance_work_mem to '128MB', work_mem to '12MB', search_path to 'sakila'
|
|
|
|
CAST type datetime to timestamptz
|
|
drop default drop not null using zero-dates-to-null,
|
|
type date drop not null drop default using zero-dates-to-null
|
|
|
|
MATERIALIZE VIEWS film_list, staff_list
|
|
|
|
-- INCLUDING ONLY TABLE NAMES MATCHING ~/film/, 'actor'
|
|
-- EXCLUDING TABLE NAMES MATCHING ~<ory>
|
|
|
|
BEFORE LOAD DO
|
|
$$ create schema if not exists sakila; $$; </code></pre><p>You can see the full list of options in the <a href="pgloader.1.html">pgloader reference manual</a>, with a complete description of the options you see here. </p><p>Note that here pgloader will benefit from the meta-data information found in the MySQL database to create a PostgreSQL database capable of hosting the data as described, then load the data. </p><p>In particular, some specific <em>casting rules</em> are given here, to cope with date values such as <code>0000-00-00</code> that MySQL allows and PostgreSQL rejects for not existing in our calendar. It's possible to add per-column casting rules too, which is useful is some of your <code>tinyint</code> are in fact <code>smallint</code> while some others are in fact <code>boolean</code> values. </p><p>Finaly note that we are using the <em>MATERIALIZE VIEWS</em> clause of pgloader: the selected views here will be migrated over to PostgreSQL <em>with their contents</em>. </p><p>It's possible to use the <em>MATERIALIZE VIEWS</em> clause and give both the name and the SQL (in MySQL dialect) definition of view, then pgloader creates the view before loading the data, then drops it again at the end. </p><h2>Loading the data</h2><p>Let's start the <code>pgloader</code> command with our <code>sakila.load</code> command file: </p><pre><code>$ pgloader sakila.load
|
|
... LOG Starting pgloader, log system is ready.
|
|
... LOG Parsing commands from file "/Users/dim/dev/pgloader/test/sakila.load"
|
|
<WARNING: table "xxx" does not exists have been edited away>
|
|
|
|
table name read imported errors time
|
|
---------------------- --------- --------- --------- --------------
|
|
before load 1 1 0 0.007s
|
|
fetch meta data 45 45 0 0.402s
|
|
create, drop 0 36 0 0.208s
|
|
---------------------- --------- --------- --------- --------------
|
|
actor 200 200 0 0.071s
|
|
address 603 603 0 0.035s
|
|
category 16 16 0 0.018s
|
|
city 600 600 0 0.037s
|
|
country 109 109 0 0.023s
|
|
customer 599 599 0 0.073s
|
|
film 1000 1000 0 0.135s
|
|
film_actor 5462 5462 0 0.236s
|
|
film_category 1000 1000 0 0.070s
|
|
film_text 1000 1000 0 0.080s
|
|
inventory 4581 4581 0 0.136s
|
|
language 6 6 0 0.036s
|
|
payment 16049 16049 0 0.539s
|
|
rental 16044 16044 0 0.648s
|
|
staff 2 2 0 0.041s
|
|
store 2 2 0 0.036s
|
|
film_list 997 997 0 0.247s
|
|
staff_list 2 2 0 0.135s
|
|
Index Build Completion 0 0 0 0.000s
|
|
---------------------- --------- --------- --------- --------------
|
|
Create Indexes 41 41 0 0.964s
|
|
Reset Sequences 0 1 0 0.035s
|
|
Foreign Keys 22 22 0 0.254s
|
|
---------------------- --------- --------- --------- --------------
|
|
Total import time 48272 48272 0 3.502s </code></pre><p>The <em>WARNING</em> messages we see here are expected as the PostgreSQL database is empty when running the command, and pgloader is using the SQL commands <code>DROP TABLE IF EXISTS</code> when the given command uses the <code>include drop</code> option. </p><p>Note that the output of the command has been edited to facilitate its browsing online. </p> </div>
|
|
<div class="col-md-2"> </div>
|
|
</div>
|
|
|
|
<!-- FOOTER -->
|
|
<footer>
|
|
<p class="pull-right"><a href="#">Back to top</a></p>
|
|
<p>© 2013-2014 Dimitri Fontaine. ·</p>
|
|
</footer>
|
|
|
|
</div><!-- /.container -->
|
|
|
|
|
|
<!-- Bootstrap core JavaScript
|
|
================================================== -->
|
|
<!-- Placed at the end of the document so the pages load faster -->
|
|
<script src="https://code.jquery.com/jquery-1.10.2.min.js"></script>
|
|
<script src="../dist/js/bootstrap.min.js"></script>
|
|
<!-- <script src="docs-assets/js/holder.js"></script> -->
|
|
|
|
<script>
|
|
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
|
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
|
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
|
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
|
|
|
|
ga('create', 'UA-47059482-2', 'tapoueh.org');
|
|
ga('send', 'pageview');
|
|
|
|
</script>
|
|
</body>
|
|
</html>
|