Switch the documentation to using Sphinx.

The website is moving to pgloader.org and readthedocs.io is going to be
integrated. Let's see what happens. The docs build fine locally with the
sphinx tools and the docs/Makefile.

Having separate files for the documentation should help ease the maintenance
and add new topics, such as support for Common Lisp Hackers level docs,
which are currently missing.

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+SPHINXPROJ    = pgloader
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/_build/html/.buildinfo

@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: 59392f424cbb8621f1b34cf4df7fbec1
+tags: 645f666f9bcd5a90fca523b33c5a78b7

docs/_build/html/_sources/bugreport.rst.txt

@@ -0,0 +1,49 @@
+Reporting Bugs
+==============
+
+pgloader is a software and as such contains bugs. Most bugs are easy to
+solve and taken care of in a short delay. For this to be possible though,
+bug reports need to follow those recommandations:
+
+  - include pgloader version,
+  - include problematic input and output,
+  - include a description of the output you expected,
+  - explain the difference between the ouput you have and the one you expected,
+  - include a self-reproducing test-case
+
+Test Cases to Reproduce Bugs
+----------------------------
+
+Use the *inline* source type to help reproduce a bug, as in the pgloader tests::
+
+  LOAD CSV
+     FROM INLINE
+     INTO postgresql://dim@localhost/pgloader?public."HS"
+
+     WITH truncate,
+          fields terminated by '\t',
+          fields not enclosed,
+          fields escaped by backslash-quote,
+          quote identifiers
+
+      SET work_mem to '128MB',
+          standard_conforming_strings to 'on',
+          application_name to 'my app name'
+
+   BEFORE LOAD DO
+    $$ create extension if not exists hstore; $$,
+    $$ drop table if exists "HS"; $$,
+    $$ CREATE TABLE "HS"
+       (
+          id     serial primary key,
+          kv     hstore
+       )
+    $$;
+  
+  
+  1	email=>foo@example.com,a=>b
+  2	test=>value
+  3	a=>b,c=>"quoted hstore value",d=>other
+  4	baddata
+
+

docs/_build/html/_sources/index.rst.txt

@@ -0,0 +1,33 @@
+.. pgloader documentation master file, created by
+   sphinx-quickstart on Tue Dec  5 19:23:32 2017.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Welcome to pgloader's documentation!
+====================================
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Table Of Contents:
+
+   intro
+   tutorial/tutorial
+   pgloader
+   ref/csv
+   ref/fixed
+   ref/copy
+   ref/dbf
+   ref/ixf
+   ref/archive
+   ref/mysql
+   ref/sqlite
+   ref/mssql
+   ref/transforms
+   bugreport
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`

docs/_build/html/_sources/intro.rst.txt

@@ -0,0 +1,63 @@
+Introduction
+============
+
+pgloader loads data from various sources into PostgreSQL. It can
+transform the data it reads on the fly and submit raw SQL before and
+after the loading.  It uses the `COPY` PostgreSQL protocol to stream
+the data into the server, and manages errors by filling a pair of
+*reject.dat* and *reject.log* files.
+
+pgloader knows how to read data from different kind of sources:
+
+  * Files
+    * CSV
+    * Fixed Format
+    * DBF
+  * Databases
+    * SQLite
+    * MySQL
+    * MS SQL Server
+
+The level of automation provided by pgloader depends on the data source
+type. In the case of CSV and Fixed Format files, a full description of the
+expected input properties must be given to pgloader. In the case of a
+database, pgloader connects to the live service and knows how to fetch the
+metadata it needs directly from it.
+
+Continuous Migration
+--------------------
+
+pgloader is meant to migrate a whole database in a single command line and
+without any manual intervention. The goal is to be able to setup a
+*Continuous Integration* environment as described in the `Project
+Methodology <http://mysqltopgsql.com/project/>`_ document of the `MySQL to
+PostgreSQL <http://mysqltopgsql.com/project/>`_ webpage.
+
+  1. Setup your target PostgreSQL Architecture
+  2. Fork a Continuous Integration environment that uses PostgreSQL
+  3. Migrate the data over and over again every night, from production
+  4. As soon as the CI is all green using PostgreSQL, schedule the D-Day
+  5. Migrate without suprise and enjoy! 
+
+In order to be able to follow this great methodology, you need tooling to
+implement the third step in a fully automated way. That's pgloader.
+
+Commands
+--------
+
+pgloader implements its own *Command Language*, a DSL that allows to specify
+every aspect of the data load and migration to implement. Some of the
+features provided in the language are only available for a specific source
+type.
+
+Command Line
+------------
+
+The pgloader command line accepts those two variants::
+
+    pgloader [<options>] [<command-file>]...
+    pgloader [<options>] SOURCE TARGET
+
+Either you have a *command-file* containing migration specifications in the
+pgloader *Command Language*, or you can give a *Source* for the data and a
+PostgreSQL database connection *Target* where to load the data into.

docs/_build/html/_sources/pgloader-usage-examples.rst.txt

@@ -0,0 +1,163 @@
+Pgloader Usage Examples
+=======================
+
+Currently not included, because redundant with the tutorial.
+
+Usage Examples
+--------------
+
+Review the command line options and pgloader's version::
+
+    pgloader --help
+    pgloader --version
+
+Loading from a complex command
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Use the command file as the pgloader command argument, pgloader will parse
+that file and execute the commands found in it::
+
+    pgloader --verbose ./test/csv-districts.load
+
+CSV
+^^^
+
+Load data from a CSV file into a pre-existing table in your database, having
+pgloader guess the CSV properties (separator, quote and escape character)::
+
+    pgloader ./test/data/matching-1.csv pgsql:///pgloader?tablename=matching
+
+Load data from a CSV file into a pre-existing table in your database, with
+expanded options::
+
+    pgloader --type csv                                   \
+             --field id --field field                     \
+             --with truncate                              \
+             --with "fields terminated by ','"            \
+             ./test/data/matching-1.csv                   \
+             postgres:///pgloader?tablename=matching
+
+In that example the whole loading is driven from the command line, bypassing
+the need for writing a command in the pgloader command syntax entirely. As
+there's no command though, the extra inforamtion needed must be provided on
+the command line using the `--type` and `--field` and `--with` switches.
+
+For documentation about the available syntaxes for the `--field` and
+`--with` switches, please refer to the CSV section later in the man page.
+
+Note also that the PostgreSQL URI includes the target *tablename*.
+
+Reading from STDIN
+^^^^^^^^^^^^^^^^^^
+
+File based pgloader sources can be loaded from the standard input, as in the
+following example::
+
+    pgloader --type csv                                         \
+             --field "usps,geoid,aland,awater,aland_sqmi,awater_sqmi,intptlat,intptlong" \
+             --with "skip header = 1"                          \
+             --with "fields terminated by '\t'"                \
+             -                                                 \
+             postgresql:///pgloader?districts_longlat          \
+             < test/data/2013_Gaz_113CDs_national.txt
+
+The dash (`-`) character as a source is used to mean *standard input*, as
+usual in Unix command lines. It's possible to stream compressed content to
+pgloader with this technique, using the Unix pipe:
+
+    gunzip -c source.gz | pgloader --type csv ... - pgsql:///target?foo
+
+Loading from CSV available through HTTP
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The same command as just above can also be run if the CSV file happens to be
+found on a remote HTTP location::
+
+    pgloader --type csv                                                     \
+             --field "usps,geoid,aland,awater,aland_sqmi,awater_sqmi,intptlat,intptlong" \
+             --with "skip header = 1"                                       \
+             --with "fields terminated by '\t'"                             \
+             http://pgsql.tapoueh.org/temp/2013_Gaz_113CDs_national.txt     \
+             postgresql:///pgloader?districts_longlat
+
+Some more options have to be used in that case, as the file contains a
+one-line header (most commonly that's column names, could be a copyright
+notice). Also, in that case, we specify all the fields right into a single
+`--field` option argument.
+
+Again, the PostgreSQL target connection string must contain the *tablename*
+option and you have to ensure that the target table exists and may fit the
+data. Here's the SQL command used in that example in case you want to try it
+yourself::
+
+    create table districts_longlat
+    (
+             usps        text,
+             geoid       text,
+             aland       bigint,
+             awater      bigint,
+             aland_sqmi  double precision,
+             awater_sqmi double precision,
+             intptlat    double precision,
+             intptlong   double precision
+    );
+
+Also notice that the same command will work against an archived version of
+the same data, e.g.
+http://pgsql.tapoueh.org/temp/2013_Gaz_113CDs_national.txt.gz.
+
+Finally, it's important to note that pgloader first fetches the content from
+the HTTP URL it to a local file, then expand the archive when it's
+recognized to be one, and only then processes the locally expanded file.
+
+In some cases, either because pgloader has no direct support for your
+archive format or maybe because expanding the archive is not feasible in
+your environment, you might want to *stream* the content straight from its
+remote location into PostgreSQL. Here's how to do that, using the old battle
+tested Unix Pipes trick::
+
+    curl http://pgsql.tapoueh.org/temp/2013_Gaz_113CDs_national.txt.gz \
+    | gunzip -c                                                        \
+    | pgloader --type csv                                              \
+               --field "usps,geoid,aland,awater,aland_sqmi,awater_sqmi,intptlat,intptlong"
+               --with "skip header = 1"                                \
+               --with "fields terminated by '\t'"                      \
+               -                                                       \
+               postgresql:///pgloader?districts_longlat
+
+Now the OS will take care of the streaming and buffering between the network
+and the commands and pgloader will take care of streaming the data down to
+PostgreSQL.
+
+Migrating from SQLite
+^^^^^^^^^^^^^^^^^^^^^
+
+The following command will open the SQLite database, discover its tables
+definitions including indexes and foreign keys, migrate those definitions
+while *casting* the data type specifications to their PostgreSQL equivalent
+and then migrate the data over::
+
+    createdb newdb
+    pgloader ./test/sqlite/sqlite.db postgresql:///newdb
+
+Migrating from MySQL
+^^^^^^^^^^^^^^^^^^^^
+
+Just create a database where to host the MySQL data and definitions and have
+pgloader do the migration for you in a single command line::
+
+    createdb pagila
+    pgloader mysql://user@localhost/sakila postgresql:///pagila
+
+Fetching an archived DBF file from a HTTP remote location
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+It's possible for pgloader to download a file from HTTP, unarchive it, and
+only then open it to discover the schema then load the data::
+
+    createdb foo
+    pgloader --type dbf http://www.insee.fr/fr/methodes/nomenclatures/cog/telechargement/2013/dbf/historiq2013.zip postgresql:///foo
+
+Here it's not possible for pgloader to guess the kind of data source it's
+being given, so it's necessary to use the `--type` command line switch.
+

docs/_build/html/_sources/pgloader.rst.txt

@@ -0,0 +1,713 @@
+PgLoader Reference Manual
+=========================
+
+pgloader loads data from various sources into PostgreSQL. It can
+transform the data it reads on the fly and submit raw SQL before and
+after the loading.  It uses the `COPY` PostgreSQL protocol to stream
+the data into the server, and manages errors by filling a pair of
+*reject.dat* and *reject.log* files.
+
+pgloader operates either using commands which are read from files::
+
+    pgloader commands.load
+
+or by using arguments and options all provided on the command line::
+
+    pgloader SOURCE TARGET
+
+Arguments
+---------
+
+The pgloader arguments can be as many load files as needed, or a couple of
+connection strings to a specific input file.
+
+Source Connection String
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+The source connection string format is as follows::
+
+    format:///absolute/path/to/file.ext
+    format://./relative/path/to/file.ext
+
+Where format might be one of `csv`, `fixed`, `copy`, `dbf`, `db3` or `ixf`.::
+
+    db://user:pass@host:port/dbname
+
+Where db might be of `sqlite`, `mysql` or `mssql`.
+
+When using a file based source format, pgloader also support natively
+fetching the file from an http location and decompressing an archive if
+needed. In that case it's necessary to use the `--type` option to specify
+the expected format of the file. See the examples below.
+
+Also note that some file formats require describing some implementation
+details such as columns to be read and delimiters and quoting when loading
+from csv.
+
+For more complex loading scenarios, you will need to write a full fledge
+load command in the syntax described later in this document.
+
+Target Connection String
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+The target connection string format is described in details later in this
+document, see Section Connection String.
+
+Options
+-------
+
+Inquiry Options
+^^^^^^^^^^^^^^^
+
+Use these options when you want to know more about how to use `pgloader`, as
+those options will cause `pgloader` not to load any data.
+
+  * `-h`, `--help`
+
+    Show command usage summary and exit.
+
+  * `-V`, `--version`
+
+    Show pgloader version string and exit.
+
+  * `-E`, `--list-encodings`
+
+    List known encodings in this version of pgloader.
+
+  * `-U`, `--upgrade-config`
+    
+    Parse given files in the command line as `pgloader.conf` files with the
+    `INI` syntax that was in use in pgloader versions 2.x, and output the
+    new command syntax for pgloader on standard output.
+
+
+General Options
+^^^^^^^^^^^^^^^
+
+Those options are meant to tweak `pgloader` behavior when loading data.
+
+  * `-v`, `--verbose`
+    
+    Be verbose.
+
+  * `-q`, `--quiet`
+    
+    Be quiet.
+
+  * `-d`, `--debug`
+    
+    Show debug level information messages.
+
+  * `-D`, `--root-dir`
+    
+    Set the root working directory (default to "/tmp/pgloader").
+
+  * `-L`, `--logfile`
+    
+    Set the pgloader log file (default to "/tmp/pgloader.log").
+
+  * `--log-min-messages`
+    
+    Minimum level of verbosity needed for log message to make it to the
+    logfile. One of critical, log, error, warning, notice, info or debug.
+
+  * `--client-min-messages`
+    
+    Minimum level of verbosity needed for log message to make it to the
+    console. One of critical, log, error, warning, notice, info or debug.
+
+  * `-S`, `--summary`
+    
+    A filename where to copy the summary output. When relative, the filename
+    is expanded into `*root-dir*`.
+
+    The format of the filename defaults to being *human readable*. It is
+    possible to have the output in machine friendly formats such as *CSV*,
+    *COPY* (PostgreSQL's own COPY format) or *JSON* by specifying a filename
+    with the extension resp. `.csv`, `.copy` or `.json`.
+
+  * `-l <file>`, `--load-lisp-file <file>`
+    
+    Specify a lisp <file> to compile and load into the pgloader image before
+    reading the commands, allowing to define extra transformation function.
+    Those functions should be defined in the `pgloader.transforms` package.
+    This option can appear more than once in the command line.
+
+  * `--dry-run`
+
+    Allow testing a `.load` file without actually trying to load any data.
+    It's useful to debug it until it's ok, in particular to fix connection
+    strings.
+
+  * `--on-error-stop`
+
+    Alter pgloader behavior: rather than trying to be smart about error
+    handling and continue loading good data, separating away the bad one,
+    just stop as soon as PostgreSQL refuses anything sent to it. Useful to
+    debug data processing, transformation function and specific type
+    casting.
+
+  * `--self-upgrade <directory>`
+
+    Specify a <directory> where to find pgloader sources so that one of the
+    very first things it does is dynamically loading-in (and compiling to
+    machine code) another version of itself, usually a newer one like a very
+    recent git checkout.
+
+Command Line Only Operations
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Those options are meant to be used when using `pgloader` from the command
+line only, rather than using a command file and the rich command clauses and
+parser. In simple cases, it can be much easier to use the *SOURCE* and
+*TARGET* directly on the command line, then tweak the loading with those
+options:
+
+  * `--with "option"`
+
+    Allows setting options from the command line. You can use that option as
+    many times as you want. The option arguments must follow the *WITH*
+    clause for the source type of the `SOURCE` specification, as described
+    later in this document.
+
+  * `--set "guc_name='value'"`
+
+    Allows setting PostgreSQL configuration from the command line. Note that
+    the option parsing is the same as when used from the *SET* command
+    clause, in particular you must enclose the guc value with single-quotes.
+
+  * `--field "..."`
+
+    Allows setting a source field definition. Fields are accumulated in the
+    order given on the command line. It's possible to either use a `--field`
+    option per field in the source file, or to separate field definitions by
+    a comma, as you would do in the *HAVING FIELDS* clause.
+
+  * `--cast "..."`
+
+    Allows setting a specific casting rule for loading the data.
+
+  * `--type csv|fixed|db3|ixf|sqlite|mysql|mssql`
+
+    Allows forcing the source type, in case when the *SOURCE* parsing isn't
+    satisfying.
+
+  * `--encoding <encoding>`
+
+    Set the encoding of the source file to load data from.
+
+  * `--before <filename>`
+
+    Parse given filename for SQL queries and run them against the target
+    database before loading the data from the source. The queries are parsed
+    by pgloader itself: they need to be terminated by a semi-colon (;) and
+    the file may include `\i` or `\ir` commands to *include* another file.
+
+  * `--after <filename>`
+
+    Parse given filename for SQL queries and run them against the target
+    database after having loaded the data from the source. The queries are
+    parsed in the same way as with the `--before` option, see above.
+
+More Debug Information
+^^^^^^^^^^^^^^^^^^^^^^
+
+To get the maximum amount of debug information, you can use both the
+`--verbose` and the `--debug` switches at the same time, which is equivalent
+to saying `--client-min-messages data`. Then the log messages will show the
+data being processed, in the cases where the code has explicit support for
+it.
+
+Batches And Retry Behaviour
+---------------------------
+
+To load data to PostgreSQL, pgloader uses the `COPY` streaming protocol.
+While this is the faster way to load data, `COPY` has an important drawback:
+as soon as PostgreSQL emits an error with any bit of data sent to it,
+whatever the problem is, the whole data set is rejected by PostgreSQL.
+
+To work around that, pgloader cuts the data into *batches* of 25000 rows
+each, so that when a problem occurs it's only impacting that many rows of
+data. Each batch is kept in memory while the `COPY` streaming happens, in
+order to be able to handle errors should some happen.
+
+When PostgreSQL rejects the whole batch, pgloader logs the error message
+then isolates the bad row(s) from the accepted ones by retrying the batched
+rows in smaller batches. To do that, pgloader parses the *CONTEXT* error
+message from the failed COPY, as the message contains the line number where
+the error was found in the batch, as in the following example::
+
+    CONTEXT: COPY errors, line 3, column b: "2006-13-11"
+
+Using that information, pgloader will reload all rows in the batch before
+the erroneous one, log the erroneous one as rejected, then try loading the
+remaining of the batch in a single attempt, which may or may not contain
+other erroneous data.
+
+At the end of a load containing rejected rows, you will find two files in
+the *root-dir* location, under a directory named the same as the target
+database of your setup. The filenames are the target table, and their
+extensions are `.dat` for the rejected data and `.log` for the file
+containing the full PostgreSQL client side logs about the rejected data.
+
+The `.dat` file is formatted in PostgreSQL the text COPY format as documented
+in `http://www.postgresql.org/docs/9.2/static/sql-copy.html#AEN66609`.
+
+A Note About Performance
+------------------------
+
+pgloader has been developed with performance in mind, to be able to cope
+with ever growing needs in loading large amounts of data into PostgreSQL.
+
+The basic architecture it uses is the old Unix pipe model, where a thread is
+responsible for loading the data (reading a CSV file, querying MySQL, etc)
+and fills pre-processed data into a queue. Another threads feeds from the
+queue, apply some more *transformations* to the input data and stream the
+end result to PostgreSQL using the COPY protocol.
+
+When given a file that the PostgreSQL `COPY` command knows how to parse, and
+if the file contains no erroneous data, then pgloader will never be as fast
+as just using the PostgreSQL `COPY` command.
+
+Note that while the `COPY` command is restricted to read either from its
+standard input or from a local file on the server's file system, the command
+line tool `psql` implements a `\copy` command that knows how to stream a
+file local to the client over the network and into the PostgreSQL server,
+using the same protocol as pgloader uses.
+
+A Note About Parallelism
+------------------------
+
+pgloader uses several concurrent tasks to process the data being loaded:
+
+  - a reader task reads the data in and pushes it to a queue,
+  
+  - at last one write task feeds from the queue and formats the raw into the
+    PostgreSQL COPY format in batches (so that it's possible to then retry a
+    failed batch without reading the data from source again), and then sends
+    the data to PostgreSQL using the COPY protocol.
+
+The parameter *workers* allows to control how many worker threads are
+allowed to be active at any time (that's the parallelism level); and the
+parameter *concurrency* allows to control how many tasks are started to
+handle the data (they may not all run at the same time, depending on the
+*workers* setting).
+
+We allow *workers* simultaneous workers to be active at the same time in the
+context of a single table. A single unit of work consist of several kinds of
+workers:
+
+  - a reader getting raw data from the source,
+  - N writers preparing and sending the data down to PostgreSQL.
+
+The N here is setup to the *concurrency* parameter: with a *CONCURRENCY* of
+2, we start (+ 1 2) = 3 concurrent tasks, with a *concurrency* of 4 we start
+(+ 1 4) = 9 concurrent tasks, of which only *workers* may be active
+simultaneously.
+
+The defaults are `workers = 4, concurrency = 1` when loading from a database
+source, and `workers = 8, concurrency = 2` when loading from something else
+(currently, a file). Those defaults are arbitrary and waiting for feedback
+from users, so please consider providing feedback if you play with the
+settings.
+
+As the `CREATE INDEX` threads started by pgloader are only waiting until
+PostgreSQL is done with the real work, those threads are *NOT* counted into
+the concurrency levels as detailed here.
+
+By default, as many `CREATE INDEX` threads as the maximum number of indexes
+per table are found in your source schema. It is possible to set the `max
+parallel create index` *WITH* option to another number in case there's just
+too many of them to create.
+
+Source Formats
+--------------
+
+pgloader supports the following input formats:
+
+  - csv, which includes also tsv and other common variants where you can
+    change the *separator* and the *quoting* rules and how to *escape* the
+    *quotes* themselves;
+
+  - fixed columns file, where pgloader is flexible enough to accomodate with
+    source files missing columns (*ragged fixed length column files* do
+    exist);
+
+  - PostgreSLQ COPY formatted files, following the COPY TEXT documentation
+    of PostgreSQL, such as the reject files prepared by pgloader;
+
+  - dbase files known as db3 or dbf file;
+
+  - ixf formated files, ixf being a binary storage format from IBM;
+
+  - sqlite databases with fully automated discovery of the schema and
+    advanced cast rules;
+
+  - mysql databases with fully automated discovery of the schema and
+    advanced cast rules;
+
+  - MS SQL databases with fully automated discovery of the schema and
+    advanced cast rules.
+
+Pgloader Commands Syntax
+------------------------
+
+pgloader implements a Domain Specific Language allowing to setup complex
+data loading scripts handling computed columns and on-the-fly sanitization
+of the input data. For more complex data loading scenarios, you will be
+required to learn that DSL's syntax. It's meant to look familiar to DBA by
+being inspired by SQL where it makes sense, which is not that much after
+all.
+
+The pgloader commands follow the same global grammar rules. Each of them
+might support only a subset of the general options and provide specific
+options.
+
+::
+
+    LOAD <source-type>
+	     FROM <source-url>
+           [ HAVING FIELDS <source-level-options> ]
+		 INTO <postgresql-url>
+           [ TARGET TABLE [ "<schema>" ]."<table name>" ]
+           [ TARGET COLUMNS <columns-and-options> ]
+
+	[ WITH <load-options> ]
+
+	[ SET <postgresql-settings> ]
+
+    [ BEFORE LOAD [ DO <sql statements> | EXECUTE <sql file> ] ... ]
+    [  AFTER LOAD [ DO <sql statements> | EXECUTE <sql file> ] ... ]
+	;
+
+The main clauses are the `LOAD`, `FROM`, `INTO` and `WITH` clauses that each
+command implements. Some command then implement the `SET` command, or some
+specific clauses such as the `CAST` clause.
+
+Templating with Mustache
+------------------------
+
+pgloader implements the https://mustache.github.io/ templating system so
+that you may have dynamic parts of your commands. See the documentation for
+this template system online.
+
+A specific feature of pgloader is the ability to fetch a variable from the
+OS environment of the pgloader process, making it possible to run pgloader
+as in the following example::
+
+    $ DBPATH=sqlite/sqlite.db pgloader ./test/sqlite-env.load
+    
+or in several steps::
+
+    $ export DBPATH=sqlite/sqlite.db
+    $ pgloader ./test/sqlite-env.load
+
+The variable can then be used in a typical mustache fashion::
+
+    load database
+         from '{{DBPATH}}'
+         into postgresql:///pgloader;
+
+It's also possible to prepare a INI file such as the following::
+
+    [pgloader]
+    
+    DBPATH = sqlite/sqlite.db
+
+And run the following command, feeding the INI values as a *context* for
+pgloader templating system::
+
+    $ pgloader --context ./test/sqlite.ini ./test/sqlite-ini.load
+
+The mustache templates implementation with OS environment support replaces
+former `GETENV` implementation, which didn't work anyway.
+
+Common Clauses
+--------------
+
+Some clauses are common to all commands:
+
+FROM
+^^^^
+
+The *FROM* clause specifies where to read the data from, and each command
+introduces its own variant of sources. For instance, the *CSV* source
+supports `inline`, `stdin`, a filename, a quoted filename, and a *FILENAME
+MATCHING* clause (see above); whereas the *MySQL* source only supports a
+MySQL database URI specification.
+
+INTO
+^^^^
+
+The PostgreSQL connection URI must contains the name of the target table
+where to load the data into. That table must have already been created in
+PostgreSQL, and the name might be schema qualified.
+
+Then *INTO* option also supports an optional comma separated list of target
+columns, which are either the name of an input *field* or the white space
+separated list of the target column name, its PostgreSQL data type and a
+*USING* expression.
+
+The *USING* expression can be any valid Common Lisp form and will be read
+with the current package set to `pgloader.transforms`, so that you can use
+functions defined in that package, such as functions loaded dynamically with
+the `--load` command line parameter.
+
+Each *USING* expression is compiled at runtime to native code.
+
+This feature allows pgloader to load any number of fields in a CSV file into
+a possibly different number of columns in the database, using custom code
+for that projection.
+
+WITH
+^^^^
+
+Set of options to apply to the command, using a global syntax of either:
+
+   - *key = value*
+   - *use option*
+   - *do not use option*
+
+See each specific command for details.
+
+All data sources specific commands support the following options:
+
+  - *on error stop*
+  - *batch rows = R*
+  - *batch size = ... MB*
+  - *prefetch rows = ...*
+
+See the section BATCH BEHAVIOUR OPTIONS for more details.
+
+In addition, the following settings are available:
+
+   - *workers = W*
+   - *concurrency = C*
+   - *max parallel create index = I*
+
+See section A NOTE ABOUT PARALLELISM for more details.
+
+SET
+^^^
+
+This clause allows to specify session parameters to be set for all the
+sessions opened by pgloader. It expects a list of parameter name, the equal
+sign, then the single-quoted value as a comma separated list.
+
+The names and values of the parameters are not validated by pgloader, they
+are given as-is to PostgreSQL.
+
+BEFORE LOAD DO
+^^^^^^^^^^^^^^
+
+You can run SQL queries against the database before loading the data from
+the `CSV` file. Most common SQL queries are `CREATE TABLE IF NOT EXISTS` so
+that the data can be loaded.
+
+Each command must be *dollar-quoted*: it must begin and end with a double
+dollar sign, `$$`. Dollar-quoted queries are then comma separated. No extra
+punctuation is expected after the last SQL query.
+
+BEFORE LOAD EXECUTE
+^^^^^^^^^^^^^^^^^^^
+
+Same behaviour as in the *BEFORE LOAD DO* clause. Allows you to read the SQL
+queries from a SQL file. Implements support for PostgreSQL dollar-quoting
+and the `\i` and `\ir` include facilities as in `psql` batch mode (where
+they are the same thing).
+
+AFTER LOAD DO
+^^^^^^^^^^^^^
+
+Same format as *BEFORE LOAD DO*, the dollar-quoted queries found in that
+section are executed once the load is done. That's the right time to create
+indexes and constraints, or re-enable triggers.
+
+AFTER LOAD EXECUTE
+^^^^^^^^^^^^^^^^^^
+
+Same behaviour as in the *AFTER LOAD DO* clause. Allows you to read the SQL
+queries from a SQL file. Implements support for PostgreSQL dollar-quoting
+and the `\i` and `\ir` include facilities as in `psql` batch mode (where
+they are the same thing).
+
+Connection String
+^^^^^^^^^^^^^^^^^
+
+The `<postgresql-url>` parameter is expected to be given as a *Connection URI*
+as documented in the PostgreSQL documentation at
+http://www.postgresql.org/docs/9.3/static/libpq-connect.html#LIBPQ-CONNSTRING.
+
+::
+
+    postgresql://[user[:password]@][netloc][:port][/dbname][?option=value&...]
+
+Where:
+
+  - *user*
+
+    Can contain any character, including colon (`:`) which must then be
+    doubled (`::`) and at-sign (`@`) which must then be doubled (`@@`).
+
+    When omitted, the *user* name defaults to the value of the `PGUSER`
+    environment variable, and if it is unset, the value of the `USER`
+    environment variable.
+
+  - *password*
+
+	Can contain any character, including the at sign (`@`) which must then
+	be doubled (`@@`). To leave the password empty, when the *user* name
+	ends with at at sign, you then have to use the syntax user:@.
+
+    When omitted, the *password* defaults to the value of the `PGPASSWORD`
+    environment variable if it is set, otherwise the password is left
+    unset.
+    
+    When no *password* is found either in the connection URI nor in the
+    environment, then pgloader looks for a `.pgpass` file as documented at
+    https://www.postgresql.org/docs/current/static/libpq-pgpass.html. The
+    implementation is not that of `libpq` though. As with `libpq` you can
+    set the environment variable `PGPASSFILE` to point to a `.pgpass` file,
+    and pgloader defaults to `~/.pgpass` on unix like systems and
+    `%APPDATA%\postgresql\pgpass.conf` on windows. Matching rules and syntax
+    are the same as with `libpq`, refer to its documentation.
+
+  - *netloc*
+
+    Can be either a hostname in dotted notation, or an ipv4, or an Unix
+    domain socket path. Empty is the default network location, under a
+    system providing *unix domain socket* that method is preferred, otherwise
+    the *netloc* default to `localhost`.
+
+	It's possible to force the *unix domain socket* path by using the syntax
+	`unix:/path/to/where/the/socket/file/is`, so to force a non default
+	socket path and a non default port, you would have:
+
+	    postgresql://unix:/tmp:54321/dbname
+
+    The *netloc* defaults to the value of the `PGHOST` environment
+    variable, and if it is unset, to either the default `unix` socket path
+    when running on a Unix system, and `localhost` otherwise.
+
+    Socket path containing colons are supported by doubling the colons
+    within the path, as in the following example:
+    
+        postgresql://unix:/tmp/project::region::instance:5432/dbname
+
+  - *dbname*
+
+	Should be a proper identifier (letter followed by a mix of letters,
+	digits and the punctuation signs comma (`,`), dash (`-`) and underscore
+	(`_`).
+
+    When omitted, the *dbname* defaults to the value of the environment
+    variable `PGDATABASE`, and if that is unset, to the *user* value as
+    determined above.
+
+  - *options*
+
+    The optional parameters must be supplied with the form `name=value`, and
+    you may use several parameters by separating them away using an
+    ampersand (`&`) character.
+
+    Only some options are supported here, *tablename* (which might be
+    qualified with a schema name) *sslmode*, *host*, *port*, *dbname*,
+    *user* and *password*.
+
+    The *sslmode* parameter values can be one of `disable`, `allow`,
+    `prefer` or `require`.
+
+    For backward compatibility reasons, it's possible to specify the
+    *tablename* option directly, without spelling out the `tablename=`
+    parts.
+
+    The options override the main URI components when both are given, and
+    using the percent-encoded option parameters allow using passwords
+    starting with a colon and bypassing other URI components parsing
+    limitations.
+
+Regular Expressions
+^^^^^^^^^^^^^^^^^^^
+
+Several clauses listed in the following accept *regular expressions* with
+the following input rules:
+
+  - A regular expression begins with a tilde sign (`~`),
+
+  - is then followed with an opening sign,
+
+  - then any character is allowed and considered part of the regular
+    expression, except for the closing sign,
+
+  - then a closing sign is expected.
+
+The opening and closing sign are allowed by pair, here's the complete list
+of allowed delimiters::
+
+    ~//
+    ~[]
+    ~{}
+    ~()
+    ~<>
+    ~""
+    ~''
+    ~||
+    ~##
+
+Pick the set of delimiters that don't collide with the *regular expression*
+you're trying to input. If your expression is such that none of the
+solutions allow you to enter it, the places where such expressions are
+allowed should allow for a list of expressions.
+
+Comments
+^^^^^^^^
+
+Any command may contain comments, following those input rules:
+
+  - the `--` delimiter begins a comment that ends with the end of the
+    current line,
+
+  - the delimiters `/*` and `*/` respectively start and end a comment, which
+    can be found in the middle of a command or span several lines.
+
+Any place where you could enter a *whitespace* will accept a comment too.
+
+Batch behaviour options
+^^^^^^^^^^^^^^^^^^^^^^^
+
+All pgloader commands have support for a *WITH* clause that allows for
+specifying options. Some options are generic and accepted by all commands,
+such as the *batch behaviour options*, and some options are specific to a
+data source kind, such as the CSV *skip header* option.
+
+The global batch behaviour options are:
+
+  - *batch rows*
+
+    Takes a numeric value as argument, used as the maximum number of rows
+    allowed in a batch. The default is `25 000` and can be changed to try
+    having better performance characteristics or to control pgloader memory
+    usage;
+
+  - *batch size*
+
+    Takes a memory unit as argument, such as *20 MB*, its default value.
+    Accepted multipliers are *kB*, *MB*, *GB*, *TB* and *PB*. The case is
+    important so as not to be confused about bits versus bytes, we're only
+    talking bytes here.
+
+  - *prefetch rows*
+
+    Takes a numeric value as argument, defaults to `100000`. That's the
+    number of rows that pgloader is allowed to read in memory in each reader
+    thread. See the *workers* setting for how many reader threads are
+    allowed to run at the same time.
+
+Other options are specific to each input source, please refer to specific
+parts of the documentation for their listing and covering.
+
+A batch is then closed as soon as either the *batch rows* or the *batch
+size* threshold is crossed, whichever comes first. In cases when a batch has
+to be closed because of the *batch size* setting, a *debug* level log
+message is printed with how many rows did fit in the *oversized* batch.
+

docs/_build/html/_sources/ref/archive.rst.txt

@@ -0,0 +1,104 @@
+Loading From an Archive
+=======================
+
+This command instructs pgloader to load data from one or more files contained
+in an archive. Currently the only supported archive format is *ZIP*, and the
+archive might be downloaded from an *HTTP* URL.
+
+Here's an example::
+
+    LOAD ARCHIVE
+       FROM /Users/dim/Downloads/GeoLiteCity-latest.zip
+       INTO postgresql:///ip4r
+
+       BEFORE LOAD
+         DO $$ create extension if not exists ip4r; $$,
+            $$ create schema if not exists geolite; $$,
+
+         EXECUTE 'geolite.sql'
+
+       LOAD CSV
+            FROM FILENAME MATCHING ~/GeoLiteCity-Location.csv/
+                 WITH ENCODING iso-8859-1
+                 (
+                    locId,
+                    country,
+                    region     null if blanks,
+                    city       null if blanks,
+                    postalCode null if blanks,
+                    latitude,
+                    longitude,
+                    metroCode  null if blanks,
+                    areaCode   null if blanks
+                 )
+            INTO postgresql:///ip4r?geolite.location
+                 (
+                    locid,country,region,city,postalCode,
+                    location point using (format nil "(~a,~a)" longitude latitude),
+                    metroCode,areaCode
+                 )
+            WITH skip header = 2,
+                 fields optionally enclosed by '"',
+                 fields escaped by double-quote,
+                 fields terminated by ','
+
+      AND LOAD CSV
+            FROM FILENAME MATCHING ~/GeoLiteCity-Blocks.csv/
+                 WITH ENCODING iso-8859-1
+                 (
+                    startIpNum, endIpNum, locId
+                 )
+            INTO postgresql:///ip4r?geolite.blocks
+                 (
+                    iprange ip4r using (ip-range startIpNum endIpNum),
+                    locId
+                 )
+            WITH skip header = 2,
+                 fields optionally enclosed by '"',
+                 fields escaped by double-quote,
+                 fields terminated by ','
+
+       FINALLY DO
+         $$ create index blocks_ip4r_idx on geolite.blocks using gist(iprange); $$;
+
+The `archive` command accepts the following clauses and options.
+
+Archive Source Specification: FROM
+----------------------------------
+
+Filename or HTTP URI where to load the data from. When given an HTTP URL the
+linked file will get downloaded locally before processing.
+
+If the file is a `zip` file, the command line utility `unzip` is used to
+expand the archive into files in `$TMPDIR`, or `/tmp` if `$TMPDIR` is unset
+or set to a non-existing directory.
+
+Then the following commands are used from the top level directory where the
+archive has been expanded.
+
+Archive Sub Commands
+--------------------
+
+  - command [ *AND* command ... ]
+
+    A series of commands against the contents of the archive, at the moment
+    only `CSV`,`'FIXED` and `DBF` commands are supported.
+
+    Note that commands are supporting the clause *FROM FILENAME MATCHING*
+    which allows the pgloader command not to depend on the exact names of
+    the archive directories.
+
+    The same clause can also be applied to several files with using the
+    spelling *FROM ALL FILENAMES MATCHING* and a regular expression.
+
+    The whole *matching* clause must follow the following rule::
+
+      FROM [ ALL FILENAMES | [ FIRST ] FILENAME ] MATCHING
+
+Archive Final SQL Commands
+--------------------------
+      
+  - *FINALLY DO*
+
+    SQL Queries to run once the data is loaded, such as `CREATE INDEX`.
+

docs/_build/html/_sources/ref/copy.rst.txt

@@ -0,0 +1,115 @@
+Loading COPY Formatted Files
+============================
+
+This commands instructs pgloader to load from a file containing COPY TEXT
+data as described in the PostgreSQL documentation. Here's an example::
+
+    LOAD COPY
+         FROM copy://./data/track.copy
+              (
+                trackid, track, album, media, genre, composer,
+                milliseconds, bytes, unitprice
+              )
+         INTO postgresql:///pgloader
+       TARGET TABLE track_full
+
+         WITH truncate
+
+          SET work_mem to '14MB',
+              standard_conforming_strings to 'on'
+
+    BEFORE LOAD DO
+         $$ drop table if exists track_full; $$,
+         $$ create table track_full (
+              trackid      bigserial,
+              track        text,
+              album        text,
+              media        text,
+              genre        text,
+              composer     text,
+              milliseconds bigint,
+              bytes        bigint,
+              unitprice    numeric
+            );
+         $$;
+
+The `COPY` format command accepts the following clauses and options.
+
+COPY Formatted Files Source Specification: FROM
+-----------------------------------------------
+
+Filename where to load the data from. This support local files, HTTP URLs
+and zip files containing a single dbf file of the same name. Fetch such a
+zip file from an HTTP address is of course supported.
+
+  - *inline*
+
+    The data is found after the end of the parsed commands. Any number of
+    empty lines between the end of the commands and the beginning of the
+    data is accepted.
+
+  - *stdin*
+
+    Reads the data from the standard input stream.
+
+  - *FILENAMES MATCHING*
+
+    The whole *matching* clause must follow the following rule::
+
+      [ ALL FILENAMES | [ FIRST ] FILENAME ]
+      MATCHING regexp
+      [ IN DIRECTORY '...' ]
+
+    The *matching* clause applies given *regular expression* (see above for
+    exact syntax, several options can be used here) to filenames. It's then
+    possible to load data from only the first match of all of them.
+
+    The optional *IN DIRECTORY* clause allows specifying which directory to
+    walk for finding the data files, and can be either relative to where the
+    command file is read from, or absolute. The given directory must exists.
+
+COPY Formatted File Options: WITH
+---------------------------------
+
+
+When loading from a `COPY` file, the following options are supported:
+
+  - *delimiter*
+
+    Takes a single character as argument, which must be found inside single
+    quotes, and might be given as the printable character itself, the
+    special value \t to denote a tabulation character, or `0x` then an
+    hexadecimal value read as the ASCII code for the character.
+
+    This character is used as the *delimiter* when reading the data, in a
+    similar way to the PostgreSQL `COPY` option.
+
+  - *null*
+
+    Takes a quoted string as an argument (quotes can be either double quotes
+    or single quotes) and uses that string as the `NULL` representation in
+    the data.
+
+    This is similar to the *null* `COPY` option in PostgreSQL.
+
+  - *truncate*
+
+    When this option is listed, pgloader issues a `TRUNCATE` command against
+    the PostgreSQL target table before reading the data file.
+
+  - *disable triggers*
+
+    When this option is listed, pgloader issues an `ALTER TABLE ... DISABLE
+    TRIGGER ALL` command against the PostgreSQL target table before copying
+    the data, then the command `ALTER TABLE ... ENABLE TRIGGER ALL` once the
+    `COPY` is done.
+
+    This option allows loading data into a pre-existing table ignoring the
+    *foreign key constraints* and user defined triggers and may result in
+    invalid *foreign key constraints* once the data is loaded. Use with
+    care.
+
+  - *skip header*
+
+    Takes a numeric value as argument. Instruct pgloader to skip that many
+    lines at the beginning of the input file.

docs/_build/html/_sources/ref/csv.rst.txt

@@ -0,0 +1,237 @@
+Loading CSV data
+================
+
+This command instructs pgloader to load data from a `CSV` file. Here's an
+example::
+
+    LOAD CSV
+       FROM 'GeoLiteCity-Blocks.csv' WITH ENCODING iso-646-us
+            HAVING FIELDS
+            (
+               startIpNum, endIpNum, locId
+            )
+       INTO postgresql://user@localhost:54393/dbname
+            TARGET TABLE geolite.blocks
+            TARGET COLUMNS
+            (
+               iprange ip4r using (ip-range startIpNum endIpNum),
+               locId
+            )
+       WITH truncate,
+            skip header = 2,
+            fields optionally enclosed by '"',
+            fields escaped by backslash-quote,
+            fields terminated by '\t'
+
+        SET work_mem to '32 MB', maintenance_work_mem to '64 MB';
+
+The `csv` format command accepts the following clauses and options.
+
+CSV Source Specification: FROM
+------------------------------
+
+Filename where to load the data from. Accepts an *ENCODING* option. Use the
+`--list-encodings` option to know which encoding names are supported.
+
+The filename may be enclosed by single quotes, and could be one of the
+following special values:
+
+  - *inline*
+
+    The data is found after the end of the parsed commands. Any number
+    of empty lines between the end of the commands and the beginning of
+    the data is accepted.
+
+  - *stdin*
+
+    Reads the data from the standard input stream.
+
+  - *FILENAMES MATCHING*
+
+    The whole *matching* clause must follow the following rule::
+
+            [ ALL FILENAMES | [ FIRST ] FILENAME ]
+            MATCHING regexp
+            [ IN DIRECTORY '...' ]
+
+    The *matching* clause applies given *regular expression* (see above
+    for exact syntax, several options can be used here) to filenames.
+    It's then possible to load data from only the first match of all of
+    them.
+
+    The optional *IN DIRECTORY* clause allows specifying which directory
+    to walk for finding the data files, and can be either relative to
+    where the command file is read from, or absolute. The given
+    directory must exists.
+
+Fields Specifications
+---------------------
+    
+The *FROM* option also supports an optional comma separated list of *field*
+names describing what is expected in the `CSV` data file, optionally
+introduced by the clause `HAVING FIELDS`.
+
+Each field name can be either only one name or a name following with
+specific reader options for that field, enclosed in square brackets and
+comma-separated. Supported per-field reader options are:
+
+  - *terminated by*
+
+     See the description of *field terminated by* below.
+
+     The processing of this option is not currently implemented.
+
+  - *date format*
+
+    When the field is expected of the date type, then this option allows
+    to specify the date format used in the file.
+
+    Date format string are template strings modeled against the
+    PostgreSQL `to_char` template strings support, limited to the
+    following patterns:
+
+      - YYYY, YYY, YY for the year part
+      - MM for the numeric month part
+      - DD for the numeric day part
+      - HH, HH12, HH24 for the hour part
+      - am, AM, a.m., A.M.
+      - pm, PM, p.m., P.M.
+      - MI for the minutes part
+      - SS for the seconds part
+      - MS for the milliseconds part (4 digits)
+      - US for the microseconds part (6 digits)
+      - unparsed punctuation signs: - . * # @ T / \ and space
+
+    Here's an example of a *date format* specification::
+
+        column-name [date format 'YYYY-MM-DD HH24-MI-SS.US']
+
+  - *null if*
+
+     This option takes an argument which is either the keyword *blanks*
+     or a double-quoted string.
+
+     When *blanks* is used and the field value that is read contains
+     only space characters, then it's automatically converted to an SQL
+     `NULL` value.
+
+     When a double-quoted string is used and that string is read as the
+     field value, then the field value is automatically converted to an
+     SQL `NULL` value.
+
+  - *trim both whitespace*, *trim left whitespace*, *trim right whitespace*
+
+    This option allows to trim whitespaces in the read data, either from
+    both sides of the data, or only the whitespace characters found on
+    the left of the streaing, or only those on the right of the string.
+
+CSV Loading Options: WITH
+-------------------------
+
+When loading from a `CSV` file, the following options are supported:
+
+  - *truncate*
+
+     When this option is listed, pgloader issues a `TRUNCATE` command
+     against the PostgreSQL target table before reading the data file.
+
+  - *drop indexes*
+
+    When this option is listed, pgloader issues `DROP INDEX` commands
+    against all the indexes defined on the target table before copying
+    the data, then `CREATE INDEX` commands once the `COPY` is done.
+
+    In order to get the best performance possible, all the indexes are
+    created in parallel and when done the primary keys are built again
+    from the unique indexes just created. This two step process allows
+    creating the primary key index in parallel with the other indexes,
+    as only the `ALTER TABLE` command needs an *access exclusive lock*
+    on the target table.
+
+  - *disable triggers*
+
+    When this option is listed, pgloader issues an `ALTER TABLE ...
+    DISABLE TRIGGER ALL` command against the PostgreSQL target table
+    before copying the data, then the command `ALTER TABLE ... ENABLE
+    TRIGGER ALL` once the `COPY` is done.
+
+    This option allows loading data into a pre-existing table ignoring
+    the *foreign key constraints* and user defined triggers and may
+    result in invalid *foreign key constraints* once the data is loaded.
+    Use with care.
+
+  - *skip header*
+
+    Takes a numeric value as argument. Instruct pgloader to skip that
+    many lines at the beginning of the input file.
+
+  - *csv header*
+
+    Use the first line read after *skip header* as the list of csv field
+    names to be found in the CSV file, using the same CSV parameters as
+    for the CSV data.
+
+  - *trim unquoted blanks*
+
+    When reading unquoted values in the `CSV` file, remove the blanks
+    found in between the separator and the value. That behaviour is the
+    default.
+
+  - *keep unquoted blanks*
+
+    When reading unquoted values in the `CSV` file, keep blanks found in
+    between the separator and the value.
+
+  - *fields optionally enclosed by*
+
+    Takes a single character as argument, which must be found inside
+    single quotes, and might be given as the printable character itself,
+    the special value \t to denote a tabulation character, or `0x` then
+    an hexadecimal value read as the ASCII code for the character.
+
+    This character is used as the quoting character in the `CSV` file,
+    and defaults to double-quote.
+
+  - *fields not enclosed*
+
+    By default, pgloader will use the double-quote character as the
+    enclosing character. If you have a CSV file where fields are not
+    enclosed and are using double-quote as an expected ordinary
+    character, then use the option *fields not enclosed* for the CSV
+    parser to accept those values.
+
+  - *fields escaped by*
+
+    Takes either the special value *backslash-quote* or *double-quote*,
+    or any value supported by the *fields terminated by* option (see
+    below). This value is used to recognize escaped field separators
+    when they are to be found within the data fields themselves.
+    Defaults to *double-quote*.
+
+  - *csv escape mode*
+
+    Takes either the special value *quote* (the default) or *following*
+    and allows the CSV parser to parse either only escaped field
+    separator or any character (including CSV data) when using the
+    *following* value.
+
+  - *fields terminated by*
+
+    Takes a single character as argument, which must be found inside
+    single quotes, and might be given as the printable character itself,
+    the special value \t to denote a tabulation character, or `0x` then
+    an hexadecimal value read as the ASCII code for the character.
+
+    This character is used as the *field separator* when reading the
+    `CSV` data.
+
+  - *lines terminated by*
+
+    Takes a single character as argument, which must be found inside
+    single quotes, and might be given as the printable character itself,
+    the special value \t to denote a tabulation character, or `0x` then
+    an hexadecimal value read as the ASCII code for the character.
+
+    This character is used to recognize *end-of-line* condition when
+    reading the `CSV` data.
+

docs/_build/html/_sources/ref/dbf.rst.txt

@@ -0,0 +1,53 @@
+Loading DBF data
+=================
+
+This command instructs pgloader to load data from a `DBF` file. Here's an
+example::
+
+    LOAD DBF
+	    FROM http://www.insee.fr/fr/methodes/nomenclatures/cog/telechargement/2013/dbf/reg2013.dbf
+        INTO postgresql://user@localhost/dbname
+        WITH truncate, create table;
+
+The `dbf` format command accepts the following clauses and options.
+
+DBF Source Specification: FROM
+------------------------------
+
+Filename where to load the data from. This support local files, HTTP URLs
+and zip files containing a single dbf file of the same name. Fetch such a
+zip file from an HTTP address is of course supported.
+
+DBF Loading Options: WITH
+-------------------------
+
+When loading from a `DBF` file, the following options are supported:
+
+  - *truncate*
+
+    When this option is listed, pgloader issues a `TRUNCATE` command against
+    the PostgreSQL target table before reading the data file.
+
+  - *disable triggers*
+
+    When this option is listed, pgloader issues an `ALTER TABLE ... DISABLE
+    TRIGGER ALL` command against the PostgreSQL target table before copying
+    the data, then the command `ALTER TABLE ... ENABLE TRIGGER ALL` once the
+    `COPY` is done.
+
+    This option allows loading data into a pre-existing table ignoring the
+    *foreign key constraints* and user defined triggers and may result in
+    invalid *foreign key constraints* once the data is loaded. Use with
+    care.
+
+  - *create table*
+
+    When this option is listed, pgloader creates the table using the meta
+    data found in the `DBF` file, which must contain a list of fields with
+    their data type. A standard data type conversion from DBF to PostgreSQL
+    is done.
+
+  - *table name*
+
+    This options expects as its value the possibly qualified name of the
+    table to create.

docs/_build/html/_sources/ref/fixed.rst.txt

@@ -0,0 +1,182 @@
+Loading Fixed Cols File Formats
+===============================
+
+This command instructs pgloader to load data from a text file containing
+columns arranged in a *fixed size* manner. Here's an example::
+
+    LOAD FIXED
+         FROM inline
+              (
+               a from  0 for 10,
+               b from 10 for  8,
+               c from 18 for  8,
+               d from 26 for 17 [null if blanks, trim right whitespace]
+              )
+         INTO postgresql:///pgloader
+       TARGET TABLE fixed
+              (
+                 a, b,
+                 c time using (time-with-no-separator c),
+                 d
+              )
+
+         WITH truncate
+
+          SET work_mem to '14MB',
+              standard_conforming_strings to 'on'
+
+    BEFORE LOAD DO
+         $$ drop table if exists fixed; $$,
+         $$ create table fixed (
+             a integer,
+             b date,
+             c time,
+             d text
+            );
+         $$;
+
+     01234567892008052011431250firstline
+        01234562008052115182300left blank-padded
+     12345678902008052208231560another line
+      2345609872014092914371500                 
+      2345678902014092914371520
+
+The `fixed` format command accepts the following clauses and options.
+
+Fixed File Format Source Specification: FROM
+--------------------------------------------
+
+Filename where to load the data from. Accepts an *ENCODING* option. Use the
+`--list-encodings` option to know which encoding names are supported.
+
+The filename may be enclosed by single quotes, and could be one of the
+following special values:
+
+  - *inline*
+
+     The data is found after the end of the parsed commands. Any number
+     of empty lines between the end of the commands and the beginning of
+     the data is accepted.
+
+  - *stdin*
+
+     Reads the data from the standard input stream.
+
+  - *FILENAMES MATCHING*
+
+    The whole *matching* clause must follow the following rule::
+
+        [ ALL FILENAMES | [ FIRST ] FILENAME ]
+        MATCHING regexp
+        [ IN DIRECTORY '...' ]
+
+    The *matching* clause applies given *regular expression* (see above
+    for exact syntax, several options can be used here) to filenames.
+    It's then possible to load data from only the first match of all of
+    them.
+
+    The optional *IN DIRECTORY* clause allows specifying which directory
+    to walk for finding the data files, and can be either relative to
+    where the command file is read from, or absolute. The given
+    directory must exists.
+
+Fields Specifications
+---------------------
+
+The *FROM* option also supports an optional comma separated list of *field*
+names describing what is expected in the `FIXED` data file.
+
+Each field name is composed of the field name followed with specific reader
+options for that field. Supported per-field reader options are the
+following, where only *start* and *length* are required.
+
+  - *start*
+
+    Position in the line where to start reading that field's value. Can
+    be entered with decimal digits or `0x` then hexadecimal digits.
+
+  - *length*
+
+    How many bytes to read from the *start* position to read that
+    field's value. Same format as *start*.
+
+Those optional parameters must be enclosed in square brackets and
+comma-separated:
+
+  - *terminated by*
+
+     See the description of *field terminated by* below.
+
+     The processing of this option is not currently implemented.
+
+  - *date format*
+
+    When the field is expected of the date type, then this option allows
+    to specify the date format used in the file.
+
+    Date format string are template strings modeled against the
+    PostgreSQL `to_char` template strings support, limited to the
+    following patterns:
+
+      - YYYY, YYY, YY for the year part
+      - MM for the numeric month part
+      - DD for the numeric day part
+      - HH, HH12, HH24 for the hour part
+      - am, AM, a.m., A.M.
+      - pm, PM, p.m., P.M.
+      - MI for the minutes part
+      - SS for the seconds part
+      - MS for the milliseconds part (4 digits)
+      - US for the microseconds part (6 digits)
+      - unparsed punctuation signs: - . * # @ T / \ and space
+
+    Here's an example of a *date format* specification::
+
+        column-name [date format 'YYYY-MM-DD HH24-MI-SS.US']
+
+  - *null if*
+
+    This option takes an argument which is either the keyword *blanks*
+    or a double-quoted string.
+
+    When *blanks* is used and the field value that is read contains only
+    space characters, then it's automatically converted to an SQL `NULL`
+    value.
+
+    When a double-quoted string is used and that string is read as the
+    field value, then the field value is automatically converted to an
+    SQL `NULL` value.
+
+  - *trim both whitespace*, *trim left whitespace*, *trim right whitespace*
+
+    This option allows to trim whitespaces in the read data, either from
+    both sides of the data, or only the whitespace characters found on
+    the left of the streaing, or only those on the right of the string.
+
+Fixed File Format Loading Options: WITH
+---------------------------------------
+
+When loading from a `FIXED` file, the following options are supported:
+
+  - *truncate*
+
+    When this option is listed, pgloader issues a `TRUNCATE` command
+    against the PostgreSQL target table before reading the data file.
+
+  - *disable triggers*
+
+    When this option is listed, pgloader issues an `ALTER TABLE ...
+    DISABLE TRIGGER ALL` command against the PostgreSQL target table
+    before copying the data, then the command `ALTER TABLE ... ENABLE
+    TRIGGER ALL` once the `COPY` is done.
+
+    This option allows loading data into a pre-existing table ignoring
+    the *foreign key constraints* and user defined triggers and may
+    result in invalid *foreign key constraints* once the data is loaded.
+    Use with care.
+
+  - *skip header*
+
+    Takes a numeric value as argument. Instruct pgloader to skip that
+    many lines at the beginning of the input file.
+

docs/_build/html/_sources/ref/ixf.rst.txt

@@ -0,0 +1,66 @@
+Loading IXF Data
+================
+
+This command instructs pgloader to load data from an IBM `IXF` file. Here's
+an example::
+
+    LOAD IXF
+        FROM data/nsitra.test1.ixf
+        INTO postgresql:///pgloader
+      TARGET TABLE nsitra.test1
+        WITH truncate, create table, timezone UTC
+
+      BEFORE LOAD DO
+       $$ create schema if not exists nsitra; $$,
+       $$ drop table if exists nsitra.test1; $$;
+
+The `ixf` format command accepts the following clauses and options.
+
+IXF Source Specification: FROM
+------------------------------
+
+Filename where to load the data from. This support local files, HTTP URLs
+and zip files containing a single ixf file of the same name. Fetch such a
+zip file from an HTTP address is of course supported.
+
+IXF Loading Options: WITH
+-------------------------
+
+When loading from a `IXF` file, the following options are supported:
+
+  - *truncate*
+
+    When this option is listed, pgloader issues a `TRUNCATE` command against
+    the PostgreSQL target table before reading the data file.
+
+  - *disable triggers*
+
+    When this option is listed, pgloader issues an `ALTER TABLE ... DISABLE
+    TRIGGER ALL` command against the PostgreSQL target table before copying
+    the data, then the command `ALTER TABLE ... ENABLE TRIGGER ALL` once the
+    `COPY` is done.
+
+    This option allows loading data into a pre-existing table ignoring the
+    *foreign key constraints* and user defined triggers and may result in
+    invalid *foreign key constraints* once the data is loaded. Use with
+    care.
+
+  - *create table*
+
+    When this option is listed, pgloader creates the table using the meta
+    data found in the `DBF` file, which must contain a list of fields with
+    their data type. A standard data type conversion from DBF to PostgreSQL
+    is done.
+
+  - *table name*
+
+    This options expects as its value the possibly qualified name of the
+    table to create.
+
+  - *timezone*
+
+    This options allows to specify which timezone is used when parsing
+    timestamps from an IXF file, and defaults to *UTC*. Expected values are
+    either `UTC`, `GMT` or a single quoted location name such as
+    `'Universal'` or `'Europe/Paris'`.
+

docs/_build/html/_sources/ref/mssql.rst.txt

@@ -0,0 +1,159 @@
+Migrating a MS SQL Database to PostgreSQL
+=========================================
+
+This command instructs pgloader to load data from a MS SQL database.
+Automatic discovery of the schema is supported, including build of the
+indexes, primary and foreign keys constraints.
+
+Here's an example::
+
+    load database
+         from mssql://user@host/dbname
+         into postgresql:///dbname
+
+    including only table names like 'GlobalAccount' in schema 'dbo'
+
+    set work_mem to '16MB', maintenance_work_mem to '512 MB'
+
+    before load do $$ drop schema if exists dbo cascade; $$;
+
+The `mssql` command accepts the following clauses and options.
+
+MS SQL Database Source Specification: FROM
+------------------------------------------
+
+Connection string to an existing MS SQL database server that listens and
+welcome external TCP/IP connection. As pgloader currently piggybacks on the
+FreeTDS driver, to change the port of the server please export the `TDSPORT`
+environment variable.
+
+MS SQL Database Migration Options: WITH
+---------------------------------------
+
+When loading from a `MS SQL` database, the same options as when loading a
+`MySQL` database are supported. Please refer to the MySQL section. The
+following options are added:
+
+  - *create schemas*
+
+    When this option is listed, pgloader creates the same schemas as found
+    on the MS SQL instance. This is the default.
+
+  - *create no schemas*
+
+    When this option is listed, pgloader refrains from creating any schemas
+    at all, you must then ensure that the target schema do exist.
+
+MS SQL Database Casting Rules
+-----------------------------
+    
+CAST
+^^^^
+
+The cast clause allows to specify custom casting rules, either to overload
+the default casting rules or to amend them with special cases.
+
+Please refer to the MySQL CAST clause for details.
+
+MS SQL Partial Migration
+------------------------
+
+
+INCLUDING ONLY TABLE NAMES LIKE
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Introduce a comma separated list of table name patterns used to limit the
+tables to migrate to a sublist. More than one such clause may be used, they
+will be accumulated together.
+
+Example::
+
+  including only table names lile 'GlobalAccount' in schema 'dbo'
+
+EXCLUDING TABLE NAMES LIKE
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Introduce a comma separated list of table name patterns used to exclude
+table names from the migration. This filter only applies to the result of
+the *INCLUDING* filter.
+
+::
+   
+   excluding table names matching 'LocalAccount' in schema 'dbo'
+
+MS SQL Schema Transformations
+-----------------------------
+
+ALTER SCHEMA '...' RENAME TO '...'
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Allows to rename a schema on the flight, so that for instance the tables
+found in the schema 'dbo' in your source database will get migrated into the
+schema 'public' in the target database with this command::
+
+  alter schema 'dbo' rename to 'public'
+
+ALTER TABLE NAMES MATCHING ... IN SCHEMA '...'
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+See the MySQL explanation for this clause above. It works the same in the
+context of migrating from MS SQL, only with the added option to specify the
+name of the schema where to find the definition of the target tables.
+
+The matching is done in pgloader itself, with a Common Lisp regular
+expression lib, so doesn't depend on the *LIKE* implementation of MS SQL,
+nor on the lack of support for regular expressions in the engine.
+
+MS SQL Driver setup and encoding
+--------------------------------
+
+pgloader is using the `FreeTDS` driver, and internally expects the data to
+be sent in utf-8. To achieve that, you can configure the FreeTDS driver with
+those defaults, in the file `~/.freetds.conf`::
+
+    [global]
+        tds version = 7.4
+        client charset = UTF-8
+
+Default MS SQL Casting Rules
+----------------------------
+
+When migrating from MS SQL the following Casting Rules are provided:
+
+Numbers::
+
+  type tinyint to smallint
+
+  type float to float   using float-to-string
+  type real to real     using float-to-string
+  type double to double precision     using float-to-string
+  type numeric to numeric     using float-to-string
+  type decimal to numeric     using float-to-string
+  type money to numeric     using float-to-string
+  type smallmoney to numeric     using float-to-string
+
+Texts::
+
+  type char      to text drop typemod
+  type nchat     to text drop typemod
+  type varchar   to text drop typemod
+  type nvarchar  to text drop typemod
+  type xml       to text drop typemod
+
+Binary::
+
+  type binary    to bytea using byte-vector-to-bytea
+  type varbinary to bytea using byte-vector-to-bytea
+
+Date::
+
+  type datetime    to timestamptz
+  type datetime2   to timestamptz
+
+Others::
+
+  type bit to boolean
+  type hierarchyid to bytea
+  type geography to bytea
+  type uniqueidentifier to uuid using sql-server-uniqueidentifier-to-uuid
+

docs/_build/html/_sources/ref/mysql.rst.txt

@@ -0,0 +1,623 @@
+Migrating a MySQL Database to PostgreSQL
+========================================
+
+This command instructs pgloader to load data from a database connection. The
+only supported database source is currently *MySQL*, and pgloader supports
+dynamically converting the schema of the source database and the indexes
+building.
+
+A default set of casting rules are provided and might be overloaded and
+appended to by the command.
+
+Here's an example using as many options as possible, some of them even being
+defaults. Chances are you don't need that complex a setup, don't copy and
+paste it, use it only as a reference!
+
+::
+   
+    LOAD DATABASE
+         FROM      mysql://root@localhost/sakila
+         INTO postgresql://localhost:54393/sakila
+
+     WITH include drop, create tables, create indexes, reset sequences,
+          workers = 8, concurrency = 1,
+          multiple readers per thread, rows per range = 50000
+
+      SET PostgreSQL PARAMETERS
+          maintenance_work_mem to '128MB',
+          work_mem to '12MB',
+          search_path to 'sakila, public, "$user"'
+    
+      SET MySQL PARAMETERS
+          net_read_timeout  = '120',
+          net_write_timeout = '120'
+
+     CAST type bigint when (= precision 20) to bigserial drop typemod,
+          type date drop not null drop default using zero-dates-to-null,
+          -- type tinyint to boolean using tinyint-to-boolean,
+          type year to integer
+
+     MATERIALIZE VIEWS film_list, staff_list
+
+     -- INCLUDING ONLY TABLE NAMES MATCHING ~/film/, 'actor'
+     -- EXCLUDING TABLE NAMES MATCHING ~<ory>
+     -- DECODING TABLE NAMES MATCHING ~/messed/, ~/encoding/ AS utf8
+     -- ALTER TABLE NAMES MATCHING 'film' RENAME TO 'films'
+     -- ALTER TABLE NAMES MATCHING ~/_list$/ SET SCHEMA 'mv'
+     
+     ALTER TABLE NAMES MATCHING ~/_list$/, 'sales_by_store', ~/sales_by/
+      SET SCHEMA 'mv'
+    
+     ALTER TABLE NAMES MATCHING 'film' RENAME TO 'films'
+     ALTER TABLE NAMES MATCHING ~/./ SET (fillfactor='40')
+    
+     ALTER SCHEMA 'sakila' RENAME TO 'pagila'
+
+     BEFORE LOAD DO
+       $$ create schema if not exists pagila; $$,
+       $$ create schema if not exists mv;     $$,
+       $$ alter database sakila set search_path to pagila, mv, public; $$;
+
+
+The `database` command accepts the following clauses and options.
+
+MySQL Database Source Specification: FROM
+-----------------------------------------
+
+Must be a connection URL pointing to a MySQL database.
+
+If the connection URI contains a table name, then only this table is
+migrated from MySQL to PostgreSQL.
+
+See the `SOURCE CONNECTION STRING` section above for details on how to write
+the connection string. The MySQL connection string accepts the same
+parameter *sslmode* as the PostgreSQL connection string, but the *verify*
+mode is not implemented (yet).
+    
+Environment variables described in
+<http://dev.mysql.com/doc/refman/5.0/en/environment-variables.html> can be
+used as default values too. If the user is not provided, then it defaults to
+`USER` environment variable value. The password can be provided with the
+environment variable `MYSQL_PWD`. The host can be provided with the
+environment variable `MYSQL_HOST` and otherwise defaults to `localhost`. The
+port can be provided with the environment variable `MYSQL_TCP_PORT` and
+otherwise defaults to `3306`.
+
+MySQL Database Migration Options: WITH
+--------------------------------------
+
+When loading from a `MySQL` database, the following options are supported,
+and the default *WITH* clause is: *no truncate*, *create schema*, *create
+tables*, *include drop*, *create indexes*, *reset sequences*, *foreign
+keys*, *downcase identifiers*, *uniquify index names*.
+
+  - *include drop*
+
+    When this option is listed, pgloader drops all the tables in the target
+    PostgreSQL database whose names appear in the MySQL database. This
+    option allows for using the same command several times in a row until
+    you figure out all the options, starting automatically from a clean
+    environment. Please note that `CASCADE` is used to ensure that tables
+    are dropped even if there are foreign keys pointing to them. This is
+    precisely what `include drop` is intended to do: drop all target tables
+    and recreate them.
+
+    Great care needs to be taken when using `include drop`, as it will
+    cascade to *all* objects referencing the target tables, possibly
+    including other tables that are not being loaded from the source DB.
+
+  - *include no drop*
+
+    When this option is listed, pgloader will not include any `DROP`
+    statement when loading the data.
+
+  - *truncate*
+
+    When this option is listed, pgloader issue the `TRUNCATE` command
+    against each PostgreSQL table just before loading data into it.
+
+  - *no truncate*
+
+    When this option is listed, pgloader issues no `TRUNCATE` command.
+
+  - *disable triggers*
+
+    When this option is listed, pgloader issues an `ALTER TABLE ... DISABLE
+    TRIGGER ALL` command against the PostgreSQL target table before copying
+    the data, then the command `ALTER TABLE ... ENABLE TRIGGER ALL` once the
+    `COPY` is done.
+
+    This option allows loading data into a pre-existing table ignoring the
+    *foreign key constraints* and user defined triggers and may result in
+    invalid *foreign key constraints* once the data is loaded. Use with
+    care.
+
+  - *create tables*
+
+    When this option is listed, pgloader creates the table using the meta
+    data found in the `MySQL` file, which must contain a list of fields with
+    their data type. A standard data type conversion from DBF to PostgreSQL
+    is done.
+
+  - *create no tables*
+
+    When this option is listed, pgloader skips the creation of table before
+    loading data, target tables must then already exist.
+
+    Also, when using *create no tables* pgloader fetches the metadata from
+    the current target database and checks type casting, then will remove
+    constraints and indexes prior to loading the data and install them back
+    again once the loading is done.
+
+  - *create indexes*
+
+    When this option is listed, pgloader gets the definitions of all the
+    indexes found in the MySQL database and create the same set of index
+    definitions against the PostgreSQL database.
+
+  - *create no indexes*
+
+    When this option is listed, pgloader skips the creating indexes.
+        
+  - *drop indexes*
+  
+    When this option is listed, pgloader drops the indexes in the target
+    database before loading the data, and creates them again at the end
+    of the data copy.
+
+  - *uniquify index names*, *preserve index names*
+
+    MySQL index names are unique per-table whereas in PostgreSQL index names
+    have to be unique per-schema. The default for pgloader is to change the
+    index name by prefixing it with `idx_OID` where `OID` is the internal
+    numeric identifier of the table the index is built against.
+
+    In somes cases like when the DDL are entirely left to a framework it
+    might be sensible for pgloader to refrain from handling index unique
+    names, that is achieved by using the *preserve index names* option.
+
+    The default is to *uniquify index names*.
+
+    Even when using the option *preserve index names*, MySQL primary key
+    indexes named "PRIMARY" will get their names uniquified. Failing to do
+    so would prevent the primary keys to be created again in PostgreSQL
+    where the index names must be unique per schema.
+
+  - *drop schema*
+  
+    When this option is listed, pgloader drops the target schema in the
+    target PostgreSQL database before creating it again and all the objects
+    it contains. The default behavior doesn't drop the target schemas.
+
+  - *foreign keys*
+
+    When this option is listed, pgloader gets the definitions of all the
+    foreign keys found in the MySQL database and create the same set of
+    foreign key definitions against the PostgreSQL database.
+
+  - *no foreign keys*
+
+    When this option is listed, pgloader skips creating foreign keys.
+
+  - *reset sequences*
+
+    When this option is listed, at the end of the data loading and after the
+    indexes have all been created, pgloader resets all the PostgreSQL
+    sequences created to the current maximum value of the column they are
+    attached to.
+
+    The options *schema only* and *data only* have no effects on this
+    option.
+
+  - *reset no sequences*
+
+    When this option is listed, pgloader skips resetting sequences after the
+    load.
+
+    The options *schema only* and *data only* have no effects on this
+    option.
+
+  - *downcase identifiers*
+
+    When this option is listed, pgloader converts all MySQL identifiers
+    (table names, index names, column names) to *downcase*, except for
+    PostgreSQL *reserved* keywords.
+
+    The PostgreSQL *reserved* keywords are determined dynamically by using
+    the system function `pg_get_keywords()`.
+
+  - *quote identifiers*
+
+    When this option is listed, pgloader quotes all MySQL identifiers so
+    that their case is respected. Note that you will then have to do the
+    same thing in your application code queries.
+
+  - *schema only*
+
+    When this option is listed pgloader refrains from migrating the data
+    over. Note that the schema in this context includes the indexes when the
+    option *create indexes* has been listed.
+
+  - *data only*
+
+    When this option is listed pgloader only issues the `COPY` statements,
+    without doing any other processing.
+
+  - *single reader per thread*, *multiple readers per thread*
+  
+    The default is *single reader per thread* and it means that each
+    MySQL table is read by a single thread as a whole, with a single
+    `SELECT` statement using no `WHERE` clause.
+    
+    When using *multiple readers per thread* pgloader may be able to
+    divide the reading work into several threads, as many as the
+    *concurrency* setting, which needs to be greater than 1 for this
+    option to kick be activated.
+    
+    For each source table, pgloader searches for a primary key over a
+    single numeric column, or a multiple-column primary key index for
+    which the first column is of a numeric data type (one of `integer`
+    or `bigint`). When such an index exists, pgloader runs a query to
+    find the *min* and *max* values on this column, and then split that
+    range into many ranges containing a maximum of *rows per range*.
+    
+    When the range list we then obtain contains at least as many ranges
+    than our concurrency setting, then we distribute those ranges to
+    each reader thread.
+    
+    So when all the conditions are met, pgloader then starts as many
+    reader thread as the *concurrency* setting, and each reader thread
+    issues several queries with a `WHERE id >= x AND id < y`, where `y -
+    x = rows per range` or less (for the last range, depending on the
+    max value just obtained.
+  
+  - *rows per range*
+  
+    How many rows are fetched per `SELECT` query when using *multiple
+    readers per thread*, see above for details.
+
+  - *SET MySQL PARAMETERS*
+  
+    The *SET MySQL PARAMETERS* allows setting MySQL parameters using the
+    MySQL `SET` command each time pgloader connects to it.
+
+MySQL Database Casting Rules
+----------------------------
+
+The command *CAST* introduces user-defined casting rules.
+
+The cast clause allows to specify custom casting rules, either to overload
+the default casting rules or to amend them with special cases.
+
+A casting rule is expected to follow one of the forms::
+
+    type <mysql-type-name> [ <guard> ... ] to <pgsql-type-name> [ <option> ... ]
+    column <table-name>.<column-name> [ <guards> ] to ...
+
+It's possible for a *casting rule* to either match against a MySQL data type
+or against a given *column name* in a given *table name*. That flexibility
+allows to cope with cases where the type `tinyint` might have been used as a
+`boolean` in some cases but as a `smallint` in others.
+
+The *casting rules* are applied in order, the first match prevents following
+rules to be applied, and user defined rules are evaluated first.
+
+The supported guards are:
+
+  - *when unsigned*
+
+    The casting rule is only applied against MySQL columns of the source
+    type that have the keyword *unsigned* in their data type definition.
+
+    Example of a casting rule using a *unsigned* guard::
+        
+      type smallint when unsigned to integer drop typemod
+
+  - *when default 'value'*
+
+    The casting rule is only applied against MySQL columns of the source
+    type that have given *value*, which must be a single-quoted or a
+    double-quoted string.
+
+  - *when typemod expression*
+
+    The casting rule is only applied against MySQL columns of the source
+    type that have a *typemod* value matching the given *typemod
+    expression*. The *typemod* is separated into its *precision* and *scale*
+    components.
+
+    Example of a cast rule using a *typemod* guard::
+
+      type char when (= precision 1) to char keep typemod
+
+    This expression casts MySQL `char(1)` column to a PostgreSQL column of
+    type `char(1)` while allowing for the general case `char(N)` will be
+    converted by the default cast rule into a PostgreSQL type `varchar(N)`.
+
+  - *with extra auto_increment*
+
+    The casting rule is only applied against MySQL columns having the
+    *extra* column `auto_increment` option set, so that it's possible to
+    target e.g. `serial` rather than `integer`.
+
+    The default matching behavior, when this option isn't set, is to match
+    both columns with the extra definition and without.
+
+    This means that if you want to implement a casting rule that target
+    either `serial` or `integer` from a `smallint` definition depending on
+    the *auto_increment* extra bit of information from MySQL, then you need
+    to spell out two casting rules as following::
+
+      type smallint  with extra auto_increment
+        to serial drop typemod keep default keep not null,
+
+      type smallint
+        to integer drop typemod keep default keep not null
+
+The supported casting options are:
+
+  - *drop default*, *keep default*
+
+    When the option *drop default* is listed, pgloader drops any
+    existing default expression in the MySQL database for columns of the
+    source type from the `CREATE TABLE` statement it generates.
+
+    The spelling *keep default* explicitly prevents that behaviour and
+    can be used to overload the default casting rules.
+
+  - *drop not null*, *keep not null*, *set not null*
+
+    When the option *drop not null* is listed, pgloader drops any
+    existing `NOT NULL` constraint associated with the given source
+    MySQL datatype when it creates the tables in the PostgreSQL
+    database.
+
+    The spelling *keep not null* explicitly prevents that behaviour and
+    can be used to overload the default casting rules.
+
+    When the option *set not null* is listed, pgloader sets a `NOT NULL`
+    constraint on the target column regardless whether it has been set
+    in the source MySQL column.
+
+  - *drop typemod*, *keep typemod*
+
+    When the option *drop typemod* is listed, pgloader drops any
+    existing *typemod* definition (e.g. *precision* and *scale*) from
+    the datatype definition found in the MySQL columns of the source
+    type when it created the tables in the PostgreSQL database.
+
+    The spelling *keep typemod* explicitly prevents that behaviour and
+    can be used to overload the default casting rules.
+
+  - *using*
+
+    This option takes as its single argument the name of a function to
+    be found in the `pgloader.transforms` Common Lisp package. See above
+    for details.
+
+    It's possible to augment a default cast rule (such as one that
+    applies against `ENUM` data type for example) with a *transformation
+    function* by omitting entirely the `type` parts of the casting rule,
+    as in the following example::
+
+      column enumerate.foo using empty-string-to-null
+
+MySQL Views Support
+-------------------
+
+MySQL views support allows pgloader to migrate view as if they were base
+tables. This feature then allows for on-the-fly transformation from MySQL to
+PostgreSQL, as the view definition is used rather than the base data.
+
+MATERIALIZE VIEWS
+^^^^^^^^^^^^^^^^^
+
+This clause allows you to implement custom data processing at the data
+source by providing a *view definition* against which pgloader will query
+the data. It's not possible to just allow for plain `SQL` because we want to
+know a lot about the exact data types of each column involved in the query
+output.
+
+This clause expect a comma separated list of view definitions, each one
+being either the name of an existing view in your database or the following
+expression::
+
+  *name* `AS` `$$` *sql query* `$$`
+
+The *name* and the *sql query* will be used in a `CREATE VIEW` statement at
+the beginning of the data loading, and the resulting view will then be
+dropped at the end of the data loading.
+
+MATERIALIZE ALL VIEWS
+^^^^^^^^^^^^^^^^^^^^^
+
+Same behaviour as *MATERIALIZE VIEWS* using the dynamic list of views as
+returned by MySQL rather than asking the user to specify the list.
+
+MySQL Partial Migration
+-----------------------
+
+INCLUDING ONLY TABLE NAMES MATCHING
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Introduce a comma separated list of table names or *regular expression* used
+to limit the tables to migrate to a sublist.
+
+Example::
+
+  including only table names matching ~/film/, 'actor'
+
+EXCLUDING TABLE NAMES MATCHING
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Introduce a comma separated list of table names or *regular expression* used
+to exclude table names from the migration. This filter only applies to the
+result of the *INCLUDING* filter.
+
+::
+  
+  excluding table names matching ~<ory>
+
+MySQL Encoding Support
+----------------------
+      
+DECODING TABLE NAMES MATCHING
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Introduce a comma separated list of table names or *regular expressions*
+used to force the encoding to use when processing data from MySQL. If the
+data encoding known to you is different from MySQL's idea about it, this is
+the option to use.
+
+::
+  
+  decoding table names matching ~/messed/, ~/encoding/ AS utf8
+
+You can use as many such rules as you need, all with possibly different
+encodings.
+
+MySQL Schema Transformations
+----------------------------
+    
+ALTER TABLE NAMES MATCHING
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Introduce a comma separated list of table names or *regular expressions*
+that you want to target in the pgloader *ALTER TABLE* command. The only two
+available actions are *SET SCHEMA* and *RENAME TO*, both take a quoted
+string as parameter::
+
+    ALTER TABLE NAMES MATCHING ~/_list$/, 'sales_by_store', ~/sales_by/
+     SET SCHEMA 'mv'
+   
+    ALTER TABLE NAMES MATCHING 'film' RENAME TO 'films'
+    
+    ALTER TABLE NAMES MATCHING ~/./ SET (fillfactor='40')
+
+You can use as many such rules as you need. The list of tables to be
+migrated is searched in pgloader memory against the *ALTER TABLE* matching
+rules, and for each command pgloader stops at the first matching criteria
+(regexp or string).
+
+No *ALTER TABLE* command is sent to PostgreSQL, the modification happens at
+the level of the pgloader in-memory representation of your source database
+schema. In case of a name change, the mapping is kept and reused in the
+*foreign key* and *index* support.
+
+The *SET ()* action takes effect as a *WITH* clause for the `CREATE TABLE`
+command that pgloader will run when it has to create a table.
+
+MySQL Migration: limitations
+----------------------------
+
+The `database` command currently only supports MySQL source database and has
+the following limitations:
+
+  - Views are not migrated,
+
+    Supporting views might require implementing a full SQL parser for the
+    MySQL dialect with a porting engine to rewrite the SQL against
+    PostgreSQL, including renaming functions and changing some constructs.
+
+    While it's not theoretically impossible, don't hold your breath.
+
+  - Triggers are not migrated
+
+    The difficulty of doing so is not yet assessed.
+
+  - Of the geometric datatypes, only the `POINT` database has been covered.
+    The other ones should be easy enough to implement now, it's just not
+    done yet.
+
+Default MySQL Casting Rules
+---------------------------
+
+When migrating from MySQL the following Casting Rules are provided:
+
+Numbers::
+
+  type int with extra auto_increment to serial when (< precision 10)
+  type int with extra auto_increment to bigserial when (<= 10 precision)
+  type int to int       when  (< precision 10)
+  type int to bigint    when  (<= 10 precision)
+  type tinyint   with extra auto_increment to serial
+  type smallint  with extra auto_increment to serial
+  type mediumint with extra auto_increment to serial
+  type bigint    with extra auto_increment to bigserial
+
+  type tinyint to boolean when (= 1 precision) using tinyint-to-boolean
+
+  type tinyint when unsigned to smallint   drop typemod
+  type smallint when unsigned to integer  drop typemod
+  type mediumint when unsigned to integer  drop typemod
+  type integer when unsigned to bigint    drop typemod
+  
+  type tinyint to smallint   drop typemod
+  type smallint to smallint  drop typemod
+  type mediumint to integer  drop typemod
+  type integer to integer    drop typemod
+  type bigint to bigint      drop typemod
+
+  type float to float        drop typemod
+  type double to double precision drop typemod
+
+  type numeric to numeric keep typemod
+  type decimal to decimal keep typemod
+
+Texts::
+
+  type char       to char keep typemod using remove-null-characters
+  type varchar    to varchar keep typemod using remove-null-characters
+  type tinytext   to text using remove-null-characters
+  type text       to text using remove-null-characters
+  type mediumtext to text using remove-null-characters
+  type longtext   to text using remove-null-characters
+
+Binary:
+
+  type binary     to bytea
+  type varbinary  to bytea
+  type tinyblob   to bytea
+  type blob       to bytea
+  type mediumblob to bytea
+  type longblob   to bytea
+
+Date::
+  
+  type datetime when default "0000-00-00 00:00:00" and not null
+    to timestamptz drop not null drop default
+	using zero-dates-to-null
+
+  type datetime when default "0000-00-00 00:00:00"
+    to timestamptz drop default
+	using zero-dates-to-null
+
+  type timestamp when default "0000-00-00 00:00:00" and not null
+    to timestamptz drop not null drop default
+	using zero-dates-to-null
+
+  type timestamp when default "0000-00-00 00:00:00"
+    to timestamptz drop default
+	using zero-dates-to-null
+
+  type date when default "0000-00-00" to date drop default
+	using zero-dates-to-null
+
+  type date to date
+  type datetime to timestamptz
+  type timestamp to timestamptz
+  type year to integer drop typemod
+
+Geometric::
+
+  type point to point using pgloader.transforms::convert-mysql-point
+
+Enum types are declared inline in MySQL and separately with a `CREATE TYPE`
+command in PostgreSQL, so each column of Enum Type is converted to a type
+named after the table and column names defined with the same labels in the
+same order.
+
+When the source type definition is not matched in the default casting rules
+nor in the casting rules provided in the command, then the type name with
+the typemod is used.
+

docs/_build/html/_sources/ref/sqlite.rst.txt

@@ -0,0 +1,207 @@
+Migrating a SQLite database to PostgreSQL
+=========================================
+
+This command instructs pgloader to load data from a SQLite file. Automatic
+discovery of the schema is supported, including build of the indexes.
+
+Here's an example::
+
+    load database
+         from sqlite:///Users/dim/Downloads/lastfm_tags.db
+         into postgresql:///tags
+
+     with include drop, create tables, create indexes, reset sequences
+
+      set work_mem to '16MB', maintenance_work_mem to '512 MB';
+
+The `sqlite` command accepts the following clauses and options.
+
+SQLite Database Source Specification: FROM
+------------------------------------------
+
+Path or HTTP URL to a SQLite file, might be a `.zip` file.
+
+SQLite Database Migration Options: WITH
+---------------------------------------
+
+When loading from a `SQLite` database, the following options are
+supported:
+
+When loading from a `SQLite` database, the following options are
+supported, and the default *WITH* clause is: *no truncate*, *create
+tables*, *include drop*, *create indexes*, *reset sequences*, *downcase
+identifiers*, *encoding 'utf-8'*.
+
+  - *include drop*
+
+    When this option is listed, pgloader drops all the tables in the target
+    PostgreSQL database whose names appear in the SQLite database. This
+    option allows for using the same command several times in a row until
+    you figure out all the options, starting automatically from a clean
+    environment. Please note that `CASCADE` is used to ensure that tables
+    are dropped even if there are foreign keys pointing to them. This is
+    precisely what `include drop` is intended to do: drop all target tables
+    and recreate them.
+
+    Great care needs to be taken when using `include drop`, as it will
+    cascade to *all* objects referencing the target tables, possibly
+    including other tables that are not being loaded from the source DB.
+
+  - *include no drop*
+
+    When this option is listed, pgloader will not include any `DROP`
+    statement when loading the data.
+
+  - *truncate*
+
+    When this option is listed, pgloader issue the `TRUNCATE` command
+    against each PostgreSQL table just before loading data into it.
+
+  - *no truncate*
+
+    When this option is listed, pgloader issues no `TRUNCATE` command.
+
+  - *disable triggers*
+
+    When this option is listed, pgloader issues an `ALTER TABLE ... DISABLE
+    TRIGGER ALL` command against the PostgreSQL target table before copying
+    the data, then the command `ALTER TABLE ... ENABLE TRIGGER ALL` once the
+    `COPY` is done.
+
+    This option allows loading data into a pre-existing table ignoring
+    the *foreign key constraints* and user defined triggers and may
+    result in invalid *foreign key constraints* once the data is loaded.
+    Use with care.
+
+  - *create tables*
+
+    When this option is listed, pgloader creates the table using the meta
+    data found in the `SQLite` file, which must contain a list of fields
+    with their data type. A standard data type conversion from SQLite to
+    PostgreSQL is done.
+
+  - *create no tables*
+
+    When this option is listed, pgloader skips the creation of table before
+    loading data, target tables must then already exist.
+
+    Also, when using *create no tables* pgloader fetches the metadata
+    from the current target database and checks type casting, then will
+    remove constraints and indexes prior to loading the data and install
+    them back again once the loading is done.
+
+  - *create indexes*
+
+    When this option is listed, pgloader gets the definitions of all the
+    indexes found in the SQLite database and create the same set of index
+    definitions against the PostgreSQL database.
+
+  - *create no indexes*
+
+    When this option is listed, pgloader skips the creating indexes.
+
+  - *drop indexes*
+  
+    When this option is listed, pgloader drops the indexes in the target
+    database before loading the data, and creates them again at the end
+    of the data copy.
+
+  - *reset sequences*
+
+    When this option is listed, at the end of the data loading and after
+    the indexes have all been created, pgloader resets all the
+    PostgreSQL sequences created to the current maximum value of the
+    column they are attached to.
+
+  - *reset no sequences*
+
+    When this option is listed, pgloader skips resetting sequences after the
+    load.
+
+    The options *schema only* and *data only* have no effects on this
+    option.
+
+  - *schema only*
+
+    When this option is listed pgloader will refrain from migrating the data
+    over. Note that the schema in this context includes the indexes when the
+    option *create indexes* has been listed.
+
+  - *data only*
+
+    When this option is listed pgloader only issues the `COPY` statements,
+    without doing any other processing.
+
+  - *encoding*
+
+    This option allows to control which encoding to parse the SQLite text
+    data with. Defaults to UTF-8.
+
+SQLite Database Casting Rules
+-----------------------------
+    
+The command *CAST* introduces user-defined casting rules.
+
+The cast clause allows to specify custom casting rules, either to overload
+the default casting rules or to amend them with special cases.
+
+SQlite Database Partial Migrations
+----------------------------------
+
+INCLUDING ONLY TABLE NAMES LIKE
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Introduce a comma separated list of table name patterns used to limit the
+tables to migrate to a sublist.
+
+Example::
+
+ including only table names like 'Invoice%'
+
+EXCLUDING TABLE NAMES LIKE
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Introduce a comma separated list of table name patterns used to exclude
+table names from the migration. This filter only applies to the result of
+the *INCLUDING* filter.
+
+::
+  
+  excluding table names like 'appointments'
+
+Default SQLite Casting Rules
+----------------------------
+
+When migrating from SQLite the following Casting Rules are provided:
+
+Numbers::
+
+  type tinyint to smallint using integer-to-string
+  type integer to bigint   using integer-to-string
+
+  type float to float   using float-to-string
+  type real to real     using float-to-string
+  type double to double precision     using float-to-string
+  type numeric to numeric     using float-to-string
+
+Texts::
+
+  type character  to text drop typemod
+  type varchar    to text drop typemod
+  type nvarchar   to text drop typemod
+  type char       to text drop typemod
+  type nchar      to text drop typemod
+  type nvarchar   to text drop typemod
+  type clob       to text drop typemod
+
+Binary::
+
+  type blob       to bytea
+
+Date::
+
+  type datetime    to timestamptz using sqlite-timestamp-to-timestamp
+  type timestamp   to timestamptz using sqlite-timestamp-to-timestamp
+  type timestamptz to timestamptz using sqlite-timestamp-to-timestamp
+
+

docs/_build/html/_sources/ref/transforms.rst.txt

@@ -0,0 +1,134 @@
+Transformation Functions
+========================
+
+Some data types are implemented in a different enough way that a
+transformation function is necessary. This function must be written in
+`Common lisp` and is searched in the `pgloader.transforms` package.
+
+Some default transformation function are provided with pgloader, and you can
+use the `--load` command line option to load and compile your own lisp file
+into pgloader at runtime. For your functions to be found, remember to begin
+your lisp file with the following form::
+
+    (in-package #:pgloader.transforms)
+
+The provided transformation functions are:
+
+  - *zero-dates-to-null*
+
+    When the input date is all zeroes, return `nil`, which gets loaded as a
+    PostgreSQL `NULL` value.
+
+  - *date-with-no-separator*
+
+    Applies *zero-dates-to-null* then transform the given date into a format
+    that PostgreSQL will actually process::
+
+            In:  "20041002152952"
+            Out: "2004-10-02 15:29:52"
+
+  - *time-with-no-separator*
+
+    Transform the given time into a format that PostgreSQL will actually
+    process::
+
+            In:  "08231560"
+            Out: "08:23:15.60"
+
+  - *tinyint-to-boolean*
+
+    As MySQL lacks a proper boolean type, *tinyint* is often used to
+    implement that. This function transforms `0` to `'false'` and anything
+    else to `'true`'.
+
+  - *bits-to-boolean*
+
+    As MySQL lacks a proper boolean type, *BIT* is often used to implement
+    that. This function transforms 1-bit bit vectors from `0` to `f` and any
+    other value to `t`..
+
+  - *int-to-ip*
+
+    Convert an integer into a dotted representation of an ip4. ::
+
+        In:  18435761
+        Out: "1.25.78.177"
+
+  - *ip-range*
+
+    Converts a couple of integers given as strings into a range of ip4. ::
+
+            In:  "16825344" "16825599"
+            Out: "1.0.188.0-1.0.188.255"
+
+  - *convert-mysql-point*
+
+    Converts from the `astext` representation of points in MySQL to the
+    PostgreSQL representation. ::
+
+        In:  "POINT(48.5513589 7.6926827)"
+        Out: "(48.5513589,7.6926827)"
+
+  - *integer-to-string*
+
+    Converts a integer string or a Common Lisp integer into a string
+    suitable for a PostgreSQL integer. Takes care of quoted integers. ::
+
+            In:  "\"0\""
+            Out: "0"
+
+  - *float-to-string*
+
+    Converts a Common Lisp float into a string suitable for a PostgreSQL float::
+
+            In:  100.0d0
+            Out: "100.0"
+
+  - *set-to-enum-array*
+
+    Converts a string representing a MySQL SET into a PostgreSQL Array of
+    Enum values from the set. ::
+
+            In: "foo,bar"
+            Out: "{foo,bar}"
+
+  - *empty-string-to-null*
+
+    Convert an empty string to a null.
+
+  - *right-trim*
+
+    Remove whitespace at end of string.
+
+  - *remove-null-characters*
+
+    Remove `NUL` characters (`0x0`) from given strings.
+
+  - *byte-vector-to-bytea*
+
+    Transform a simple array of unsigned bytes to the PostgreSQL bytea Hex
+    Format representation as documented at
+    http://www.postgresql.org/docs/9.3/interactive/datatype-binary.html
+
+  - *sqlite-timestamp-to-timestamp*
+
+    SQLite type system is quite interesting, so cope with it here to produce
+    timestamp literals as expected by PostgreSQL. That covers year only on 4
+    digits, 0 dates to null, and proper date strings.
+
+  - *sql-server-uniqueidentifier-to-uuid*
+
+    The SQL Server driver receives data fo type uniqueidentifier as byte
+    vector that we then need to convert to an UUID string for PostgreSQL
+    COPY input format to process.
+
+  - *unix-timestamp-to-timestamptz*
+
+    Converts a unix timestamp (number of seconds elapsed since beginning of
+    1970) into a proper PostgreSQL timestamp format.
+
+  - *varbinary-to-string*
+
+    Converts binary encoded string (such as a MySQL `varbinary` entry) to a
+    decoded text, using the table's encoding that may be overloaded with the
+    *DECODING TABLE NAMES MATCHING* clause.

docs/_build/html/_sources/tutorial/csv.rst.txt

@@ -1,13 +1,15 @@
-# Loading CSV Data with pgloader
+Loading CSV Data with pgloader
+------------------------------
 
 CSV means *comma separated values* and is often found with quite varying
 specifications. pgloader allows you to describe those specs in its command.
 
-## The Command
+The Command
+^^^^^^^^^^^
 
 To load data with [pgloader](http://pgloader.io/) you need to define in a
 *command* the operations in some details. Here's our example for loading CSV
-data:
+data::
 
      LOAD CSV
           FROM 'path/to/file.csv' (x, y, a, b, c, d)
@@ -33,13 +35,10 @@ data:
             );
        $$;
 
-You can see the full list of options in the
-[pgloader reference manual](pgloader.1.html), with a complete description
-of the options you see here.
+The Data
+^^^^^^^^
 
-## The Data
-
-This command allows loading the following CSV file content:
+This command allows loading the following CSV file content::
 
     Header, with a © sign
     "2.6.190.56","2.6.190.63","33996344","33996351","GB","United Kingdom"
@@ -49,10 +48,11 @@ This command allows loading the following CSV file content:
     "4.17.143.0","4.17.143.15","68259584","68259599","CA","Canada"
     "4.17.143.16","4.18.32.71","68259600","68296775","US","United States"
 
-## Loading the data
+Loading the data
+^^^^^^^^^^^^^^^^
 
 Here's how to start loading the data. Note that the ouput here has been
-edited so as to facilitate its browsing online.
+edited so as to facilitate its browsing online::
 
     $ pgloader csv.load
     ... LOG Starting pgloader, log system is ready.
@@ -66,11 +66,12 @@ edited so as to facilitate its browsing online.
     -----------------  ---------  ---------  ---------  --------------
     Total import time          6          6          0          0.058s
 
-## The result
+The result
+^^^^^^^^^^
 
 As you can see, the command described above is filtering the input and only
 importing some of the columns from the example data file. Here's what gets
-loaded in the PostgreSQL database:
+loaded in the PostgreSQL database::
 
     pgloader# table csv;
         a     |    b     | c  |       d        

docs/_build/html/_sources/tutorial/dBase.rst.txt

@@ -1,20 +1,22 @@
-# Loading dBase files with pgloader
+Loading dBase files with pgloader
+---------------------------------
 
 The dBase format is still in use in some places as modern tools such as
 *Filemaker* and *Excel* offer some level of support for it. Speaking of
 support in modern tools, pgloader is right there on the list too!
 
-## The Command
+The Command
+^^^^^^^^^^^
 
 To load data with [pgloader](http://pgloader.io/) you need to define in a
 *command* the operations in some details. Here's our example for loading a
 dBase file, using a file provided by the french administration.
 
-You can find more files from them at the
-[Insee](http://www.insee.fr/fr/methodes/nomenclatures/cog/telechargement.asp)
+You can find more files from them at the `Insee
+<http://www.insee.fr/fr/methodes/nomenclatures/cog/telechargement.asp>`_
 website.
 
-Here's our command:
+Here's our command::
 
     LOAD DBF
         FROM http://www.insee.fr/fr/methodes/nomenclatures/cog/telechargement/2013/dbf/historiq2013.zip
@@ -22,17 +24,14 @@ Here's our command:
         WITH truncate, create table
          SET client_encoding TO 'latin1';
 
-You can see the full list of options in the
-[pgloader reference manual](pgloader.1.html), with a complete description
-of the options you see here.
-
 Note that here pgloader will benefit from the meta-data information found in
 the dBase file to create a PostgreSQL table capable of hosting the data as
 described, then load the data.
 
-## Loading the data
+Loading the data
+^^^^^^^^^^^^^^^^
 
-Let's start the `pgloader` command with our `dbf-zip.load` command file:
+Let's start the `pgloader` command with our `dbf-zip.load` command file::
 
     $ pgloader dbf-zip.load
     ... LOG Starting pgloader, log system is ready.
@@ -50,7 +49,7 @@ Let's start the `pgloader` command with our `dbf-zip.load` command file:
     -----------------  ---------  ---------  ---------  --------------
     Total import time       9181       9181          0          1.906s
 
-We can see that [http://pgloader.io](pgloader) did download the file from
+We can see that `pgloader <http://pgloader.io>`_ did download the file from
 its HTTP URL location then *unziped* it before the loading itself.
 
 Note that the output of the command has been edited to facilitate its

docs/_build/html/_sources/tutorial/fixed.rst.txt

@@ -1,10 +1,12 @@
-# Loading Fixed Width Data File with pgloader
+Loading Fixed Width Data File with pgloader
+-------------------------------------------
 
 Some data providers still use a format where each column is specified with a
 starting index position and a given length. Usually the columns are
 blank-padded when the data is shorter than the full reserved range.
 
-## The Command
+The Command
+^^^^^^^^^^^
 
 To load data with [pgloader](http://pgloader.io/) you need to define in a
 *command* the operations in some details. Here's our example for loading
@@ -13,7 +15,7 @@ Fixed Width Data, using a file provided by the US census.
 You can find more files from them at the
 [Census 2000 Gazetteer Files](http://www.census.gov/geo/maps-data/data/gazetteer2000.html).
 
-Here's our command:
+Here's our command::
 
     LOAD ARCHIVE
        FROM http://www.census.gov/geo/maps-data/data/docs/gazetteer/places2k.zip
@@ -52,14 +54,11 @@ Here's our command:
     	        usps, fips, fips_code, "LocationName"
                  );
 
-You can see the full list of options in the
-[pgloader reference manual](pgloader.1.html), with a complete description
-of the options you see here.
-
-## The Data
+The Data
+^^^^^^^^
 
 This command allows loading the following file content, where we are only
-showing the first couple of lines:
+showing the first couple of lines::
 
     AL0100124Abbeville city                                                       2987     1353      40301945        120383   15.560669    0.046480 31.566367 -85.251300
     AL0100460Adamsville city                                                      4965     2042      50779330         14126   19.606010    0.005454 33.590411 -86.949166
@@ -69,9 +68,10 @@ showing the first couple of lines:
     AL0100988Albertville city                                                    17247     7090      67212867        258738   25.951034    0.099899 34.265362 -86.211261
     AL0101132Alexander City city                                                 15008     6855     100534344        433413   38.816529    0.167342 32.933157 -85.936008
 
-## Loading the data
+Loading the data
+^^^^^^^^^^^^^^^^
 
-Let's start the `pgloader` command with our `census-places.load` command file:
+Let's start the `pgloader` command with our `census-places.load` command file::
 
     $ pgloader census-places.load
     ... LOG Starting pgloader, log system is ready.
@@ -89,8 +89,8 @@ Let's start the `pgloader` command with our `census-places.load` command file:
     -----------------  ---------  ---------  ---------  --------------
     Total import time      25375      25375          0          3.019s
 
-We can see that [http://pgloader.io](pgloader) did download the file from
-its HTTP URL location then *unziped* it before the loading itself.
+We can see that pgloader did download the file from its HTTP URL location
+then *unziped* it before the loading itself.
 
 Note that the output of the command has been edited to facilitate its
 browsing online.

docs/_build/html/_sources/tutorial/geolite.rst.txt

@@ -1,15 +1,16 @@
-# Loading MaxMind Geolite Data with pgloader
+Loading MaxMind Geolite Data with pgloader
+------------------------------------------
 
-The [MaxMind](http://www.maxmind.com/) provides a free dataset for
+`MaxMind <http://www.maxmind.com/>`_ provides a free dataset for
 geolocation, which is quite popular. Using pgloader you can download the
 lastest version of it, extract the CSV files from the archive and load their
 content into your database directly.
 
-## The Command
+The Command
+^^^^^^^^^^^
 
-To load data with [pgloader](http://pgloader.io/) you need to define in a
-*command* the operations in some details. Here's our example for loading the
-Geolite data:
+To load data with pgloader you need to define in a *command* the operations
+in some details. Here's our example for loading the Geolite data::
 
     /*
      * Loading from a ZIP archive containing CSV files. The full test can be
@@ -92,27 +93,24 @@ Geolite data:
        FINALLY DO
          $$ create index blocks_ip4r_idx on geolite.blocks using gist(iprange); $$;
 
-You can see the full list of options in the
-[pgloader reference manual](pgloader.1.html), with a complete description
-of the options you see here.
-
 Note that while the *Geolite* data is using a pair of integers (*start*,
-*end*) to represent *ipv4* data, we use the very poweful
-[ip4r](https://github.com/RhodiumToad/ip4r) PostgreSQL Extension instead.
+*end*) to represent *ipv4* data, we use the very poweful `ip4r
+<https://github.com/RhodiumToad/ip4r>`_ PostgreSQL Extension instead.
 
 The transformation from a pair of integers into an IP is done dynamically by
 the pgloader process.
 
 Also, the location is given as a pair of *float* columns for the *longitude*
 and the *latitude* where PostgreSQL offers the
-[point](http://www.postgresql.org/docs/9.3/interactive/functions-geometry.html)
+`point <http://www.postgresql.org/docs/9.3/interactive/functions-geometry.html>`_
 datatype, so the pgloader command here will actually transform the data on
 the fly to use the appropriate data type and its input representation.
 
-## Loading the data
+Loading the data
+^^^^^^^^^^^^^^^^
 
 Here's how to start loading the data. Note that the ouput here has been
-edited so as to facilitate its browsing online.
+edited so as to facilitate its browsing online::
 
     $ pgloader archive.load
     ... LOG Starting pgloader, log system is ready.
@@ -135,12 +133,12 @@ edited so as to facilitate its browsing online.
 
 The timing of course includes the transformation of the *1.9 million* pairs
 of integer into a single *ipv4 range* each. The *finally* step consists of
-creating the *GiST* specialized index as given in the main command:
+creating the *GiST* specialized index as given in the main command::
 
     CREATE INDEX blocks_ip4r_idx ON geolite.blocks USING gist(iprange);
 
 That index will then be used to speed up queries wanting to find which
-recorded geolocation contains a specific IP address:
+recorded geolocation contains a specific IP address::
 
     ip4r> select *
             from      geolite.location l

docs/_build/html/_sources/tutorial/mysql.rst.txt

@@ -1,28 +1,29 @@
-# Migrating from MySQL to PostgreSQL
+Migrating from MySQL to PostgreSQL
+----------------------------------
 
-If you want to migrate your data over to
-[PostgreSQL](http://www.postgresql.org) from MySQL then pgloader is the tool
-of choice!
+If you want to migrate your data over to `PostgreSQL
+<http://www.postgresql.org>`_ from MySQL then pgloader is the tool of
+choice!
 
 Most tools around are skipping the main problem with migrating from MySQL,
 which is to do with the type casting and data sanitizing that needs to be
 done. pgloader will not leave you alone on those topics.
 
-## In a Single Command Line
+In a Single Command Line
+^^^^^^^^^^^^^^^^^^^^^^^^
 
 As an example, we will use the f1db database from <http://ergast.com/mrd/>
 which which provides a historical record of motor racing data for
 non-commercial purposes. You can either use their API or download the whole
-database
-at
-[http://ergast.com/downloads/f1db.sql.gz](http://ergast.com/downloads/f1db.sql.gz).
-Once you've done that load the database in MySQL:
+database at `http://ergast.com/downloads/f1db.sql.gz
+<http://ergast.com/downloads/f1db.sql.gz>`_. Once you've done that load the
+database in MySQL::
 
     $ mysql -u root
     > create database f1db;
     > source f1db.sql
 
-Now let's migrate this database into PostgreSQL in a single command line:
+Now let's migrate this database into PostgreSQL in a single command line::
 
     $ createdb f1db
     $ pgloader mysql://root@localhost/f1db pgsql:///f1db
@@ -30,9 +31,12 @@ Now let's migrate this database into PostgreSQL in a single command line:
 Done! All with schema, table definitions, constraints, indexes, primary
 keys, *auto_increment* columns turned into *bigserial* , foreign keys,
 comments, and if you had some MySQL default values such as *ON UPDATE
-CURRENT_TIMESTAMP* they would have been translated to
-a
-[PostgreSQL before update trigger](https://www.postgresql.org/docs/current/static/plpgsql-trigger.html) automatically.
+CURRENT_TIMESTAMP* they would have been translated to a `PostgreSQL before
+update trigger
+<https://www.postgresql.org/docs/current/static/plpgsql-trigger.html>`_
+automatically.
+
+::
 
     $ pgloader mysql://root@localhost/f1db pgsql:///f1db
     2017-06-16T08:56:14.064000+02:00 LOG Main logs in '/private/tmp/pgloader/pgloader.log'
@@ -75,14 +79,14 @@ You may need to have special cases to take care of tho, or views that you
 want to materialize while doing the migration. In advanced case you can use
 the pgloader command.
 
-## The Command
+The Command
+^^^^^^^^^^^
 
-To load data with [pgloader](http://pgloader.tapoueh.org/) you need to
-define in a *command* the operations in some details. Here's our example for
-loading the
-[MySQL Sakila Sample Database](http://dev.mysql.com/doc/sakila/en/):
+To load data with pgloader you need to define in a *command* the operations
+in some details. Here's our example for loading the `MySQL Sakila Sample
+Database <http://dev.mysql.com/doc/sakila/en/>`_.
 
-Here's our command:
+Here's our command::
 
     load database
          from      mysql://root@localhost/sakila
@@ -105,10 +109,6 @@ Here's our command:
      BEFORE LOAD DO
      $$ create schema if not exists sakila; $$;
 
-You can see the full list of options in the
-[pgloader reference manual](pgloader.1.html), with a complete description
-of the options you see here.
-
 Note that here pgloader will benefit from the meta-data information found in
 the MySQL database to create a PostgreSQL database capable of hosting the
 data as described, then load the data.
@@ -129,7 +129,7 @@ view before loading the data, then drops it again at the end.
 
 ## Loading the data
 
-Let's start the `pgloader` command with our `sakila.load` command file:
+Let's start the `pgloader` command with our `sakila.load` command file::
 
     $ pgloader sakila.load
     ... LOG Starting pgloader, log system is ready.

docs/_build/html/_sources/tutorial/quickstart.rst.txt

@@ -1,10 +1,12 @@
-# pgloader: a quickstart
+PgLoader Quick Start
+--------------------
 
 In simple cases, pgloader is very easy to use.
 
-## CSV
+CSV
+^^^
 
-Load data from a CSV file into a pre-existing table in your database:
+Load data from a CSV file into a pre-existing table in your database::
 
     pgloader --type csv                                   \
              --field id --field field                     \
@@ -23,10 +25,11 @@ For documentation about the available syntaxes for the `--field` and
 
 Note also that the PostgreSQL URI includes the target *tablename*.
 
-## Reading from STDIN
+Reading from STDIN
+^^^^^^^^^^^^^^^^^^
 
 File based pgloader sources can be loaded from the standard input, as in the
-following example:
+following example::
 
     pgloader --type csv                                         \
              --field "usps,geoid,aland,awater,aland_sqmi,awater_sqmi,intptlat,intptlong" \
@@ -38,14 +41,15 @@ following example:
 
 The dash (`-`) character as a source is used to mean *standard input*, as
 usual in Unix command lines. It's possible to stream compressed content to
-pgloader with this technique, using the Unix pipe:
+pgloader with this technique, using the Unix pipe::
 
     gunzip -c source.gz | pgloader --type csv ... - pgsql:///target?foo
 
-## Loading from CSV available through HTTP
+Loading from CSV available through HTTP
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 The same command as just above can also be run if the CSV file happens to be
-found on a remote HTTP location:
+found on a remote HTTP location::
 
     pgloader --type csv                                                     \
              --field "usps,geoid,aland,awater,aland_sqmi,awater_sqmi,intptlat,intptlong" \
@@ -62,7 +66,7 @@ notice). Also, in that case, we specify all the fields right into a single
 Again, the PostgreSQL target connection string must contain the *tablename*
 option and you have to ensure that the target table exists and may fit the
 data. Here's the SQL command used in that example in case you want to try it
-yourself:
+yourself::
 
     create table districts_longlat
     (
@@ -79,7 +83,8 @@ yourself:
 Also notice that the same command will work against an archived version of
 the same data.
 
-## Streaming CSV data from an HTTP compressed file
+Streaming CSV data from an HTTP compressed file
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Finally, it's important to note that pgloader first fetches the content from
 the HTTP URL it to a local file, then expand the archive when it's
@@ -89,7 +94,7 @@ In some cases, either because pgloader has no direct support for your
 archive format or maybe because expanding the archive is not feasible in
 your environment, you might want to *stream* the content straight from its
 remote location into PostgreSQL. Here's how to do that, using the old battle
-tested Unix Pipes trick:
+tested Unix Pipes trick::
 
     curl http://pgsql.tapoueh.org/temp/2013_Gaz_113CDs_national.txt.gz \
     | gunzip -c                                                        \
@@ -104,28 +109,31 @@ Now the OS will take care of the streaming and buffering between the network
 and the commands and pgloader will take care of streaming the data down to
 PostgreSQL.
 
-## Migrating from SQLite
+Migrating from SQLite
+^^^^^^^^^^^^^^^^^^^^^
 
 The following command will open the SQLite database, discover its tables
 definitions including indexes and foreign keys, migrate those definitions
 while *casting* the data type specifications to their PostgreSQL equivalent
-and then migrate the data over:
+and then migrate the data over::
 
     createdb newdb
     pgloader ./test/sqlite/sqlite.db postgresql:///newdb
 
-## Migrating from MySQL
+Migrating from MySQL
+^^^^^^^^^^^^^^^^^^^^
 
 Just create a database where to host the MySQL data and definitions and have
-pgloader do the migration for you in a single command line:
+pgloader do the migration for you in a single command line::
 
     createdb pagila
     pgloader mysql://user@localhost/sakila postgresql:///pagila
 
-## Fetching an archived DBF file from a HTTP remote location
+Fetching an archived DBF file from a HTTP remote location
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 It's possible for pgloader to download a file from HTTP, unarchive it, and
-only then open it to discover the schema then load the data:
+only then open it to discover the schema then load the data::
 
     createdb foo
     pgloader --type dbf http://www.insee.fr/fr/methodes/nomenclatures/cog/telechargement/2013/dbf/historiq2013.zip postgresql:///foo

docs/_build/html/_sources/tutorial/sqlite.rst.txt

@@ -1,13 +1,15 @@
-# Loading SQLite files with pgloader
+Loading SQLite files with pgloader
+----------------------------------
 
 The SQLite database is a respected solution to manage your data with. Its
 embeded nature makes it a source of migrations when a projects now needs to
 handle more concurrency, which [PostgreSQL](http://www.postgresql.org/) is
 very good at. pgloader can help you there.
 
-## In a Single Command Line
+In a Single Command Line
+^^^^^^^^^^^^^^^^^^^^^^^^
 
-You can 
+You can ::
 
     $ createdb chinook
     $ pgloader https://github.com/lerocha/chinook-database/raw/master/ChinookDatabase/DataSources/Chinook_Sqlite_AutoIncrementPKs.sqlite pgsql:///chinook
@@ -15,7 +17,7 @@ You can
 Done! All with the schema, data, constraints, primary keys and foreign keys,
 etc. We also see an error with the Chinook schema that contains several
 primary key definitions against the same table, which is not accepted by
-PostgreSQL:
+PostgreSQL::
 
     2017-06-20T16:18:59.019000+02:00 LOG Data errors in '/private/tmp/pgloader/'
     2017-06-20T16:18:59.236000+02:00 LOG Fetching 'https://github.com/lerocha/chinook-database/raw/master/ChinookDatabase/DataSources/Chinook_Sqlite_AutoIncrementPKs.sqlite'
@@ -57,10 +59,11 @@ PostgreSQL:
 You may need to have special cases to take care of tho. In advanced case you
 can use the pgloader command.
 
-## The Command
+The Command
+^^^^^^^^^^^
 
-To load data with [pgloader](http://pgloader.io/) you need to
-define in a *command* the operations in some details. Here's our command:
+To load data with [pgloader](http://pgloader.io/) you need to define in a
+*command* the operations in some details. Here's our command::
 
     load database
          from 'sqlite/Chinook_Sqlite_AutoIncrementPKs.sqlite'
@@ -70,17 +73,14 @@ define in a *command* the operations in some details. Here's our command:
     
       set work_mem to '16MB', maintenance_work_mem to '512 MB';
 
-You can see the full list of options in the
-[pgloader reference manual](pgloader.1.html), with a complete description
-of the options you see here.
-
 Note that here pgloader will benefit from the meta-data information found in
 the SQLite file to create a PostgreSQL database capable of hosting the data
 as described, then load the data.
 
-## Loading the data
+Loading the data
+^^^^^^^^^^^^^^^^
 
-Let's start the `pgloader` command with our `sqlite.load` command file:
+Let's start the `pgloader` command with our `sqlite.load` command file::
 
     $ pgloader sqlite.load
     ... LOG Starting pgloader, log system is ready.
@@ -117,7 +117,7 @@ Let's start the `pgloader` command with our `sqlite.load` command file:
     ----------------------  ---------  ---------  ---------  --------------
       Total streaming time      15607      15607          0          0.476s
 
-We can see that [http://pgloader.io](pgloader) did download the file from
+We can see that `pgloader <http://pgloader.io>`_ did download the file from
 its HTTP URL location then *unziped* it before loading it.
 
 Also, the *WARNING* messages we see here are expected as the PostgreSQL

docs/_build/html/_sources/tutorial/tutorial.rst.txt

@@ -0,0 +1,10 @@
+PgLoader Tutorial
+=================
+
+.. include:: quickstart.rst
+.. include:: csv.rst
+.. include:: fixed.rst
+.. include:: geolite.rst
+.. include:: dBase.rst
+.. include:: sqlite.rst
+.. include:: mysql.rst

docs/_build/html/_static/alabaster.css

@@ -0,0 +1,689 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+@import url("basic.css");
+
+/* -- page layout ----------------------------------------------------------- */
+
+body {
+    font-family: 'goudy old style', 'minion pro', 'bell mt', Georgia, 'Hiragino Mincho Pro', serif;
+    font-size: 17px;
+    background-color: #fff;
+    color: #000;
+    margin: 0;
+    padding: 0;
+}
+
+
+div.document {
+    width: 940px;
+    margin: 30px auto 0 auto;
+}
+
+div.documentwrapper {
+    float: left;
+    width: 100%;
+}
+
+div.bodywrapper {
+    margin: 0 0 0 220px;
+}
+
+div.sphinxsidebar {
+    width: 220px;
+    font-size: 14px;
+    line-height: 1.5;
+}
+
+hr {
+    border: 1px solid #B1B4B6;
+}
+
+div.body {
+    background-color: #fff;
+    color: #3E4349;
+    padding: 0 30px 0 30px;
+}
+
+div.body > .section {
+    text-align: left;
+}
+
+div.footer {
+    width: 940px;
+    margin: 20px auto 30px auto;
+    font-size: 14px;
+    color: #888;
+    text-align: right;
+}
+
+div.footer a {
+    color: #888;
+}
+
+p.caption {
+    font-family: inherit;
+    font-size: inherit;
+}
+
+
+
+div.sphinxsidebar a {
+    color: #444;
+    text-decoration: none;
+    border-bottom: 1px dotted #999;
+}
+
+div.sphinxsidebar a:hover {
+    border-bottom: 1px solid #999;
+}
+
+div.sphinxsidebarwrapper {
+    padding: 18px 10px;
+}
+
+div.sphinxsidebarwrapper p.logo {
+    padding: 0;
+    margin: -10px 0 0 0px;
+    text-align: center;
+}
+
+div.sphinxsidebarwrapper h1.logo {
+    margin-top: -10px;
+    text-align: center;
+    margin-bottom: 5px;
+    text-align: left;
+}
+
+div.sphinxsidebarwrapper h1.logo-name {
+    margin-top: 0px;
+}
+
+div.sphinxsidebarwrapper p.blurb {
+    margin-top: 0;
+    font-style: normal;
+}
+
+div.sphinxsidebar h3,
+div.sphinxsidebar h4 {
+    font-family: 'Garamond', 'Georgia', serif;
+    color: #444;
+    font-size: 24px;
+    font-weight: normal;
+    margin: 0 0 5px 0;
+    padding: 0;
+}
+
+div.sphinxsidebar h4 {
+    font-size: 20px;
+}
+
+div.sphinxsidebar h3 a {
+    color: #444;
+}
+
+div.sphinxsidebar p.logo a,
+div.sphinxsidebar h3 a,
+div.sphinxsidebar p.logo a:hover,
+div.sphinxsidebar h3 a:hover {
+    border: none;
+}
+
+div.sphinxsidebar p {
+    color: #555;
+    margin: 10px 0;
+}
+
+div.sphinxsidebar ul {
+    margin: 10px 0;
+    padding: 0;
+    color: #000;
+}
+
+div.sphinxsidebar ul li.toctree-l1 > a {
+    font-size: 120%;
+}
+
+div.sphinxsidebar ul li.toctree-l2 > a {
+    font-size: 110%;
+}
+
+div.sphinxsidebar input {
+    border: 1px solid #CCC;
+    font-family: 'goudy old style', 'minion pro', 'bell mt', Georgia, 'Hiragino Mincho Pro', serif;
+    font-size: 1em;
+}
+
+div.sphinxsidebar hr {
+    border: none;
+    height: 1px;
+    color: #AAA;
+    background: #AAA;
+
+    text-align: left;
+    margin-left: 0;
+    width: 50%;
+}
+
+/* -- body styles ----------------------------------------------------------- */
+
+a {
+    color: #004B6B;
+    text-decoration: underline;
+}
+
+a:hover {
+    color: #6D4100;
+    text-decoration: underline;
+}
+
+div.body h1,
+div.body h2,
+div.body h3,
+div.body h4,
+div.body h5,
+div.body h6 {
+    font-family: 'Garamond', 'Georgia', serif;
+    font-weight: normal;
+    margin: 30px 0px 10px 0px;
+    padding: 0;
+}
+
+div.body h1 { margin-top: 0; padding-top: 0; font-size: 240%; }
+div.body h2 { font-size: 180%; }
+div.body h3 { font-size: 150%; }
+div.body h4 { font-size: 130%; }
+div.body h5 { font-size: 100%; }
+div.body h6 { font-size: 100%; }
+
+a.headerlink {
+    color: #DDD;
+    padding: 0 4px;
+    text-decoration: none;
+}
+
+a.headerlink:hover {
+    color: #444;
+    background: #EAEAEA;
+}
+
+div.body p, div.body dd, div.body li {
+    line-height: 1.4em;
+}
+
+div.admonition {
+    margin: 20px 0px;
+    padding: 10px 30px;
+    background-color: #EEE;
+    border: 1px solid #CCC;
+}
+
+div.admonition tt.xref, div.admonition code.xref, div.admonition a tt {
+    background-color: #FBFBFB;
+    border-bottom: 1px solid #fafafa;
+}
+
+div.admonition p.admonition-title {
+    font-family: 'Garamond', 'Georgia', serif;
+    font-weight: normal;
+    font-size: 24px;
+    margin: 0 0 10px 0;
+    padding: 0;
+    line-height: 1;
+}
+
+div.admonition p.last {
+    margin-bottom: 0;
+}
+
+div.highlight {
+    background-color: #fff;
+}
+
+dt:target, .highlight {
+    background: #FAF3E8;
+}
+
+div.warning {
+    background-color: #FCC;
+    border: 1px solid #FAA;
+}
+
+div.danger {
+    background-color: #FCC;
+    border: 1px solid #FAA;
+    -moz-box-shadow: 2px 2px 4px #D52C2C;
+    -webkit-box-shadow: 2px 2px 4px #D52C2C;
+    box-shadow: 2px 2px 4px #D52C2C;
+}
+
+div.error {
+    background-color: #FCC;
+    border: 1px solid #FAA;
+    -moz-box-shadow: 2px 2px 4px #D52C2C;
+    -webkit-box-shadow: 2px 2px 4px #D52C2C;
+    box-shadow: 2px 2px 4px #D52C2C;
+}
+
+div.caution {
+    background-color: #FCC;
+    border: 1px solid #FAA;
+}
+
+div.attention {
+    background-color: #FCC;
+    border: 1px solid #FAA;
+}
+
+div.important {
+    background-color: #EEE;
+    border: 1px solid #CCC;
+}
+
+div.note {
+    background-color: #EEE;
+    border: 1px solid #CCC;
+}
+
+div.tip {
+    background-color: #EEE;
+    border: 1px solid #CCC;
+}
+
+div.hint {
+    background-color: #EEE;
+    border: 1px solid #CCC;
+}
+
+div.seealso {
+    background-color: #EEE;
+    border: 1px solid #CCC;
+}
+
+div.topic {
+    background-color: #EEE;
+}
+
+p.admonition-title {
+    display: inline;
+}
+
+p.admonition-title:after {
+    content: ":";
+}
+
+pre, tt, code {
+    font-family: 'Consolas', 'Menlo', 'Deja Vu Sans Mono', 'Bitstream Vera Sans Mono', monospace;
+    font-size: 0.9em;
+}
+
+.hll {
+    background-color: #FFC;
+    margin: 0 -12px;
+    padding: 0 12px;
+    display: block;
+}
+
+img.screenshot {
+}
+
+tt.descname, tt.descclassname, code.descname, code.descclassname {
+    font-size: 0.95em;
+}
+
+tt.descname, code.descname {
+    padding-right: 0.08em;
+}
+
+img.screenshot {
+    -moz-box-shadow: 2px 2px 4px #EEE;
+    -webkit-box-shadow: 2px 2px 4px #EEE;
+    box-shadow: 2px 2px 4px #EEE;
+}
+
+table.docutils {
+    border: 1px solid #888;
+    -moz-box-shadow: 2px 2px 4px #EEE;
+    -webkit-box-shadow: 2px 2px 4px #EEE;
+    box-shadow: 2px 2px 4px #EEE;
+}
+
+table.docutils td, table.docutils th {
+    border: 1px solid #888;
+    padding: 0.25em 0.7em;
+}
+
+table.field-list, table.footnote {
+    border: none;
+    -moz-box-shadow: none;
+    -webkit-box-shadow: none;
+    box-shadow: none;
+}
+
+table.footnote {
+    margin: 15px 0;
+    width: 100%;
+    border: 1px solid #EEE;
+    background: #FDFDFD;
+    font-size: 0.9em;
+}
+
+table.footnote + table.footnote {
+    margin-top: -15px;
+    border-top: none;
+}
+
+table.field-list th {
+    padding: 0 0.8em 0 0;
+}
+
+table.field-list td {
+    padding: 0;
+}
+
+table.field-list p {
+    margin-bottom: 0.8em;
+}
+
+/* Cloned from
+ * https://github.com/sphinx-doc/sphinx/commit/ef60dbfce09286b20b7385333d63a60321784e68
+ */
+.field-name {
+    -moz-hyphens: manual;
+    -ms-hyphens: manual;
+    -webkit-hyphens: manual;
+    hyphens: manual;
+}
+
+table.footnote td.label {
+    width: .1px;
+    padding: 0.3em 0 0.3em 0.5em;
+}
+
+table.footnote td {
+    padding: 0.3em 0.5em;
+}
+
+dl {
+    margin: 0;
+    padding: 0;
+}
+
+dl dd {
+    margin-left: 30px;
+}
+
+blockquote {
+    margin: 0 0 0 30px;
+    padding: 0;
+}
+
+ul, ol {
+    /* Matches the 30px from the narrow-screen "li > ul" selector below */
+    margin: 10px 0 10px 30px;
+    padding: 0;
+}
+
+pre {
+    background: #EEE;
+    padding: 7px 30px;
+    margin: 15px 0px;
+    line-height: 1.3em;
+}
+
+div.viewcode-block:target {
+    background: #ffd;
+}
+
+dl pre, blockquote pre, li pre {
+    margin-left: 0;
+    padding-left: 30px;
+}
+
+tt, code {
+    background-color: #ecf0f3;
+    color: #222;
+    /* padding: 1px 2px; */
+}
+
+tt.xref, code.xref, a tt {
+    background-color: #FBFBFB;
+    border-bottom: 1px solid #fff;
+}
+
+a.reference {
+    text-decoration: none;
+    border-bottom: 1px dotted #004B6B;
+}
+
+/* Don't put an underline on images */
+a.image-reference, a.image-reference:hover {
+    border-bottom: none;
+}
+
+a.reference:hover {
+    border-bottom: 1px solid #6D4100;
+}
+
+a.footnote-reference {
+    text-decoration: none;
+    font-size: 0.7em;
+    vertical-align: top;
+    border-bottom: 1px dotted #004B6B;
+}
+
+a.footnote-reference:hover {
+    border-bottom: 1px solid #6D4100;
+}
+
+a:hover tt, a:hover code {
+    background: #EEE;
+}
+
+
+@media screen and (max-width: 870px) {
+
+    div.sphinxsidebar {
+    	display: none;
+    }
+
+    div.document {
+       width: 100%;
+
+    }
+
+    div.documentwrapper {
+    	margin-left: 0;
+    	margin-top: 0;
+    	margin-right: 0;
+    	margin-bottom: 0;
+    }
+
+    div.bodywrapper {
+    	margin-top: 0;
+    	margin-right: 0;
+    	margin-bottom: 0;
+    	margin-left: 0;
+    }
+
+    ul {
+    	margin-left: 0;
+    }
+
+	li > ul {
+        /* Matches the 30px from the "ul, ol" selector above */
+		margin-left: 30px;
+	}
+
+    .document {
+    	width: auto;
+    }
+
+    .footer {
+    	width: auto;
+    }
+
+    .bodywrapper {
+    	margin: 0;
+    }
+
+    .footer {
+    	width: auto;
+    }
+
+    .github {
+        display: none;
+    }
+
+
+
+}
+
+
+
+@media screen and (max-width: 875px) {
+
+    body {
+        margin: 0;
+        padding: 20px 30px;
+    }
+
+    div.documentwrapper {
+        float: none;
+        background: #fff;
+    }
+
+    div.sphinxsidebar {
+        display: block;
+        float: none;
+        width: 102.5%;
+        margin: 50px -30px -20px -30px;
+        padding: 10px 20px;
+        background: #333;
+        color: #FFF;
+    }
+
+    div.sphinxsidebar h3, div.sphinxsidebar h4, div.sphinxsidebar p,
+    div.sphinxsidebar h3 a {
+        color: #fff;
+    }
+
+    div.sphinxsidebar a {
+        color: #AAA;
+    }
+
+    div.sphinxsidebar p.logo {
+        display: none;
+    }
+
+    div.document {
+        width: 100%;
+        margin: 0;
+    }
+
+    div.footer {
+        display: none;
+    }
+
+    div.bodywrapper {
+        margin: 0;
+    }
+
+    div.body {
+        min-height: 0;
+        padding: 0;
+    }
+
+    .rtd_doc_footer {
+        display: none;
+    }
+
+    .document {
+        width: auto;
+    }
+
+    .footer {
+        width: auto;
+    }
+
+    .footer {
+        width: auto;
+    }
+
+    .github {
+        display: none;
+    }
+}
+
+
+/* misc. */
+
+.revsys-inline {
+    display: none!important;
+}
+
+/* Make nested-list/multi-paragraph items look better in Releases changelog
+ * pages. Without this, docutils' magical list fuckery causes inconsistent
+ * formatting between different release sub-lists.
+ */
+div#changelog > div.section > ul > li > p:only-child {
+    margin-bottom: 0;
+}
+
+/* Hide fugly table cell borders in ..bibliography:: directive output */
+table.docutils.citation, table.docutils.citation td, table.docutils.citation th {
+  border: none;
+  /* Below needed in some edge cases; if not applied, bottom shadows appear */
+  -moz-box-shadow: none;
+  -webkit-box-shadow: none;
+  box-shadow: none;
+}

docs/_build/html/_static/basic.css

@@ -0,0 +1,643 @@
+/*
+ * basic.css
+ * ~~~~~~~~~
+ *
+ * Sphinx stylesheet -- basic theme.
+ *
+ * :copyright: Copyright 2007-2017 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+
+/* -- main layout ----------------------------------------------------------- */
+
+div.clearer {
+    clear: both;
+}
+
+/* -- relbar ---------------------------------------------------------------- */
+
+div.related {
+    width: 100%;
+    font-size: 90%;
+}
+
+div.related h3 {
+    display: none;
+}
+
+div.related ul {
+    margin: 0;
+    padding: 0 0 0 10px;
+    list-style: none;
+}
+
+div.related li {
+    display: inline;
+}
+
+div.related li.right {
+    float: right;
+    margin-right: 5px;
+}
+
+/* -- sidebar --------------------------------------------------------------- */
+
+div.sphinxsidebarwrapper {
+    padding: 10px 5px 0 10px;
+}
+
+div.sphinxsidebar {
+    float: left;
+    width: 230px;
+    margin-left: -100%;
+    font-size: 90%;
+    word-wrap: break-word;
+    overflow-wrap : break-word;
+}
+
+div.sphinxsidebar ul {
+    list-style: none;
+}
+
+div.sphinxsidebar ul ul,
+div.sphinxsidebar ul.want-points {
+    margin-left: 20px;
+    list-style: square;
+}
+
+div.sphinxsidebar ul ul {
+    margin-top: 0;
+    margin-bottom: 0;
+}
+
+div.sphinxsidebar form {
+    margin-top: 10px;
+}
+
+div.sphinxsidebar input {
+    border: 1px solid #98dbcc;
+    font-family: sans-serif;
+    font-size: 1em;
+}
+
+div.sphinxsidebar #searchbox input[type="text"] {
+    width: 170px;
+}
+
+img {
+    border: 0;
+    max-width: 100%;
+}
+
+/* -- search page ----------------------------------------------------------- */
+
+ul.search {
+    margin: 10px 0 0 20px;
+    padding: 0;
+}
+
+ul.search li {
+    padding: 5px 0 5px 20px;
+    background-image: url(file.png);
+    background-repeat: no-repeat;
+    background-position: 0 7px;
+}
+
+ul.search li a {
+    font-weight: bold;
+}
+
+ul.search li div.context {
+    color: #888;
+    margin: 2px 0 0 30px;
+    text-align: left;
+}
+
+ul.keywordmatches li.goodmatch a {
+    font-weight: bold;
+}
+
+/* -- index page ------------------------------------------------------------ */
+
+table.contentstable {
+    width: 90%;
+    margin-left: auto;
+    margin-right: auto;
+}
+
+table.contentstable p.biglink {
+    line-height: 150%;
+}
+
+a.biglink {
+    font-size: 1.3em;
+}
+
+span.linkdescr {
+    font-style: italic;
+    padding-top: 5px;
+    font-size: 90%;
+}
+
+/* -- general index --------------------------------------------------------- */
+
+table.indextable {
+    width: 100%;
+}
+
+table.indextable td {
+    text-align: left;
+    vertical-align: top;
+}
+
+table.indextable ul {
+    margin-top: 0;
+    margin-bottom: 0;
+    list-style-type: none;
+}
+
+table.indextable > tbody > tr > td > ul {
+    padding-left: 0em;
+}
+
+table.indextable tr.pcap {
+    height: 10px;
+}
+
+table.indextable tr.cap {
+    margin-top: 10px;
+    background-color: #f2f2f2;
+}
+
+img.toggler {
+    margin-right: 3px;
+    margin-top: 3px;
+    cursor: pointer;
+}
+
+div.modindex-jumpbox {
+    border-top: 1px solid #ddd;
+    border-bottom: 1px solid #ddd;
+    margin: 1em 0 1em 0;
+    padding: 0.4em;
+}
+
+div.genindex-jumpbox {
+    border-top: 1px solid #ddd;
+    border-bottom: 1px solid #ddd;
+    margin: 1em 0 1em 0;
+    padding: 0.4em;
+}
+
+/* -- domain module index --------------------------------------------------- */
+
+table.modindextable td {
+    padding: 2px;
+    border-collapse: collapse;
+}
+
+/* -- general body styles --------------------------------------------------- */
+
+div.body p, div.body dd, div.body li, div.body blockquote {
+    -moz-hyphens: auto;
+    -ms-hyphens: auto;
+    -webkit-hyphens: auto;
+    hyphens: auto;
+}
+
+a.headerlink {
+    visibility: hidden;
+}
+
+h1:hover > a.headerlink,
+h2:hover > a.headerlink,
+h3:hover > a.headerlink,
+h4:hover > a.headerlink,
+h5:hover > a.headerlink,
+h6:hover > a.headerlink,
+dt:hover > a.headerlink,
+caption:hover > a.headerlink,
+p.caption:hover > a.headerlink,
+div.code-block-caption:hover > a.headerlink {
+    visibility: visible;
+}
+
+div.body p.caption {
+    text-align: inherit;
+}
+
+div.body td {
+    text-align: left;
+}
+
+.first {
+    margin-top: 0 !important;
+}
+
+p.rubric {
+    margin-top: 30px;
+    font-weight: bold;
+}
+
+img.align-left, .figure.align-left, object.align-left {
+    clear: left;
+    float: left;
+    margin-right: 1em;
+}
+
+img.align-right, .figure.align-right, object.align-right {
+    clear: right;
+    float: right;
+    margin-left: 1em;
+}
+
+img.align-center, .figure.align-center, object.align-center {
+  display: block;
+  margin-left: auto;
+  margin-right: auto;
+}
+
+.align-left {
+    text-align: left;
+}
+
+.align-center {
+    text-align: center;
+}
+
+.align-right {
+    text-align: right;
+}
+
+/* -- sidebars -------------------------------------------------------------- */
+
+div.sidebar {
+    margin: 0 0 0.5em 1em;
+    border: 1px solid #ddb;
+    padding: 7px 7px 0 7px;
+    background-color: #ffe;
+    width: 40%;
+    float: right;
+}
+
+p.sidebar-title {
+    font-weight: bold;
+}
+
+/* -- topics ---------------------------------------------------------------- */
+
+div.topic {
+    border: 1px solid #ccc;
+    padding: 7px 7px 0 7px;
+    margin: 10px 0 10px 0;
+}
+
+p.topic-title {
+    font-size: 1.1em;
+    font-weight: bold;
+    margin-top: 10px;
+}
+
+/* -- admonitions ----------------------------------------------------------- */
+
+div.admonition {
+    margin-top: 10px;
+    margin-bottom: 10px;
+    padding: 7px;
+}
+
+div.admonition dt {
+    font-weight: bold;
+}
+
+div.admonition dl {
+    margin-bottom: 0;
+}
+
+p.admonition-title {
+    margin: 0px 10px 5px 0px;
+    font-weight: bold;
+}
+
+div.body p.centered {
+    text-align: center;
+    margin-top: 25px;
+}
+
+/* -- tables ---------------------------------------------------------------- */
+
+table.docutils {
+    border: 0;
+    border-collapse: collapse;
+}
+
+table caption span.caption-number {
+    font-style: italic;
+}
+
+table caption span.caption-text {
+}
+
+table.docutils td, table.docutils th {
+    padding: 1px 8px 1px 5px;
+    border-top: 0;
+    border-left: 0;
+    border-right: 0;
+    border-bottom: 1px solid #aaa;
+}
+
+table.footnote td, table.footnote th {
+    border: 0 !important;
+}
+
+th {
+    text-align: left;
+    padding-right: 5px;
+}
+
+table.citation {
+    border-left: solid 1px gray;
+    margin-left: 1px;
+}
+
+table.citation td {
+    border-bottom: none;
+}
+
+/* -- figures --------------------------------------------------------------- */
+
+div.figure {
+    margin: 0.5em;
+    padding: 0.5em;
+}
+
+div.figure p.caption {
+    padding: 0.3em;
+}
+
+div.figure p.caption span.caption-number {
+    font-style: italic;
+}
+
+div.figure p.caption span.caption-text {
+}
+
+/* -- field list styles ----------------------------------------------------- */
+
+table.field-list td, table.field-list th {
+    border: 0 !important;
+}
+
+.field-list ul {
+    margin: 0;
+    padding-left: 1em;
+}
+
+.field-list p {
+    margin: 0;
+}
+
+.field-name {
+    -moz-hyphens: manual;
+    -ms-hyphens: manual;
+    -webkit-hyphens: manual;
+    hyphens: manual;
+}
+
+/* -- other body styles ----------------------------------------------------- */
+
+ol.arabic {
+    list-style: decimal;
+}
+
+ol.loweralpha {
+    list-style: lower-alpha;
+}
+
+ol.upperalpha {
+    list-style: upper-alpha;
+}
+
+ol.lowerroman {
+    list-style: lower-roman;
+}
+
+ol.upperroman {
+    list-style: upper-roman;
+}
+
+dl {
+    margin-bottom: 15px;
+}
+
+dd p {
+    margin-top: 0px;
+}
+
+dd ul, dd table {
+    margin-bottom: 10px;
+}
+
+dd {
+    margin-top: 3px;
+    margin-bottom: 10px;
+    margin-left: 30px;
+}
+
+dt:target, span.highlighted {
+    background-color: #fbe54e;
+}
+
+rect.highlighted {
+    fill: #fbe54e;
+}
+
+dl.glossary dt {
+    font-weight: bold;
+    font-size: 1.1em;
+}
+
+.optional {
+    font-size: 1.3em;
+}
+
+.sig-paren {
+    font-size: larger;
+}
+
+.versionmodified {
+    font-style: italic;
+}
+
+.system-message {
+    background-color: #fda;
+    padding: 5px;
+    border: 3px solid red;
+}
+
+.footnote:target  {
+    background-color: #ffa;
+}
+
+.line-block {
+    display: block;
+    margin-top: 1em;
+    margin-bottom: 1em;
+}
+
+.line-block .line-block {
+    margin-top: 0;
+    margin-bottom: 0;
+    margin-left: 1.5em;
+}
+
+.guilabel, .menuselection {
+    font-family: sans-serif;
+}
+
+.accelerator {
+    text-decoration: underline;
+}
+
+.classifier {
+    font-style: oblique;
+}
+
+abbr, acronym {
+    border-bottom: dotted 1px;
+    cursor: help;
+}
+
+/* -- code displays --------------------------------------------------------- */
+
+pre {
+    overflow: auto;
+    overflow-y: hidden;  /* fixes display issues on Chrome browsers */
+}
+
+span.pre {
+    -moz-hyphens: none;
+    -ms-hyphens: none;
+    -webkit-hyphens: none;
+    hyphens: none;
+}
+
+td.linenos pre {
+    padding: 5px 0px;
+    border: 0;
+    background-color: transparent;
+    color: #aaa;
+}
+
+table.highlighttable {
+    margin-left: 0.5em;
+}
+
+table.highlighttable td {
+    padding: 0 0.5em 0 0.5em;
+}
+
+div.code-block-caption {
+    padding: 2px 5px;
+    font-size: small;
+}
+
+div.code-block-caption code {
+    background-color: transparent;
+}
+
+div.code-block-caption + div > div.highlight > pre {
+    margin-top: 0;
+}
+
+div.code-block-caption span.caption-number {
+    padding: 0.1em 0.3em;
+    font-style: italic;
+}
+
+div.code-block-caption span.caption-text {
+}
+
+div.literal-block-wrapper {
+    padding: 1em 1em 0;
+}
+
+div.literal-block-wrapper div.highlight {
+    margin: 0;
+}
+
+code.descname {
+    background-color: transparent;
+    font-weight: bold;
+    font-size: 1.2em;
+}
+
+code.descclassname {
+    background-color: transparent;
+}
+
+code.xref, a code {
+    background-color: transparent;
+    font-weight: bold;
+}
+
+h1 code, h2 code, h3 code, h4 code, h5 code, h6 code {
+    background-color: transparent;
+}
+
+.viewcode-link {
+    float: right;
+}
+
+.viewcode-back {
+    float: right;
+    font-family: sans-serif;
+}
+
+div.viewcode-block:target {
+    margin: -1px -10px;
+    padding: 0 10px;
+}
+
+/* -- math display ---------------------------------------------------------- */
+
+img.math {
+    vertical-align: middle;
+}
+
+div.body div.math p {
+    text-align: center;
+}
+
+span.eqno {
+    float: right;
+}
+
+span.eqno a.headerlink {
+    position: relative;
+    left: 0px;
+    z-index: 1;
+}
+
+div.math:hover a.headerlink {
+    visibility: visible;
+}
+
+/* -- printout stylesheet --------------------------------------------------- */
+
+@media print {
+    div.document,
+    div.documentwrapper,
+    div.bodywrapper {
+        margin: 0 !important;
+        width: 100%;
+    }
+
+    div.sphinxsidebar,
+    div.related,
+    div.footer,
+    #top-link {
+        display: none;
+    }
+}

docs/_build/html/_static/custom.css

@@ -0,0 +1 @@
+/* This file intentionally left blank. */

docs/_build/html/_static/doctools.js

@@ -0,0 +1,311 @@
+/*
+ * doctools.js
+ * ~~~~~~~~~~~
+ *
+ * Sphinx JavaScript utilities for all documentation.
+ *
+ * :copyright: Copyright 2007-2017 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+
+/**
+ * select a different prefix for underscore
+ */
+$u = _.noConflict();
+
+/**
+ * make the code below compatible with browsers without
+ * an installed firebug like debugger
+if (!window.console || !console.firebug) {
+  var names = ["log", "debug", "info", "warn", "error", "assert", "dir",
+    "dirxml", "group", "groupEnd", "time", "timeEnd", "count", "trace",
+    "profile", "profileEnd"];
+  window.console = {};
+  for (var i = 0; i < names.length; ++i)
+    window.console[names[i]] = function() {};
+}
+ */
+
+/**
+ * small helper function to urldecode strings
+ */
+jQuery.urldecode = function(x) {
+  return decodeURIComponent(x).replace(/\+/g, ' ');
+};
+
+/**
+ * small helper function to urlencode strings
+ */
+jQuery.urlencode = encodeURIComponent;
+
+/**
+ * This function returns the parsed url parameters of the
+ * current request. Multiple values per key are supported,
+ * it will always return arrays of strings for the value parts.
+ */
+jQuery.getQueryParameters = function(s) {
+  if (typeof s === 'undefined')
+    s = document.location.search;
+  var parts = s.substr(s.indexOf('?') + 1).split('&');
+  var result = {};
+  for (var i = 0; i < parts.length; i++) {
+    var tmp = parts[i].split('=', 2);
+    var key = jQuery.urldecode(tmp[0]);
+    var value = jQuery.urldecode(tmp[1]);
+    if (key in result)
+      result[key].push(value);
+    else
+      result[key] = [value];
+  }
+  return result;
+};
+
+/**
+ * highlight a given string on a jquery object by wrapping it in
+ * span elements with the given class name.
+ */
+jQuery.fn.highlightText = function(text, className) {
+  function highlight(node, addItems) {
+    if (node.nodeType === 3) {
+      var val = node.nodeValue;
+      var pos = val.toLowerCase().indexOf(text);
+      if (pos >= 0 && !jQuery(node.parentNode).hasClass(className)) {
+        var span;
+        var isInSVG = jQuery(node).closest("body, svg, foreignObject").is("svg");
+        if (isInSVG) {
+          span = document.createElementNS("http://www.w3.org/2000/svg", "tspan");
+        } else {
+          span = document.createElement("span");
+          span.className = className;
+        }
+        span.appendChild(document.createTextNode(val.substr(pos, text.length)));
+        node.parentNode.insertBefore(span, node.parentNode.insertBefore(
+          document.createTextNode(val.substr(pos + text.length)),
+          node.nextSibling));
+        node.nodeValue = val.substr(0, pos);
+        if (isInSVG) {
+          var bbox = span.getBBox();
+          var rect = document.createElementNS("http://www.w3.org/2000/svg", "rect");
+       	  rect.x.baseVal.value = bbox.x;
+          rect.y.baseVal.value = bbox.y;
+          rect.width.baseVal.value = bbox.width;
+          rect.height.baseVal.value = bbox.height;
+          rect.setAttribute('class', className);
+          var parentOfText = node.parentNode.parentNode;
+          addItems.push({
+              "parent": node.parentNode,
+              "target": rect});
+        }
+      }
+    }
+    else if (!jQuery(node).is("button, select, textarea")) {
+      jQuery.each(node.childNodes, function() {
+        highlight(this, addItems);
+      });
+    }
+  }
+  var addItems = [];
+  var result = this.each(function() {
+    highlight(this, addItems);
+  });
+  for (var i = 0; i < addItems.length; ++i) {
+    jQuery(addItems[i].parent).before(addItems[i].target);
+  }
+  return result;
+};
+
+/*
+ * backward compatibility for jQuery.browser
+ * This will be supported until firefox bug is fixed.
+ */
+if (!jQuery.browser) {
+  jQuery.uaMatch = function(ua) {
+    ua = ua.toLowerCase();
+
+    var match = /(chrome)[ \/]([\w.]+)/.exec(ua) ||
+      /(webkit)[ \/]([\w.]+)/.exec(ua) ||
+      /(opera)(?:.*version|)[ \/]([\w.]+)/.exec(ua) ||
+      /(msie) ([\w.]+)/.exec(ua) ||
+      ua.indexOf("compatible") < 0 && /(mozilla)(?:.*? rv:([\w.]+)|)/.exec(ua) ||
+      [];
+
+    return {
+      browser: match[ 1 ] || "",
+      version: match[ 2 ] || "0"
+    };
+  };
+  jQuery.browser = {};
+  jQuery.browser[jQuery.uaMatch(navigator.userAgent).browser] = true;
+}
+
+/**
+ * Small JavaScript module for the documentation.
+ */
+var Documentation = {
+
+  init : function() {
+    this.fixFirefoxAnchorBug();
+    this.highlightSearchWords();
+    this.initIndexTable();
+    
+  },
+
+  /**
+   * i18n support
+   */
+  TRANSLATIONS : {},
+  PLURAL_EXPR : function(n) { return n === 1 ? 0 : 1; },
+  LOCALE : 'unknown',
+
+  // gettext and ngettext don't access this so that the functions
+  // can safely bound to a different name (_ = Documentation.gettext)
+  gettext : function(string) {
+    var translated = Documentation.TRANSLATIONS[string];
+    if (typeof translated === 'undefined')
+      return string;
+    return (typeof translated === 'string') ? translated : translated[0];
+  },
+
+  ngettext : function(singular, plural, n) {
+    var translated = Documentation.TRANSLATIONS[singular];
+    if (typeof translated === 'undefined')
+      return (n == 1) ? singular : plural;
+    return translated[Documentation.PLURALEXPR(n)];
+  },
+
+  addTranslations : function(catalog) {
+    for (var key in catalog.messages)
+      this.TRANSLATIONS[key] = catalog.messages[key];
+    this.PLURAL_EXPR = new Function('n', 'return +(' + catalog.plural_expr + ')');
+    this.LOCALE = catalog.locale;
+  },
+
+  /**
+   * add context elements like header anchor links
+   */
+  addContextElements : function() {
+    $('div[id] > :header:first').each(function() {
+      $('<a class="headerlink">\u00B6</a>').
+      attr('href', '#' + this.id).
+      attr('title', _('Permalink to this headline')).
+      appendTo(this);
+    });
+    $('dt[id]').each(function() {
+      $('<a class="headerlink">\u00B6</a>').
+      attr('href', '#' + this.id).
+      attr('title', _('Permalink to this definition')).
+      appendTo(this);
+    });
+  },
+
+  /**
+   * workaround a firefox stupidity
+   * see: https://bugzilla.mozilla.org/show_bug.cgi?id=645075
+   */
+  fixFirefoxAnchorBug : function() {
+    if (document.location.hash)
+      window.setTimeout(function() {
+        document.location.href += '';
+      }, 10);
+  },
+
+  /**
+   * highlight the search words provided in the url in the text
+   */
+  highlightSearchWords : function() {
+    var params = $.getQueryParameters();
+    var terms = (params.highlight) ? params.highlight[0].split(/\s+/) : [];
+    if (terms.length) {
+      var body = $('div.body');
+      if (!body.length) {
+        body = $('body');
+      }
+      window.setTimeout(function() {
+        $.each(terms, function() {
+          body.highlightText(this.toLowerCase(), 'highlighted');
+        });
+      }, 10);
+      $('<p class="highlight-link"><a href="javascript:Documentation.' +
+        'hideSearchWords()">' + _('Hide Search Matches') + '</a></p>')
+          .appendTo($('#searchbox'));
+    }
+  },
+
+  /**
+   * init the domain index toggle buttons
+   */
+  initIndexTable : function() {
+    var togglers = $('img.toggler').click(function() {
+      var src = $(this).attr('src');
+      var idnum = $(this).attr('id').substr(7);
+      $('tr.cg-' + idnum).toggle();
+      if (src.substr(-9) === 'minus.png')
+        $(this).attr('src', src.substr(0, src.length-9) + 'plus.png');
+      else
+        $(this).attr('src', src.substr(0, src.length-8) + 'minus.png');
+    }).css('display', '');
+    if (DOCUMENTATION_OPTIONS.COLLAPSE_INDEX) {
+        togglers.click();
+    }
+  },
+
+  /**
+   * helper function to hide the search marks again
+   */
+  hideSearchWords : function() {
+    $('#searchbox .highlight-link').fadeOut(300);
+    $('span.highlighted').removeClass('highlighted');
+  },
+
+  /**
+   * make the url absolute
+   */
+  makeURL : function(relativeURL) {
+    return DOCUMENTATION_OPTIONS.URL_ROOT + '/' + relativeURL;
+  },
+
+  /**
+   * get the current relative url
+   */
+  getCurrentURL : function() {
+    var path = document.location.pathname;
+    var parts = path.split(/\//);
+    $.each(DOCUMENTATION_OPTIONS.URL_ROOT.split(/\//), function() {
+      if (this === '..')
+        parts.pop();
+    });
+    var url = parts.join('/');
+    return path.substring(url.lastIndexOf('/') + 1, path.length - 1);
+  },
+
+  initOnKeyListeners: function() {
+    $(document).keyup(function(event) {
+      var activeElementType = document.activeElement.tagName;
+      // don't navigate when in search box or textarea
+      if (activeElementType !== 'TEXTAREA' && activeElementType !== 'INPUT' && activeElementType !== 'SELECT') {
+        switch (event.keyCode) {
+          case 37: // left
+            var prevHref = $('link[rel="prev"]').prop('href');
+            if (prevHref) {
+              window.location.href = prevHref;
+              return false;
+            }
+          case 39: // right
+            var nextHref = $('link[rel="next"]').prop('href');
+            if (nextHref) {
+              window.location.href = nextHref;
+              return false;
+            }
+        }
+      }
+    });
+  }
+};
+
+// quick alias for translations
+_ = Documentation.gettext;
+
+$(document).ready(function() {
+  Documentation.init();
+});

docs/_build/html/_static/pygments.css

@@ -0,0 +1,69 @@
+.highlight .hll { background-color: #ffffcc }
+.highlight  { background: #eeffcc; }
+.highlight .c { color: #408090; font-style: italic } /* Comment */
+.highlight .err { border: 1px solid #FF0000 } /* Error */
+.highlight .k { color: #007020; font-weight: bold } /* Keyword */
+.highlight .o { color: #666666 } /* Operator */
+.highlight .ch { color: #408090; font-style: italic } /* Comment.Hashbang */
+.highlight .cm { color: #408090; font-style: italic } /* Comment.Multiline */
+.highlight .cp { color: #007020 } /* Comment.Preproc */
+.highlight .cpf { color: #408090; font-style: italic } /* Comment.PreprocFile */
+.highlight .c1 { color: #408090; font-style: italic } /* Comment.Single */
+.highlight .cs { color: #408090; background-color: #fff0f0 } /* Comment.Special */
+.highlight .gd { color: #A00000 } /* Generic.Deleted */
+.highlight .ge { font-style: italic } /* Generic.Emph */
+.highlight .gr { color: #FF0000 } /* Generic.Error */
+.highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */
+.highlight .gi { color: #00A000 } /* Generic.Inserted */
+.highlight .go { color: #333333 } /* Generic.Output */
+.highlight .gp { color: #c65d09; font-weight: bold } /* Generic.Prompt */
+.highlight .gs { font-weight: bold } /* Generic.Strong */
+.highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */
+.highlight .gt { color: #0044DD } /* Generic.Traceback */
+.highlight .kc { color: #007020; font-weight: bold } /* Keyword.Constant */
+.highlight .kd { color: #007020; font-weight: bold } /* Keyword.Declaration */
+.highlight .kn { color: #007020; font-weight: bold } /* Keyword.Namespace */
+.highlight .kp { color: #007020 } /* Keyword.Pseudo */
+.highlight .kr { color: #007020; font-weight: bold } /* Keyword.Reserved */
+.highlight .kt { color: #902000 } /* Keyword.Type */
+.highlight .m { color: #208050 } /* Literal.Number */
+.highlight .s { color: #4070a0 } /* Literal.String */
+.highlight .na { color: #4070a0 } /* Name.Attribute */
+.highlight .nb { color: #007020 } /* Name.Builtin */
+.highlight .nc { color: #0e84b5; font-weight: bold } /* Name.Class */
+.highlight .no { color: #60add5 } /* Name.Constant */
+.highlight .nd { color: #555555; font-weight: bold } /* Name.Decorator */
+.highlight .ni { color: #d55537; font-weight: bold } /* Name.Entity */
+.highlight .ne { color: #007020 } /* Name.Exception */
+.highlight .nf { color: #06287e } /* Name.Function */
+.highlight .nl { color: #002070; font-weight: bold } /* Name.Label */
+.highlight .nn { color: #0e84b5; font-weight: bold } /* Name.Namespace */
+.highlight .nt { color: #062873; font-weight: bold } /* Name.Tag */
+.highlight .nv { color: #bb60d5 } /* Name.Variable */
+.highlight .ow { color: #007020; font-weight: bold } /* Operator.Word */
+.highlight .w { color: #bbbbbb } /* Text.Whitespace */
+.highlight .mb { color: #208050 } /* Literal.Number.Bin */
+.highlight .mf { color: #208050 } /* Literal.Number.Float */
+.highlight .mh { color: #208050 } /* Literal.Number.Hex */
+.highlight .mi { color: #208050 } /* Literal.Number.Integer */
+.highlight .mo { color: #208050 } /* Literal.Number.Oct */
+.highlight .sa { color: #4070a0 } /* Literal.String.Affix */
+.highlight .sb { color: #4070a0 } /* Literal.String.Backtick */
+.highlight .sc { color: #4070a0 } /* Literal.String.Char */
+.highlight .dl { color: #4070a0 } /* Literal.String.Delimiter */
+.highlight .sd { color: #4070a0; font-style: italic } /* Literal.String.Doc */
+.highlight .s2 { color: #4070a0 } /* Literal.String.Double */
+.highlight .se { color: #4070a0; font-weight: bold } /* Literal.String.Escape */
+.highlight .sh { color: #4070a0 } /* Literal.String.Heredoc */
+.highlight .si { color: #70a0d0; font-style: italic } /* Literal.String.Interpol */
+.highlight .sx { color: #c65d09 } /* Literal.String.Other */
+.highlight .sr { color: #235388 } /* Literal.String.Regex */
+.highlight .s1 { color: #4070a0 } /* Literal.String.Single */
+.highlight .ss { color: #517918 } /* Literal.String.Symbol */
+.highlight .bp { color: #007020 } /* Name.Builtin.Pseudo */
+.highlight .fm { color: #06287e } /* Name.Function.Magic */
+.highlight .vc { color: #bb60d5 } /* Name.Variable.Class */
+.highlight .vg { color: #bb60d5 } /* Name.Variable.Global */
+.highlight .vi { color: #bb60d5 } /* Name.Variable.Instance */
+.highlight .vm { color: #bb60d5 } /* Name.Variable.Magic */
+.highlight .il { color: #208050 } /* Literal.Number.Integer.Long */

docs/_build/html/_static/searchtools.js

@@ -0,0 +1,761 @@
+/*
+ * searchtools.js_t
+ * ~~~~~~~~~~~~~~~~
+ *
+ * Sphinx JavaScript utilities for the full-text search.
+ *
+ * :copyright: Copyright 2007-2017 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+
+
+/* Non-minified version JS is _stemmer.js if file is provided */ 
+/**
+ * Porter Stemmer
+ */
+var Stemmer = function() {
+
+  var step2list = {
+    ational: 'ate',
+    tional: 'tion',
+    enci: 'ence',
+    anci: 'ance',
+    izer: 'ize',
+    bli: 'ble',
+    alli: 'al',
+    entli: 'ent',
+    eli: 'e',
+    ousli: 'ous',
+    ization: 'ize',
+    ation: 'ate',
+    ator: 'ate',
+    alism: 'al',
+    iveness: 'ive',
+    fulness: 'ful',
+    ousness: 'ous',
+    aliti: 'al',
+    iviti: 'ive',
+    biliti: 'ble',
+    logi: 'log'
+  };
+
+  var step3list = {
+    icate: 'ic',
+    ative: '',
+    alize: 'al',
+    iciti: 'ic',
+    ical: 'ic',
+    ful: '',
+    ness: ''
+  };
+
+  var c = "[^aeiou]";          // consonant
+  var v = "[aeiouy]";          // vowel
+  var C = c + "[^aeiouy]*";    // consonant sequence
+  var V = v + "[aeiou]*";      // vowel sequence
+
+  var mgr0 = "^(" + C + ")?" + V + C;                      // [C]VC... is m>0
+  var meq1 = "^(" + C + ")?" + V + C + "(" + V + ")?$";    // [C]VC[V] is m=1
+  var mgr1 = "^(" + C + ")?" + V + C + V + C;              // [C]VCVC... is m>1
+  var s_v   = "^(" + C + ")?" + v;                         // vowel in stem
+
+  this.stemWord = function (w) {
+    var stem;
+    var suffix;
+    var firstch;
+    var origword = w;
+
+    if (w.length < 3)
+      return w;
+
+    var re;
+    var re2;
+    var re3;
+    var re4;
+
+    firstch = w.substr(0,1);
+    if (firstch == "y")
+      w = firstch.toUpperCase() + w.substr(1);
+
+    // Step 1a
+    re = /^(.+?)(ss|i)es$/;
+    re2 = /^(.+?)([^s])s$/;
+
+    if (re.test(w))
+      w = w.replace(re,"$1$2");
+    else if (re2.test(w))
+      w = w.replace(re2,"$1$2");
+
+    // Step 1b
+    re = /^(.+?)eed$/;
+    re2 = /^(.+?)(ed|ing)$/;
+    if (re.test(w)) {
+      var fp = re.exec(w);
+      re = new RegExp(mgr0);
+      if (re.test(fp[1])) {
+        re = /.$/;
+        w = w.replace(re,"");
+      }
+    }
+    else if (re2.test(w)) {
+      var fp = re2.exec(w);
+      stem = fp[1];
+      re2 = new RegExp(s_v);
+      if (re2.test(stem)) {
+        w = stem;
+        re2 = /(at|bl|iz)$/;
+        re3 = new RegExp("([^aeiouylsz])\\1$");
+        re4 = new RegExp("^" + C + v + "[^aeiouwxy]$");
+        if (re2.test(w))
+          w = w + "e";
+        else if (re3.test(w)) {
+          re = /.$/;
+          w = w.replace(re,"");
+        }
+        else if (re4.test(w))
+          w = w + "e";
+      }
+    }
+
+    // Step 1c
+    re = /^(.+?)y$/;
+    if (re.test(w)) {
+      var fp = re.exec(w);
+      stem = fp[1];
+      re = new RegExp(s_v);
+      if (re.test(stem))
+        w = stem + "i";
+    }
+
+    // Step 2
+    re = /^(.+?)(ational|tional|enci|anci|izer|bli|alli|entli|eli|ousli|ization|ation|ator|alism|iveness|fulness|ousness|aliti|iviti|biliti|logi)$/;
+    if (re.test(w)) {
+      var fp = re.exec(w);
+      stem = fp[1];
+      suffix = fp[2];
+      re = new RegExp(mgr0);
+      if (re.test(stem))
+        w = stem + step2list[suffix];
+    }
+
+    // Step 3
+    re = /^(.+?)(icate|ative|alize|iciti|ical|ful|ness)$/;
+    if (re.test(w)) {
+      var fp = re.exec(w);
+      stem = fp[1];
+      suffix = fp[2];
+      re = new RegExp(mgr0);
+      if (re.test(stem))
+        w = stem + step3list[suffix];
+    }
+
+    // Step 4
+    re = /^(.+?)(al|ance|ence|er|ic|able|ible|ant|ement|ment|ent|ou|ism|ate|iti|ous|ive|ize)$/;
+    re2 = /^(.+?)(s|t)(ion)$/;
+    if (re.test(w)) {
+      var fp = re.exec(w);
+      stem = fp[1];
+      re = new RegExp(mgr1);
+      if (re.test(stem))
+        w = stem;
+    }
+    else if (re2.test(w)) {
+      var fp = re2.exec(w);
+      stem = fp[1] + fp[2];
+      re2 = new RegExp(mgr1);
+      if (re2.test(stem))
+        w = stem;
+    }
+
+    // Step 5
+    re = /^(.+?)e$/;
+    if (re.test(w)) {
+      var fp = re.exec(w);
+      stem = fp[1];
+      re = new RegExp(mgr1);
+      re2 = new RegExp(meq1);
+      re3 = new RegExp("^" + C + v + "[^aeiouwxy]$");
+      if (re.test(stem) || (re2.test(stem) && !(re3.test(stem))))
+        w = stem;
+    }
+    re = /ll$/;
+    re2 = new RegExp(mgr1);
+    if (re.test(w) && re2.test(w)) {
+      re = /.$/;
+      w = w.replace(re,"");
+    }
+
+    // and turn initial Y back to y
+    if (firstch == "y")
+      w = firstch.toLowerCase() + w.substr(1);
+    return w;
+  }
+}
+
+
+
+/**
+ * Simple result scoring code.
+ */
+var Scorer = {
+  // Implement the following function to further tweak the score for each result
+  // The function takes a result array [filename, title, anchor, descr, score]
+  // and returns the new score.
+  /*
+  score: function(result) {
+    return result[4];
+  },
+  */
+
+  // query matches the full name of an object
+  objNameMatch: 11,
+  // or matches in the last dotted part of the object name
+  objPartialMatch: 6,
+  // Additive scores depending on the priority of the object
+  objPrio: {0:  15,   // used to be importantResults
+            1:  5,   // used to be objectResults
+            2: -5},  // used to be unimportantResults
+  //  Used when the priority is not in the mapping.
+  objPrioDefault: 0,
+
+  // query found in title
+  title: 15,
+  // query found in terms
+  term: 5
+};
+
+
+
+
+
+var splitChars = (function() {
+    var result = {};
+    var singles = [96, 180, 187, 191, 215, 247, 749, 885, 903, 907, 909, 930, 1014, 1648,
+         1748, 1809, 2416, 2473, 2481, 2526, 2601, 2609, 2612, 2615, 2653, 2702,
+         2706, 2729, 2737, 2740, 2857, 2865, 2868, 2910, 2928, 2948, 2961, 2971,
+         2973, 3085, 3089, 3113, 3124, 3213, 3217, 3241, 3252, 3295, 3341, 3345,
+         3369, 3506, 3516, 3633, 3715, 3721, 3736, 3744, 3748, 3750, 3756, 3761,
+         3781, 3912, 4239, 4347, 4681, 4695, 4697, 4745, 4785, 4799, 4801, 4823,
+         4881, 5760, 5901, 5997, 6313, 7405, 8024, 8026, 8028, 8030, 8117, 8125,
+         8133, 8181, 8468, 8485, 8487, 8489, 8494, 8527, 11311, 11359, 11687, 11695,
+         11703, 11711, 11719, 11727, 11735, 12448, 12539, 43010, 43014, 43019, 43587,
+         43696, 43713, 64286, 64297, 64311, 64317, 64319, 64322, 64325, 65141];
+    var i, j, start, end;
+    for (i = 0; i < singles.length; i++) {
+        result[singles[i]] = true;
+    }
+    var ranges = [[0, 47], [58, 64], [91, 94], [123, 169], [171, 177], [182, 184], [706, 709],
+         [722, 735], [741, 747], [751, 879], [888, 889], [894, 901], [1154, 1161],
+         [1318, 1328], [1367, 1368], [1370, 1376], [1416, 1487], [1515, 1519], [1523, 1568],
+         [1611, 1631], [1642, 1645], [1750, 1764], [1767, 1773], [1789, 1790], [1792, 1807],
+         [1840, 1868], [1958, 1968], [1970, 1983], [2027, 2035], [2038, 2041], [2043, 2047],
+         [2070, 2073], [2075, 2083], [2085, 2087], [2089, 2307], [2362, 2364], [2366, 2383],
+         [2385, 2391], [2402, 2405], [2419, 2424], [2432, 2436], [2445, 2446], [2449, 2450],
+         [2483, 2485], [2490, 2492], [2494, 2509], [2511, 2523], [2530, 2533], [2546, 2547],
+         [2554, 2564], [2571, 2574], [2577, 2578], [2618, 2648], [2655, 2661], [2672, 2673],
+         [2677, 2692], [2746, 2748], [2750, 2767], [2769, 2783], [2786, 2789], [2800, 2820],
+         [2829, 2830], [2833, 2834], [2874, 2876], [2878, 2907], [2914, 2917], [2930, 2946],
+         [2955, 2957], [2966, 2968], [2976, 2978], [2981, 2983], [2987, 2989], [3002, 3023],
+         [3025, 3045], [3059, 3076], [3130, 3132], [3134, 3159], [3162, 3167], [3170, 3173],
+         [3184, 3191], [3199, 3204], [3258, 3260], [3262, 3293], [3298, 3301], [3312, 3332],
+         [3386, 3388], [3390, 3423], [3426, 3429], [3446, 3449], [3456, 3460], [3479, 3481],
+         [3518, 3519], [3527, 3584], [3636, 3647], [3655, 3663], [3674, 3712], [3717, 3718],
+         [3723, 3724], [3726, 3731], [3752, 3753], [3764, 3772], [3774, 3775], [3783, 3791],
+         [3802, 3803], [3806, 3839], [3841, 3871], [3892, 3903], [3949, 3975], [3980, 4095],
+         [4139, 4158], [4170, 4175], [4182, 4185], [4190, 4192], [4194, 4196], [4199, 4205],
+         [4209, 4212], [4226, 4237], [4250, 4255], [4294, 4303], [4349, 4351], [4686, 4687],
+         [4702, 4703], [4750, 4751], [4790, 4791], [4806, 4807], [4886, 4887], [4955, 4968],
+         [4989, 4991], [5008, 5023], [5109, 5120], [5741, 5742], [5787, 5791], [5867, 5869],
+         [5873, 5887], [5906, 5919], [5938, 5951], [5970, 5983], [6001, 6015], [6068, 6102],
+         [6104, 6107], [6109, 6111], [6122, 6127], [6138, 6159], [6170, 6175], [6264, 6271],
+         [6315, 6319], [6390, 6399], [6429, 6469], [6510, 6511], [6517, 6527], [6572, 6592],
+         [6600, 6607], [6619, 6655], [6679, 6687], [6741, 6783], [6794, 6799], [6810, 6822],
+         [6824, 6916], [6964, 6980], [6988, 6991], [7002, 7042], [7073, 7085], [7098, 7167],
+         [7204, 7231], [7242, 7244], [7294, 7400], [7410, 7423], [7616, 7679], [7958, 7959],
+         [7966, 7967], [8006, 8007], [8014, 8015], [8062, 8063], [8127, 8129], [8141, 8143],
+         [8148, 8149], [8156, 8159], [8173, 8177], [8189, 8303], [8306, 8307], [8314, 8318],
+         [8330, 8335], [8341, 8449], [8451, 8454], [8456, 8457], [8470, 8472], [8478, 8483],
+         [8506, 8507], [8512, 8516], [8522, 8525], [8586, 9311], [9372, 9449], [9472, 10101],
+         [10132, 11263], [11493, 11498], [11503, 11516], [11518, 11519], [11558, 11567],
+         [11622, 11630], [11632, 11647], [11671, 11679], [11743, 11822], [11824, 12292],
+         [12296, 12320], [12330, 12336], [12342, 12343], [12349, 12352], [12439, 12444],
+         [12544, 12548], [12590, 12592], [12687, 12689], [12694, 12703], [12728, 12783],
+         [12800, 12831], [12842, 12880], [12896, 12927], [12938, 12976], [12992, 13311],
+         [19894, 19967], [40908, 40959], [42125, 42191], [42238, 42239], [42509, 42511],
+         [42540, 42559], [42592, 42593], [42607, 42622], [42648, 42655], [42736, 42774],
+         [42784, 42785], [42889, 42890], [42893, 43002], [43043, 43055], [43062, 43071],
+         [43124, 43137], [43188, 43215], [43226, 43249], [43256, 43258], [43260, 43263],
+         [43302, 43311], [43335, 43359], [43389, 43395], [43443, 43470], [43482, 43519],
+         [43561, 43583], [43596, 43599], [43610, 43615], [43639, 43641], [43643, 43647],
+         [43698, 43700], [43703, 43704], [43710, 43711], [43715, 43738], [43742, 43967],
+         [44003, 44015], [44026, 44031], [55204, 55215], [55239, 55242], [55292, 55295],
+         [57344, 63743], [64046, 64047], [64110, 64111], [64218, 64255], [64263, 64274],
+         [64280, 64284], [64434, 64466], [64830, 64847], [64912, 64913], [64968, 65007],
+         [65020, 65135], [65277, 65295], [65306, 65312], [65339, 65344], [65371, 65381],
+         [65471, 65473], [65480, 65481], [65488, 65489], [65496, 65497]];
+    for (i = 0; i < ranges.length; i++) {
+        start = ranges[i][0];
+        end = ranges[i][1];
+        for (j = start; j <= end; j++) {
+            result[j] = true;
+        }
+    }
+    return result;
+})();
+
+function splitQuery(query) {
+    var result = [];
+    var start = -1;
+    for (var i = 0; i < query.length; i++) {
+        if (splitChars[query.charCodeAt(i)]) {
+            if (start !== -1) {
+                result.push(query.slice(start, i));
+                start = -1;
+            }
+        } else if (start === -1) {
+            start = i;
+        }
+    }
+    if (start !== -1) {
+        result.push(query.slice(start));
+    }
+    return result;
+}
+
+
+
+
+/**
+ * Search Module
+ */
+var Search = {
+
+  _index : null,
+  _queued_query : null,
+  _pulse_status : -1,
+
+  init : function() {
+      var params = $.getQueryParameters();
+      if (params.q) {
+          var query = params.q[0];
+          $('input[name="q"]')[0].value = query;
+          this.performSearch(query);
+      }
+  },
+
+  loadIndex : function(url) {
+    $.ajax({type: "GET", url: url, data: null,
+            dataType: "script", cache: true,
+            complete: function(jqxhr, textstatus) {
+              if (textstatus != "success") {
+                document.getElementById("searchindexloader").src = url;
+              }
+            }});
+  },
+
+  setIndex : function(index) {
+    var q;
+    this._index = index;
+    if ((q = this._queued_query) !== null) {
+      this._queued_query = null;
+      Search.query(q);
+    }
+  },
+
+  hasIndex : function() {
+      return this._index !== null;
+  },
+
+  deferQuery : function(query) {
+      this._queued_query = query;
+  },
+
+  stopPulse : function() {
+      this._pulse_status = 0;
+  },
+
+  startPulse : function() {
+    if (this._pulse_status >= 0)
+        return;
+    function pulse() {
+      var i;
+      Search._pulse_status = (Search._pulse_status + 1) % 4;
+      var dotString = '';
+      for (i = 0; i < Search._pulse_status; i++)
+        dotString += '.';
+      Search.dots.text(dotString);
+      if (Search._pulse_status > -1)
+        window.setTimeout(pulse, 500);
+    }
+    pulse();
+  },
+
+  /**
+   * perform a search for something (or wait until index is loaded)
+   */
+  performSearch : function(query) {
+    // create the required interface elements
+    this.out = $('#search-results');
+    this.title = $('<h2>' + _('Searching') + '</h2>').appendTo(this.out);
+    this.dots = $('<span></span>').appendTo(this.title);
+    this.status = $('<p style="display: none"></p>').appendTo(this.out);
+    this.output = $('<ul class="search"/>').appendTo(this.out);
+
+    $('#search-progress').text(_('Preparing search...'));
+    this.startPulse();
+
+    // index already loaded, the browser was quick!
+    if (this.hasIndex())
+      this.query(query);
+    else
+      this.deferQuery(query);
+  },
+
+  /**
+   * execute search (requires search index to be loaded)
+   */
+  query : function(query) {
+    var i;
+    var stopwords = ["a","and","are","as","at","be","but","by","for","if","in","into","is","it","near","no","not","of","on","or","such","that","the","their","then","there","these","they","this","to","was","will","with"];
+
+    // stem the searchterms and add them to the correct list
+    var stemmer = new Stemmer();
+    var searchterms = [];
+    var excluded = [];
+    var hlterms = [];
+    var tmp = splitQuery(query);
+    var objectterms = [];
+    for (i = 0; i < tmp.length; i++) {
+      if (tmp[i] !== "") {
+          objectterms.push(tmp[i].toLowerCase());
+      }
+
+      if ($u.indexOf(stopwords, tmp[i].toLowerCase()) != -1 || tmp[i].match(/^\d+$/) ||
+          tmp[i] === "") {
+        // skip this "word"
+        continue;
+      }
+      // stem the word
+      var word = stemmer.stemWord(tmp[i].toLowerCase());
+      // prevent stemmer from cutting word smaller than two chars
+      if(word.length < 3 && tmp[i].length >= 3) {
+        word = tmp[i];
+      }
+      var toAppend;
+      // select the correct list
+      if (word[0] == '-') {
+        toAppend = excluded;
+        word = word.substr(1);
+      }
+      else {
+        toAppend = searchterms;
+        hlterms.push(tmp[i].toLowerCase());
+      }
+      // only add if not already in the list
+      if (!$u.contains(toAppend, word))
+        toAppend.push(word);
+    }
+    var highlightstring = '?highlight=' + $.urlencode(hlterms.join(" "));
+
+    // console.debug('SEARCH: searching for:');
+    // console.info('required: ', searchterms);
+    // console.info('excluded: ', excluded);
+
+    // prepare search
+    var terms = this._index.terms;
+    var titleterms = this._index.titleterms;
+
+    // array of [filename, title, anchor, descr, score]
+    var results = [];
+    $('#search-progress').empty();
+
+    // lookup as object
+    for (i = 0; i < objectterms.length; i++) {
+      var others = [].concat(objectterms.slice(0, i),
+                             objectterms.slice(i+1, objectterms.length));
+      results = results.concat(this.performObjectSearch(objectterms[i], others));
+    }
+
+    // lookup as search terms in fulltext
+    results = results.concat(this.performTermsSearch(searchterms, excluded, terms, titleterms));
+
+    // let the scorer override scores with a custom scoring function
+    if (Scorer.score) {
+      for (i = 0; i < results.length; i++)
+        results[i][4] = Scorer.score(results[i]);
+    }
+
+    // now sort the results by score (in opposite order of appearance, since the
+    // display function below uses pop() to retrieve items) and then
+    // alphabetically
+    results.sort(function(a, b) {
+      var left = a[4];
+      var right = b[4];
+      if (left > right) {
+        return 1;
+      } else if (left < right) {
+        return -1;
+      } else {
+        // same score: sort alphabetically
+        left = a[1].toLowerCase();
+        right = b[1].toLowerCase();
+        return (left > right) ? -1 : ((left < right) ? 1 : 0);
+      }
+    });
+
+    // for debugging
+    //Search.lastresults = results.slice();  // a copy
+    //console.info('search results:', Search.lastresults);
+
+    // print the results
+    var resultCount = results.length;
+    function displayNextItem() {
+      // results left, load the summary and display it
+      if (results.length) {
+        var item = results.pop();
+        var listItem = $('<li style="display:none"></li>');
+        if (DOCUMENTATION_OPTIONS.FILE_SUFFIX === '') {
+          // dirhtml builder
+          var dirname = item[0] + '/';
+          if (dirname.match(/\/index\/$/)) {
+            dirname = dirname.substring(0, dirname.length-6);
+          } else if (dirname == 'index/') {
+            dirname = '';
+          }
+          listItem.append($('<a/>').attr('href',
+            DOCUMENTATION_OPTIONS.URL_ROOT + dirname +
+            highlightstring + item[2]).html(item[1]));
+        } else {
+          // normal html builders
+          listItem.append($('<a/>').attr('href',
+            item[0] + DOCUMENTATION_OPTIONS.FILE_SUFFIX +
+            highlightstring + item[2]).html(item[1]));
+        }
+        if (item[3]) {
+          listItem.append($('<span> (' + item[3] + ')</span>'));
+          Search.output.append(listItem);
+          listItem.slideDown(5, function() {
+            displayNextItem();
+          });
+        } else if (DOCUMENTATION_OPTIONS.HAS_SOURCE) {
+          var suffix = DOCUMENTATION_OPTIONS.SOURCELINK_SUFFIX;
+          if (suffix === undefined) {
+            suffix = '.txt';
+          }
+          $.ajax({url: DOCUMENTATION_OPTIONS.URL_ROOT + '_sources/' + item[5] + (item[5].slice(-suffix.length) === suffix ? '' : suffix),
+                  dataType: "text",
+                  complete: function(jqxhr, textstatus) {
+                    var data = jqxhr.responseText;
+                    if (data !== '' && data !== undefined) {
+                      listItem.append(Search.makeSearchSummary(data, searchterms, hlterms));
+                    }
+                    Search.output.append(listItem);
+                    listItem.slideDown(5, function() {
+                      displayNextItem();
+                    });
+                  }});
+        } else {
+          // no source available, just display title
+          Search.output.append(listItem);
+          listItem.slideDown(5, function() {
+            displayNextItem();
+          });
+        }
+      }
+      // search finished, update title and status message
+      else {
+        Search.stopPulse();
+        Search.title.text(_('Search Results'));
+        if (!resultCount)
+          Search.status.text(_('Your search did not match any documents. Please make sure that all words are spelled correctly and that you\'ve selected enough categories.'));
+        else
+            Search.status.text(_('Search finished, found %s page(s) matching the search query.').replace('%s', resultCount));
+        Search.status.fadeIn(500);
+      }
+    }
+    displayNextItem();
+  },
+
+  /**
+   * search for object names
+   */
+  performObjectSearch : function(object, otherterms) {
+    var filenames = this._index.filenames;
+    var docnames = this._index.docnames;
+    var objects = this._index.objects;
+    var objnames = this._index.objnames;
+    var titles = this._index.titles;
+
+    var i;
+    var results = [];
+
+    for (var prefix in objects) {
+      for (var name in objects[prefix]) {
+        var fullname = (prefix ? prefix + '.' : '') + name;
+        if (fullname.toLowerCase().indexOf(object) > -1) {
+          var score = 0;
+          var parts = fullname.split('.');
+          // check for different match types: exact matches of full name or
+          // "last name" (i.e. last dotted part)
+          if (fullname == object || parts[parts.length - 1] == object) {
+            score += Scorer.objNameMatch;
+          // matches in last name
+          } else if (parts[parts.length - 1].indexOf(object) > -1) {
+            score += Scorer.objPartialMatch;
+          }
+          var match = objects[prefix][name];
+          var objname = objnames[match[1]][2];
+          var title = titles[match[0]];
+          // If more than one term searched for, we require other words to be
+          // found in the name/title/description
+          if (otherterms.length > 0) {
+            var haystack = (prefix + ' ' + name + ' ' +
+                            objname + ' ' + title).toLowerCase();
+            var allfound = true;
+            for (i = 0; i < otherterms.length; i++) {
+              if (haystack.indexOf(otherterms[i]) == -1) {
+                allfound = false;
+                break;
+              }
+            }
+            if (!allfound) {
+              continue;
+            }
+          }
+          var descr = objname + _(', in ') + title;
+
+          var anchor = match[3];
+          if (anchor === '')
+            anchor = fullname;
+          else if (anchor == '-')
+            anchor = objnames[match[1]][1] + '-' + fullname;
+          // add custom score for some objects according to scorer
+          if (Scorer.objPrio.hasOwnProperty(match[2])) {
+            score += Scorer.objPrio[match[2]];
+          } else {
+            score += Scorer.objPrioDefault;
+          }
+          results.push([docnames[match[0]], fullname, '#'+anchor, descr, score, filenames[match[0]]]);
+        }
+      }
+    }
+
+    return results;
+  },
+
+  /**
+   * search for full-text terms in the index
+   */
+  performTermsSearch : function(searchterms, excluded, terms, titleterms) {
+    var docnames = this._index.docnames;
+    var filenames = this._index.filenames;
+    var titles = this._index.titles;
+
+    var i, j, file;
+    var fileMap = {};
+    var scoreMap = {};
+    var results = [];
+
+    // perform the search on the required terms
+    for (i = 0; i < searchterms.length; i++) {
+      var word = searchterms[i];
+      var files = [];
+      var _o = [
+        {files: terms[word], score: Scorer.term},
+        {files: titleterms[word], score: Scorer.title}
+      ];
+
+      // no match but word was a required one
+      if ($u.every(_o, function(o){return o.files === undefined;})) {
+        break;
+      }
+      // found search word in contents
+      $u.each(_o, function(o) {
+        var _files = o.files;
+        if (_files === undefined)
+          return
+
+        if (_files.length === undefined)
+          _files = [_files];
+        files = files.concat(_files);
+
+        // set score for the word in each file to Scorer.term
+        for (j = 0; j < _files.length; j++) {
+          file = _files[j];
+          if (!(file in scoreMap))
+            scoreMap[file] = {}
+          scoreMap[file][word] = o.score;
+        }
+      });
+
+      // create the mapping
+      for (j = 0; j < files.length; j++) {
+        file = files[j];
+        if (file in fileMap)
+          fileMap[file].push(word);
+        else
+          fileMap[file] = [word];
+      }
+    }
+
+    // now check if the files don't contain excluded terms
+    for (file in fileMap) {
+      var valid = true;
+
+      // check if all requirements are matched
+      if (fileMap[file].length != searchterms.length)
+          continue;
+
+      // ensure that none of the excluded terms is in the search result
+      for (i = 0; i < excluded.length; i++) {
+        if (terms[excluded[i]] == file ||
+            titleterms[excluded[i]] == file ||
+            $u.contains(terms[excluded[i]] || [], file) ||
+            $u.contains(titleterms[excluded[i]] || [], file)) {
+          valid = false;
+          break;
+        }
+      }
+
+      // if we have still a valid result we can add it to the result list
+      if (valid) {
+        // select one (max) score for the file.
+        // for better ranking, we should calculate ranking by using words statistics like basic tf-idf...
+        var score = $u.max($u.map(fileMap[file], function(w){return scoreMap[file][w]}));
+        results.push([docnames[file], titles[file], '', null, score, filenames[file]]);
+      }
+    }
+    return results;
+  },
+
+  /**
+   * helper function to return a node containing the
+   * search summary for a given text. keywords is a list
+   * of stemmed words, hlwords is the list of normal, unstemmed
+   * words. the first one is used to find the occurrence, the
+   * latter for highlighting it.
+   */
+  makeSearchSummary : function(text, keywords, hlwords) {
+    var textLower = text.toLowerCase();
+    var start = 0;
+    $.each(keywords, function() {
+      var i = textLower.indexOf(this.toLowerCase());
+      if (i > -1)
+        start = i;
+    });
+    start = Math.max(start - 120, 0);
+    var excerpt = ((start > 0) ? '...' : '') +
+      $.trim(text.substr(start, 240)) +
+      ((start + 240 - text.length) ? '...' : '');
+    var rv = $('<div class="context"></div>').text(excerpt);
+    $.each(hlwords, function() {
+      rv = rv.highlightText(this, 'highlighted');
+    });
+    return rv;
+  }
+};
+
+$(document).ready(function() {
+  Search.init();
+});

docs/_build/html/_static/underscore-1.3.1.js

@@ -0,0 +1,999 @@
+//     Underscore.js 1.3.1
+//     (c) 2009-2012 Jeremy Ashkenas, DocumentCloud Inc.
+//     Underscore is freely distributable under the MIT license.
+//     Portions of Underscore are inspired or borrowed from Prototype,
+//     Oliver Steele's Functional, and John Resig's Micro-Templating.
+//     For all details and documentation:
+//     http://documentcloud.github.com/underscore
+
+(function() {
+
+  // Baseline setup
+  // --------------
+
+  // Establish the root object, `window` in the browser, or `global` on the server.
+  var root = this;
+
+  // Save the previous value of the `_` variable.
+  var previousUnderscore = root._;
+
+  // Establish the object that gets returned to break out of a loop iteration.
+  var breaker = {};
+
+  // Save bytes in the minified (but not gzipped) version:
+  var ArrayProto = Array.prototype, ObjProto = Object.prototype, FuncProto = Function.prototype;
+
+  // Create quick reference variables for speed access to core prototypes.
+  var slice            = ArrayProto.slice,
+      unshift          = ArrayProto.unshift,
+      toString         = ObjProto.toString,
+      hasOwnProperty   = ObjProto.hasOwnProperty;
+
+  // All **ECMAScript 5** native function implementations that we hope to use
+  // are declared here.
+  var
+    nativeForEach      = ArrayProto.forEach,
+    nativeMap          = ArrayProto.map,
+    nativeReduce       = ArrayProto.reduce,
+    nativeReduceRight  = ArrayProto.reduceRight,
+    nativeFilter       = ArrayProto.filter,
+    nativeEvery        = ArrayProto.every,
+    nativeSome         = ArrayProto.some,
+    nativeIndexOf      = ArrayProto.indexOf,
+    nativeLastIndexOf  = ArrayProto.lastIndexOf,
+    nativeIsArray      = Array.isArray,
+    nativeKeys         = Object.keys,
+    nativeBind         = FuncProto.bind;
+
+  // Create a safe reference to the Underscore object for use below.
+  var _ = function(obj) { return new wrapper(obj); };
+
+  // Export the Underscore object for **Node.js**, with
+  // backwards-compatibility for the old `require()` API. If we're in
+  // the browser, add `_` as a global object via a string identifier,
+  // for Closure Compiler "advanced" mode.
+  if (typeof exports !== 'undefined') {
+    if (typeof module !== 'undefined' && module.exports) {
+      exports = module.exports = _;
+    }
+    exports._ = _;
+  } else {
+    root['_'] = _;
+  }
+
+  // Current version.
+  _.VERSION = '1.3.1';
+
+  // Collection Functions
+  // --------------------
+
+  // The cornerstone, an `each` implementation, aka `forEach`.
+  // Handles objects with the built-in `forEach`, arrays, and raw objects.
+  // Delegates to **ECMAScript 5**'s native `forEach` if available.
+  var each = _.each = _.forEach = function(obj, iterator, context) {
+    if (obj == null) return;
+    if (nativeForEach && obj.forEach === nativeForEach) {
+      obj.forEach(iterator, context);
+    } else if (obj.length === +obj.length) {
+      for (var i = 0, l = obj.length; i < l; i++) {
+        if (i in obj && iterator.call(context, obj[i], i, obj) === breaker) return;
+      }
+    } else {
+      for (var key in obj) {
+        if (_.has(obj, key)) {
+          if (iterator.call(context, obj[key], key, obj) === breaker) return;
+        }
+      }
+    }
+  };
+
+  // Return the results of applying the iterator to each element.
+  // Delegates to **ECMAScript 5**'s native `map` if available.
+  _.map = _.collect = function(obj, iterator, context) {
+    var results = [];
+    if (obj == null) return results;
+    if (nativeMap && obj.map === nativeMap) return obj.map(iterator, context);
+    each(obj, function(value, index, list) {
+      results[results.length] = iterator.call(context, value, index, list);
+    });
+    if (obj.length === +obj.length) results.length = obj.length;
+    return results;
+  };
+
+  // **Reduce** builds up a single result from a list of values, aka `inject`,
+  // or `foldl`. Delegates to **ECMAScript 5**'s native `reduce` if available.
+  _.reduce = _.foldl = _.inject = function(obj, iterator, memo, context) {
+    var initial = arguments.length > 2;
+    if (obj == null) obj = [];
+    if (nativeReduce && obj.reduce === nativeReduce) {
+      if (context) iterator = _.bind(iterator, context);
+      return initial ? obj.reduce(iterator, memo) : obj.reduce(iterator);
+    }
+    each(obj, function(value, index, list) {
+      if (!initial) {
+        memo = value;
+        initial = true;
+      } else {
+        memo = iterator.call(context, memo, value, index, list);
+      }
+    });
+    if (!initial) throw new TypeError('Reduce of empty array with no initial value');
+    return memo;
+  };
+
+  // The right-associative version of reduce, also known as `foldr`.
+  // Delegates to **ECMAScript 5**'s native `reduceRight` if available.
+  _.reduceRight = _.foldr = function(obj, iterator, memo, context) {
+    var initial = arguments.length > 2;
+    if (obj == null) obj = [];
+    if (nativeReduceRight && obj.reduceRight === nativeReduceRight) {
+      if (context) iterator = _.bind(iterator, context);
+      return initial ? obj.reduceRight(iterator, memo) : obj.reduceRight(iterator);
+    }
+    var reversed = _.toArray(obj).reverse();
+    if (context && !initial) iterator = _.bind(iterator, context);
+    return initial ? _.reduce(reversed, iterator, memo, context) : _.reduce(reversed, iterator);
+  };
+
+  // Return the first value which passes a truth test. Aliased as `detect`.
+  _.find = _.detect = function(obj, iterator, context) {
+    var result;
+    any(obj, function(value, index, list) {
+      if (iterator.call(context, value, index, list)) {
+        result = value;
+        return true;
+      }
+    });
+    return result;
+  };
+
+  // Return all the elements that pass a truth test.
+  // Delegates to **ECMAScript 5**'s native `filter` if available.
+  // Aliased as `select`.
+  _.filter = _.select = function(obj, iterator, context) {
+    var results = [];
+    if (obj == null) return results;
+    if (nativeFilter && obj.filter === nativeFilter) return obj.filter(iterator, context);
+    each(obj, function(value, index, list) {
+      if (iterator.call(context, value, index, list)) results[results.length] = value;
+    });
+    return results;
+  };
+
+  // Return all the elements for which a truth test fails.
+  _.reject = function(obj, iterator, context) {
+    var results = [];
+    if (obj == null) return results;
+    each(obj, function(value, index, list) {
+      if (!iterator.call(context, value, index, list)) results[results.length] = value;
+    });
+    return results;
+  };
+
+  // Determine whether all of the elements match a truth test.
+  // Delegates to **ECMAScript 5**'s native `every` if available.
+  // Aliased as `all`.
+  _.every = _.all = function(obj, iterator, context) {
+    var result = true;
+    if (obj == null) return result;
+    if (nativeEvery && obj.every === nativeEvery) return obj.every(iterator, context);
+    each(obj, function(value, index, list) {
+      if (!(result = result && iterator.call(context, value, index, list))) return breaker;
+    });
+    return result;
+  };
+
+  // Determine if at least one element in the object matches a truth test.
+  // Delegates to **ECMAScript 5**'s native `some` if available.
+  // Aliased as `any`.
+  var any = _.some = _.any = function(obj, iterator, context) {
+    iterator || (iterator = _.identity);
+    var result = false;
+    if (obj == null) return result;
+    if (nativeSome && obj.some === nativeSome) return obj.some(iterator, context);
+    each(obj, function(value, index, list) {
+      if (result || (result = iterator.call(context, value, index, list))) return breaker;
+    });
+    return !!result;
+  };
+
+  // Determine if a given value is included in the array or object using `===`.
+  // Aliased as `contains`.
+  _.include = _.contains = function(obj, target) {
+    var found = false;
+    if (obj == null) return found;
+    if (nativeIndexOf && obj.indexOf === nativeIndexOf) return obj.indexOf(target) != -1;
+    found = any(obj, function(value) {
+      return value === target;
+    });
+    return found;
+  };
+
+  // Invoke a method (with arguments) on every item in a collection.
+  _.invoke = function(obj, method) {
+    var args = slice.call(arguments, 2);
+    return _.map(obj, function(value) {
+      return (_.isFunction(method) ? method || value : value[method]).apply(value, args);
+    });
+  };
+
+  // Convenience version of a common use case of `map`: fetching a property.
+  _.pluck = function(obj, key) {
+    return _.map(obj, function(value){ return value[key]; });
+  };
+
+  // Return the maximum element or (element-based computation).
+  _.max = function(obj, iterator, context) {
+    if (!iterator && _.isArray(obj)) return Math.max.apply(Math, obj);
+    if (!iterator && _.isEmpty(obj)) return -Infinity;
+    var result = {computed : -Infinity};
+    each(obj, function(value, index, list) {
+      var computed = iterator ? iterator.call(context, value, index, list) : value;
+      computed >= result.computed && (result = {value : value, computed : computed});
+    });
+    return result.value;
+  };
+
+  // Return the minimum element (or element-based computation).
+  _.min = function(obj, iterator, context) {
+    if (!iterator && _.isArray(obj)) return Math.min.apply(Math, obj);
+    if (!iterator && _.isEmpty(obj)) return Infinity;
+    var result = {computed : Infinity};
+    each(obj, function(value, index, list) {
+      var computed = iterator ? iterator.call(context, value, index, list) : value;
+      computed < result.computed && (result = {value : value, computed : computed});
+    });
+    return result.value;
+  };
+
+  // Shuffle an array.
+  _.shuffle = function(obj) {
+    var shuffled = [], rand;
+    each(obj, function(value, index, list) {
+      if (index == 0) {
+        shuffled[0] = value;
+      } else {
+        rand = Math.floor(Math.random() * (index + 1));
+        shuffled[index] = shuffled[rand];
+        shuffled[rand] = value;
+      }
+    });
+    return shuffled;
+  };
+
+  // Sort the object's values by a criterion produced by an iterator.
+  _.sortBy = function(obj, iterator, context) {
+    return _.pluck(_.map(obj, function(value, index, list) {
+      return {
+        value : value,
+        criteria : iterator.call(context, value, index, list)
+      };
+    }).sort(function(left, right) {
+      var a = left.criteria, b = right.criteria;
+      return a < b ? -1 : a > b ? 1 : 0;
+    }), 'value');
+  };
+
+  // Groups the object's values by a criterion. Pass either a string attribute
+  // to group by, or a function that returns the criterion.
+  _.groupBy = function(obj, val) {
+    var result = {};
+    var iterator = _.isFunction(val) ? val : function(obj) { return obj[val]; };
+    each(obj, function(value, index) {
+      var key = iterator(value, index);
+      (result[key] || (result[key] = [])).push(value);
+    });
+    return result;
+  };
+
+  // Use a comparator function to figure out at what index an object should
+  // be inserted so as to maintain order. Uses binary search.
+  _.sortedIndex = function(array, obj, iterator) {
+    iterator || (iterator = _.identity);
+    var low = 0, high = array.length;
+    while (low < high) {
+      var mid = (low + high) >> 1;
+      iterator(array[mid]) < iterator(obj) ? low = mid + 1 : high = mid;
+    }
+    return low;
+  };
+
+  // Safely convert anything iterable into a real, live array.
+  _.toArray = function(iterable) {
+    if (!iterable)                return [];
+    if (iterable.toArray)         return iterable.toArray();
+    if (_.isArray(iterable))      return slice.call(iterable);
+    if (_.isArguments(iterable))  return slice.call(iterable);
+    return _.values(iterable);
+  };
+
+  // Return the number of elements in an object.
+  _.size = function(obj) {
+    return _.toArray(obj).length;
+  };
+
+  // Array Functions
+  // ---------------
+
+  // Get the first element of an array. Passing **n** will return the first N
+  // values in the array. Aliased as `head`. The **guard** check allows it to work
+  // with `_.map`.
+  _.first = _.head = function(array, n, guard) {
+    return (n != null) && !guard ? slice.call(array, 0, n) : array[0];
+  };
+
+  // Returns everything but the last entry of the array. Especcialy useful on
+  // the arguments object. Passing **n** will return all the values in
+  // the array, excluding the last N. The **guard** check allows it to work with
+  // `_.map`.
+  _.initial = function(array, n, guard) {
+    return slice.call(array, 0, array.length - ((n == null) || guard ? 1 : n));
+  };
+
+  // Get the last element of an array. Passing **n** will return the last N
+  // values in the array. The **guard** check allows it to work with `_.map`.
+  _.last = function(array, n, guard) {
+    if ((n != null) && !guard) {
+      return slice.call(array, Math.max(array.length - n, 0));
+    } else {
+      return array[array.length - 1];
+    }
+  };
+
+  // Returns everything but the first entry of the array. Aliased as `tail`.
+  // Especially useful on the arguments object. Passing an **index** will return
+  // the rest of the values in the array from that index onward. The **guard**
+  // check allows it to work with `_.map`.
+  _.rest = _.tail = function(array, index, guard) {
+    return slice.call(array, (index == null) || guard ? 1 : index);
+  };
+
+  // Trim out all falsy values from an array.
+  _.compact = function(array) {
+    return _.filter(array, function(value){ return !!value; });
+  };
+
+  // Return a completely flattened version of an array.
+  _.flatten = function(array, shallow) {
+    return _.reduce(array, function(memo, value) {
+      if (_.isArray(value)) return memo.concat(shallow ? value : _.flatten(value));
+      memo[memo.length] = value;
+      return memo;
+    }, []);
+  };
+
+  // Return a version of the array that does not contain the specified value(s).
+  _.without = function(array) {
+    return _.difference(array, slice.call(arguments, 1));
+  };
+
+  // Produce a duplicate-free version of the array. If the array has already
+  // been sorted, you have the option of using a faster algorithm.
+  // Aliased as `unique`.
+  _.uniq = _.unique = function(array, isSorted, iterator) {
+    var initial = iterator ? _.map(array, iterator) : array;
+    var result = [];
+    _.reduce(initial, function(memo, el, i) {
+      if (0 == i || (isSorted === true ? _.last(memo) != el : !_.include(memo, el))) {
+        memo[memo.length] = el;
+        result[result.length] = array[i];
+      }
+      return memo;
+    }, []);
+    return result;
+  };
+
+  // Produce an array that contains the union: each distinct element from all of
+  // the passed-in arrays.
+  _.union = function() {
+    return _.uniq(_.flatten(arguments, true));
+  };
+
+  // Produce an array that contains every item shared between all the
+  // passed-in arrays. (Aliased as "intersect" for back-compat.)
+  _.intersection = _.intersect = function(array) {
+    var rest = slice.call(arguments, 1);
+    return _.filter(_.uniq(array), function(item) {
+      return _.every(rest, function(other) {
+        return _.indexOf(other, item) >= 0;
+      });
+    });
+  };
+
+  // Take the difference between one array and a number of other arrays.
+  // Only the elements present in just the first array will remain.
+  _.difference = function(array) {
+    var rest = _.flatten(slice.call(arguments, 1));
+    return _.filter(array, function(value){ return !_.include(rest, value); });
+  };
+
+  // Zip together multiple lists into a single array -- elements that share
+  // an index go together.
+  _.zip = function() {
+    var args = slice.call(arguments);
+    var length = _.max(_.pluck(args, 'length'));
+    var results = new Array(length);
+    for (var i = 0; i < length; i++) results[i] = _.pluck(args, "" + i);
+    return results;
+  };
+
+  // If the browser doesn't supply us with indexOf (I'm looking at you, **MSIE**),
+  // we need this function. Return the position of the first occurrence of an
+  // item in an array, or -1 if the item is not included in the array.
+  // Delegates to **ECMAScript 5**'s native `indexOf` if available.
+  // If the array is large and already in sort order, pass `true`
+  // for **isSorted** to use binary search.
+  _.indexOf = function(array, item, isSorted) {
+    if (array == null) return -1;
+    var i, l;
+    if (isSorted) {
+      i = _.sortedIndex(array, item);
+      return array[i] === item ? i : -1;
+    }
+    if (nativeIndexOf && array.indexOf === nativeIndexOf) return array.indexOf(item);
+    for (i = 0, l = array.length; i < l; i++) if (i in array && array[i] === item) return i;
+    return -1;
+  };
+
+  // Delegates to **ECMAScript 5**'s native `lastIndexOf` if available.
+  _.lastIndexOf = function(array, item) {
+    if (array == null) return -1;
+    if (nativeLastIndexOf && array.lastIndexOf === nativeLastIndexOf) return array.lastIndexOf(item);
+    var i = array.length;
+    while (i--) if (i in array && array[i] === item) return i;
+    return -1;
+  };
+
+  // Generate an integer Array containing an arithmetic progression. A port of
+  // the native Python `range()` function. See
+  // [the Python documentation](http://docs.python.org/library/functions.html#range).
+  _.range = function(start, stop, step) {
+    if (arguments.length <= 1) {
+      stop = start || 0;
+      start = 0;
+    }
+    step = arguments[2] || 1;
+
+    var len = Math.max(Math.ceil((stop - start) / step), 0);
+    var idx = 0;
+    var range = new Array(len);
+
+    while(idx < len) {
+      range[idx++] = start;
+      start += step;
+    }
+
+    return range;
+  };
+
+  // Function (ahem) Functions
+  // ------------------
+
+  // Reusable constructor function for prototype setting.
+  var ctor = function(){};
+
+  // Create a function bound to a given object (assigning `this`, and arguments,
+  // optionally). Binding with arguments is also known as `curry`.
+  // Delegates to **ECMAScript 5**'s native `Function.bind` if available.
+  // We check for `func.bind` first, to fail fast when `func` is undefined.
+  _.bind = function bind(func, context) {
+    var bound, args;
+    if (func.bind === nativeBind && nativeBind) return nativeBind.apply(func, slice.call(arguments, 1));
+    if (!_.isFunction(func)) throw new TypeError;
+    args = slice.call(arguments, 2);
+    return bound = function() {
+      if (!(this instanceof bound)) return func.apply(context, args.concat(slice.call(arguments)));
+      ctor.prototype = func.prototype;
+      var self = new ctor;
+      var result = func.apply(self, args.concat(slice.call(arguments)));
+      if (Object(result) === result) return result;
+      return self;
+    };
+  };
+
+  // Bind all of an object's methods to that object. Useful for ensuring that
+  // all callbacks defined on an object belong to it.
+  _.bindAll = function(obj) {
+    var funcs = slice.call(arguments, 1);
+    if (funcs.length == 0) funcs = _.functions(obj);
+    each(funcs, function(f) { obj[f] = _.bind(obj[f], obj); });
+    return obj;
+  };
+
+  // Memoize an expensive function by storing its results.
+  _.memoize = function(func, hasher) {
+    var memo = {};
+    hasher || (hasher = _.identity);
+    return function() {
+      var key = hasher.apply(this, arguments);
+      return _.has(memo, key) ? memo[key] : (memo[key] = func.apply(this, arguments));
+    };
+  };
+
+  // Delays a function for the given number of milliseconds, and then calls
+  // it with the arguments supplied.
+  _.delay = function(func, wait) {
+    var args = slice.call(arguments, 2);
+    return setTimeout(function(){ return func.apply(func, args); }, wait);
+  };
+
+  // Defers a function, scheduling it to run after the current call stack has
+  // cleared.
+  _.defer = function(func) {
+    return _.delay.apply(_, [func, 1].concat(slice.call(arguments, 1)));
+  };
+
+  // Returns a function, that, when invoked, will only be triggered at most once
+  // during a given window of time.
+  _.throttle = function(func, wait) {
+    var context, args, timeout, throttling, more;
+    var whenDone = _.debounce(function(){ more = throttling = false; }, wait);
+    return function() {
+      context = this; args = arguments;
+      var later = function() {
+        timeout = null;
+        if (more) func.apply(context, args);
+        whenDone();
+      };
+      if (!timeout) timeout = setTimeout(later, wait);
+      if (throttling) {
+        more = true;
+      } else {
+        func.apply(context, args);
+      }
+      whenDone();
+      throttling = true;
+    };
+  };
+
+  // Returns a function, that, as long as it continues to be invoked, will not
+  // be triggered. The function will be called after it stops being called for
+  // N milliseconds.
+  _.debounce = function(func, wait) {
+    var timeout;
+    return function() {
+      var context = this, args = arguments;
+      var later = function() {
+        timeout = null;
+        func.apply(context, args);
+      };
+      clearTimeout(timeout);
+      timeout = setTimeout(later, wait);
+    };
+  };
+
+  // Returns a function that will be executed at most one time, no matter how
+  // often you call it. Useful for lazy initialization.
+  _.once = function(func) {
+    var ran = false, memo;
+    return function() {
+      if (ran) return memo;
+      ran = true;
+      return memo = func.apply(this, arguments);
+    };
+  };
+
+  // Returns the first function passed as an argument to the second,
+  // allowing you to adjust arguments, run code before and after, and
+  // conditionally execute the original function.
+  _.wrap = function(func, wrapper) {
+    return function() {
+      var args = [func].concat(slice.call(arguments, 0));
+      return wrapper.apply(this, args);
+    };
+  };
+
+  // Returns a function that is the composition of a list of functions, each
+  // consuming the return value of the function that follows.
+  _.compose = function() {
+    var funcs = arguments;
+    return function() {
+      var args = arguments;
+      for (var i = funcs.length - 1; i >= 0; i--) {
+        args = [funcs[i].apply(this, args)];
+      }
+      return args[0];
+    };
+  };
+
+  // Returns a function that will only be executed after being called N times.
+  _.after = function(times, func) {
+    if (times <= 0) return func();
+    return function() {
+      if (--times < 1) { return func.apply(this, arguments); }
+    };
+  };
+
+  // Object Functions
+  // ----------------
+
+  // Retrieve the names of an object's properties.
+  // Delegates to **ECMAScript 5**'s native `Object.keys`
+  _.keys = nativeKeys || function(obj) {
+    if (obj !== Object(obj)) throw new TypeError('Invalid object');
+    var keys = [];
+    for (var key in obj) if (_.has(obj, key)) keys[keys.length] = key;
+    return keys;
+  };
+
+  // Retrieve the values of an object's properties.
+  _.values = function(obj) {
+    return _.map(obj, _.identity);
+  };
+
+  // Return a sorted list of the function names available on the object.
+  // Aliased as `methods`
+  _.functions = _.methods = function(obj) {
+    var names = [];
+    for (var key in obj) {
+      if (_.isFunction(obj[key])) names.push(key);
+    }
+    return names.sort();
+  };
+
+  // Extend a given object with all the properties in passed-in object(s).
+  _.extend = function(obj) {
+    each(slice.call(arguments, 1), function(source) {
+      for (var prop in source) {
+        obj[prop] = source[prop];
+      }
+    });
+    return obj;
+  };
+
+  // Fill in a given object with default properties.
+  _.defaults = function(obj) {
+    each(slice.call(arguments, 1), function(source) {
+      for (var prop in source) {
+        if (obj[prop] == null) obj[prop] = source[prop];
+      }
+    });
+    return obj;
+  };
+
+  // Create a (shallow-cloned) duplicate of an object.
+  _.clone = function(obj) {
+    if (!_.isObject(obj)) return obj;
+    return _.isArray(obj) ? obj.slice() : _.extend({}, obj);
+  };
+
+  // Invokes interceptor with the obj, and then returns obj.
+  // The primary purpose of this method is to "tap into" a method chain, in
+  // order to perform operations on intermediate results within the chain.
+  _.tap = function(obj, interceptor) {
+    interceptor(obj);
+    return obj;
+  };
+
+  // Internal recursive comparison function.
+  function eq(a, b, stack) {
+    // Identical objects are equal. `0 === -0`, but they aren't identical.
+    // See the Harmony `egal` proposal: http://wiki.ecmascript.org/doku.php?id=harmony:egal.
+    if (a === b) return a !== 0 || 1 / a == 1 / b;
+    // A strict comparison is necessary because `null == undefined`.
+    if (a == null || b == null) return a === b;
+    // Unwrap any wrapped objects.
+    if (a._chain) a = a._wrapped;
+    if (b._chain) b = b._wrapped;
+    // Invoke a custom `isEqual` method if one is provided.
+    if (a.isEqual && _.isFunction(a.isEqual)) return a.isEqual(b);
+    if (b.isEqual && _.isFunction(b.isEqual)) return b.isEqual(a);
+    // Compare `[[Class]]` names.
+    var className = toString.call(a);
+    if (className != toString.call(b)) return false;
+    switch (className) {
+      // Strings, numbers, dates, and booleans are compared by value.
+      case '[object String]':
+        // Primitives and their corresponding object wrappers are equivalent; thus, `"5"` is
+        // equivalent to `new String("5")`.
+        return a == String(b);
+      case '[object Number]':
+        // `NaN`s are equivalent, but non-reflexive. An `egal` comparison is performed for
+        // other numeric values.
+        return a != +a ? b != +b : (a == 0 ? 1 / a == 1 / b : a == +b);
+      case '[object Date]':
+      case '[object Boolean]':
+        // Coerce dates and booleans to numeric primitive values. Dates are compared by their
+        // millisecond representations. Note that invalid dates with millisecond representations
+        // of `NaN` are not equivalent.
+        return +a == +b;
+      // RegExps are compared by their source patterns and flags.
+      case '[object RegExp]':
+        return a.source == b.source &&
+               a.global == b.global &&
+               a.multiline == b.multiline &&
+               a.ignoreCase == b.ignoreCase;
+    }
+    if (typeof a != 'object' || typeof b != 'object') return false;
+    // Assume equality for cyclic structures. The algorithm for detecting cyclic
+    // structures is adapted from ES 5.1 section 15.12.3, abstract operation `JO`.
+    var length = stack.length;
+    while (length--) {
+      // Linear search. Performance is inversely proportional to the number of
+      // unique nested structures.
+      if (stack[length] == a) return true;
+    }
+    // Add the first object to the stack of traversed objects.
+    stack.push(a);
+    var size = 0, result = true;
+    // Recursively compare objects and arrays.
+    if (className == '[object Array]') {
+      // Compare array lengths to determine if a deep comparison is necessary.
+      size = a.length;
+      result = size == b.length;
+      if (result) {
+        // Deep compare the contents, ignoring non-numeric properties.
+        while (size--) {
+          // Ensure commutative equality for sparse arrays.
+          if (!(result = size in a == size in b && eq(a[size], b[size], stack))) break;
+        }
+      }
+    } else {
+      // Objects with different constructors are not equivalent.
+      if ('constructor' in a != 'constructor' in b || a.constructor != b.constructor) return false;
+      // Deep compare objects.
+      for (var key in a) {
+        if (_.has(a, key)) {
+          // Count the expected number of properties.
+          size++;
+          // Deep compare each member.
+          if (!(result = _.has(b, key) && eq(a[key], b[key], stack))) break;
+        }
+      }
+      // Ensure that both objects contain the same number of properties.
+      if (result) {
+        for (key in b) {
+          if (_.has(b, key) && !(size--)) break;
+        }
+        result = !size;
+      }
+    }
+    // Remove the first object from the stack of traversed objects.
+    stack.pop();
+    return result;
+  }
+
+  // Perform a deep comparison to check if two objects are equal.
+  _.isEqual = function(a, b) {
+    return eq(a, b, []);
+  };
+
+  // Is a given array, string, or object empty?
+  // An "empty" object has no enumerable own-properties.
+  _.isEmpty = function(obj) {
+    if (_.isArray(obj) || _.isString(obj)) return obj.length === 0;
+    for (var key in obj) if (_.has(obj, key)) return false;
+    return true;
+  };
+
+  // Is a given value a DOM element?
+  _.isElement = function(obj) {
+    return !!(obj && obj.nodeType == 1);
+  };
+
+  // Is a given value an array?
+  // Delegates to ECMA5's native Array.isArray
+  _.isArray = nativeIsArray || function(obj) {
+    return toString.call(obj) == '[object Array]';
+  };
+
+  // Is a given variable an object?
+  _.isObject = function(obj) {
+    return obj === Object(obj);
+  };
+
+  // Is a given variable an arguments object?
+  _.isArguments = function(obj) {
+    return toString.call(obj) == '[object Arguments]';
+  };
+  if (!_.isArguments(arguments)) {
+    _.isArguments = function(obj) {
+      return !!(obj && _.has(obj, 'callee'));
+    };
+  }
+
+  // Is a given value a function?
+  _.isFunction = function(obj) {
+    return toString.call(obj) == '[object Function]';
+  };
+
+  // Is a given value a string?
+  _.isString = function(obj) {
+    return toString.call(obj) == '[object String]';
+  };
+
+  // Is a given value a number?
+  _.isNumber = function(obj) {
+    return toString.call(obj) == '[object Number]';
+  };
+
+  // Is the given value `NaN`?
+  _.isNaN = function(obj) {
+    // `NaN` is the only value for which `===` is not reflexive.
+    return obj !== obj;
+  };
+
+  // Is a given value a boolean?
+  _.isBoolean = function(obj) {
+    return obj === true || obj === false || toString.call(obj) == '[object Boolean]';
+  };
+
+  // Is a given value a date?
+  _.isDate = function(obj) {
+    return toString.call(obj) == '[object Date]';
+  };
+
+  // Is the given value a regular expression?
+  _.isRegExp = function(obj) {
+    return toString.call(obj) == '[object RegExp]';
+  };
+
+  // Is a given value equal to null?
+  _.isNull = function(obj) {
+    return obj === null;
+  };
+
+  // Is a given variable undefined?
+  _.isUndefined = function(obj) {
+    return obj === void 0;
+  };
+
+  // Has own property?
+  _.has = function(obj, key) {
+    return hasOwnProperty.call(obj, key);
+  };
+
+  // Utility Functions
+  // -----------------
+
+  // Run Underscore.js in *noConflict* mode, returning the `_` variable to its
+  // previous owner. Returns a reference to the Underscore object.
+  _.noConflict = function() {
+    root._ = previousUnderscore;
+    return this;
+  };
+
+  // Keep the identity function around for default iterators.
+  _.identity = function(value) {
+    return value;
+  };
+
+  // Run a function **n** times.
+  _.times = function (n, iterator, context) {
+    for (var i = 0; i < n; i++) iterator.call(context, i);
+  };
+
+  // Escape a string for HTML interpolation.
+  _.escape = function(string) {
+    return (''+string).replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;').replace(/"/g, '&quot;').replace(/'/g, '&#x27;').replace(/\//g,'&#x2F;');
+  };
+
+  // Add your own custom functions to the Underscore object, ensuring that
+  // they're correctly added to the OOP wrapper as well.
+  _.mixin = function(obj) {
+    each(_.functions(obj), function(name){
+      addToWrapper(name, _[name] = obj[name]);
+    });
+  };
+
+  // Generate a unique integer id (unique within the entire client session).
+  // Useful for temporary DOM ids.
+  var idCounter = 0;
+  _.uniqueId = function(prefix) {
+    var id = idCounter++;
+    return prefix ? prefix + id : id;
+  };
+
+  // By default, Underscore uses ERB-style template delimiters, change the
+  // following template settings to use alternative delimiters.
+  _.templateSettings = {
+    evaluate    : /<%([\s\S]+?)%>/g,
+    interpolate : /<%=([\s\S]+?)%>/g,
+    escape      : /<%-([\s\S]+?)%>/g
+  };
+
+  // When customizing `templateSettings`, if you don't want to define an
+  // interpolation, evaluation or escaping regex, we need one that is
+  // guaranteed not to match.
+  var noMatch = /.^/;
+
+  // Within an interpolation, evaluation, or escaping, remove HTML escaping
+  // that had been previously added.
+  var unescape = function(code) {
+    return code.replace(/\\\\/g, '\\').replace(/\\'/g, "'");
+  };
+
+  // JavaScript micro-templating, similar to John Resig's implementation.
+  // Underscore templating handles arbitrary delimiters, preserves whitespace,
+  // and correctly escapes quotes within interpolated code.
+  _.template = function(str, data) {
+    var c  = _.templateSettings;
+    var tmpl = 'var __p=[],print=function(){__p.push.apply(__p,arguments);};' +
+      'with(obj||{}){__p.push(\'' +
+      str.replace(/\\/g, '\\\\')
+         .replace(/'/g, "\\'")
+         .replace(c.escape || noMatch, function(match, code) {
+           return "',_.escape(" + unescape(code) + "),'";
+         })
+         .replace(c.interpolate || noMatch, function(match, code) {
+           return "'," + unescape(code) + ",'";
+         })
+         .replace(c.evaluate || noMatch, function(match, code) {
+           return "');" + unescape(code).replace(/[\r\n\t]/g, ' ') + ";__p.push('";
+         })
+         .replace(/\r/g, '\\r')
+         .replace(/\n/g, '\\n')
+         .replace(/\t/g, '\\t')
+         + "');}return __p.join('');";
+    var func = new Function('obj', '_', tmpl);
+    if (data) return func(data, _);
+    return function(data) {
+      return func.call(this, data, _);
+    };
+  };
+
+  // Add a "chain" function, which will delegate to the wrapper.
+  _.chain = function(obj) {
+    return _(obj).chain();
+  };
+
+  // The OOP Wrapper
+  // ---------------
+
+  // If Underscore is called as a function, it returns a wrapped object that
+  // can be used OO-style. This wrapper holds altered versions of all the
+  // underscore functions. Wrapped objects may be chained.
+  var wrapper = function(obj) { this._wrapped = obj; };
+
+  // Expose `wrapper.prototype` as `_.prototype`
+  _.prototype = wrapper.prototype;
+
+  // Helper function to continue chaining intermediate results.
+  var result = function(obj, chain) {
+    return chain ? _(obj).chain() : obj;
+  };
+
+  // A method to easily add functions to the OOP wrapper.
+  var addToWrapper = function(name, func) {
+    wrapper.prototype[name] = function() {
+      var args = slice.call(arguments);
+      unshift.call(args, this._wrapped);
+      return result(func.apply(_, args), this._chain);
+    };
+  };
+
+  // Add all of the Underscore functions to the wrapper object.
+  _.mixin(_);
+
+  // Add all mutator Array functions to the wrapper.
+  each(['pop', 'push', 'reverse', 'shift', 'sort', 'splice', 'unshift'], function(name) {
+    var method = ArrayProto[name];
+    wrapper.prototype[name] = function() {
+      var wrapped = this._wrapped;
+      method.apply(wrapped, arguments);
+      var length = wrapped.length;
+      if ((name == 'shift' || name == 'splice') && length === 0) delete wrapped[0];
+      return result(wrapped, this._chain);
+    };
+  });
+
+  // Add all accessor Array functions to the wrapper.
+  each(['concat', 'join', 'slice'], function(name) {
+    var method = ArrayProto[name];
+    wrapper.prototype[name] = function() {
+      return result(method.apply(this._wrapped, arguments), this._chain);
+    };
+  });
+
+  // Start chaining a wrapped Underscore object.
+  wrapper.prototype.chain = function() {
+    this._chain = true;
+    return this;
+  };
+
+  // Extracts the result from a wrapped and chained object.
+  wrapper.prototype.value = function() {
+    return this._wrapped;
+  };
+
+}).call(this);

docs/_build/html/_static/underscore.js

@@ -0,0 +1,31 @@
+// Underscore.js 1.3.1
+// (c) 2009-2012 Jeremy Ashkenas, DocumentCloud Inc.
+// Underscore is freely distributable under the MIT license.
+// Portions of Underscore are inspired or borrowed from Prototype,
+// Oliver Steele's Functional, and John Resig's Micro-Templating.
+// For all details and documentation:
+// http://documentcloud.github.com/underscore
+(function(){function q(a,c,d){if(a===c)return a!==0||1/a==1/c;if(a==null||c==null)return a===c;if(a._chain)a=a._wrapped;if(c._chain)c=c._wrapped;if(a.isEqual&&b.isFunction(a.isEqual))return a.isEqual(c);if(c.isEqual&&b.isFunction(c.isEqual))return c.isEqual(a);var e=l.call(a);if(e!=l.call(c))return false;switch(e){case "[object String]":return a==String(c);case "[object Number]":return a!=+a?c!=+c:a==0?1/a==1/c:a==+c;case "[object Date]":case "[object Boolean]":return+a==+c;case "[object RegExp]":return a.source==
+c.source&&a.global==c.global&&a.multiline==c.multiline&&a.ignoreCase==c.ignoreCase}if(typeof a!="object"||typeof c!="object")return false;for(var f=d.length;f--;)if(d[f]==a)return true;d.push(a);var f=0,g=true;if(e=="[object Array]"){if(f=a.length,g=f==c.length)for(;f--;)if(!(g=f in a==f in c&&q(a[f],c[f],d)))break}else{if("constructor"in a!="constructor"in c||a.constructor!=c.constructor)return false;for(var h in a)if(b.has(a,h)&&(f++,!(g=b.has(c,h)&&q(a[h],c[h],d))))break;if(g){for(h in c)if(b.has(c,
+h)&&!f--)break;g=!f}}d.pop();return g}var r=this,G=r._,n={},k=Array.prototype,o=Object.prototype,i=k.slice,H=k.unshift,l=o.toString,I=o.hasOwnProperty,w=k.forEach,x=k.map,y=k.reduce,z=k.reduceRight,A=k.filter,B=k.every,C=k.some,p=k.indexOf,D=k.lastIndexOf,o=Array.isArray,J=Object.keys,s=Function.prototype.bind,b=function(a){return new m(a)};if(typeof exports!=="undefined"){if(typeof module!=="undefined"&&module.exports)exports=module.exports=b;exports._=b}else r._=b;b.VERSION="1.3.1";var j=b.each=
+b.forEach=function(a,c,d){if(a!=null)if(w&&a.forEach===w)a.forEach(c,d);else if(a.length===+a.length)for(var e=0,f=a.length;e<f;e++){if(e in a&&c.call(d,a[e],e,a)===n)break}else for(e in a)if(b.has(a,e)&&c.call(d,a[e],e,a)===n)break};b.map=b.collect=function(a,c,b){var e=[];if(a==null)return e;if(x&&a.map===x)return a.map(c,b);j(a,function(a,g,h){e[e.length]=c.call(b,a,g,h)});if(a.length===+a.length)e.length=a.length;return e};b.reduce=b.foldl=b.inject=function(a,c,d,e){var f=arguments.length>2;a==
+null&&(a=[]);if(y&&a.reduce===y)return e&&(c=b.bind(c,e)),f?a.reduce(c,d):a.reduce(c);j(a,function(a,b,i){f?d=c.call(e,d,a,b,i):(d=a,f=true)});if(!f)throw new TypeError("Reduce of empty array with no initial value");return d};b.reduceRight=b.foldr=function(a,c,d,e){var f=arguments.length>2;a==null&&(a=[]);if(z&&a.reduceRight===z)return e&&(c=b.bind(c,e)),f?a.reduceRight(c,d):a.reduceRight(c);var g=b.toArray(a).reverse();e&&!f&&(c=b.bind(c,e));return f?b.reduce(g,c,d,e):b.reduce(g,c)};b.find=b.detect=
+function(a,c,b){var e;E(a,function(a,g,h){if(c.call(b,a,g,h))return e=a,true});return e};b.filter=b.select=function(a,c,b){var e=[];if(a==null)return e;if(A&&a.filter===A)return a.filter(c,b);j(a,function(a,g,h){c.call(b,a,g,h)&&(e[e.length]=a)});return e};b.reject=function(a,c,b){var e=[];if(a==null)return e;j(a,function(a,g,h){c.call(b,a,g,h)||(e[e.length]=a)});return e};b.every=b.all=function(a,c,b){var e=true;if(a==null)return e;if(B&&a.every===B)return a.every(c,b);j(a,function(a,g,h){if(!(e=
+e&&c.call(b,a,g,h)))return n});return e};var E=b.some=b.any=function(a,c,d){c||(c=b.identity);var e=false;if(a==null)return e;if(C&&a.some===C)return a.some(c,d);j(a,function(a,b,h){if(e||(e=c.call(d,a,b,h)))return n});return!!e};b.include=b.contains=function(a,c){var b=false;if(a==null)return b;return p&&a.indexOf===p?a.indexOf(c)!=-1:b=E(a,function(a){return a===c})};b.invoke=function(a,c){var d=i.call(arguments,2);return b.map(a,function(a){return(b.isFunction(c)?c||a:a[c]).apply(a,d)})};b.pluck=
+function(a,c){return b.map(a,function(a){return a[c]})};b.max=function(a,c,d){if(!c&&b.isArray(a))return Math.max.apply(Math,a);if(!c&&b.isEmpty(a))return-Infinity;var e={computed:-Infinity};j(a,function(a,b,h){b=c?c.call(d,a,b,h):a;b>=e.computed&&(e={value:a,computed:b})});return e.value};b.min=function(a,c,d){if(!c&&b.isArray(a))return Math.min.apply(Math,a);if(!c&&b.isEmpty(a))return Infinity;var e={computed:Infinity};j(a,function(a,b,h){b=c?c.call(d,a,b,h):a;b<e.computed&&(e={value:a,computed:b})});
+return e.value};b.shuffle=function(a){var b=[],d;j(a,function(a,f){f==0?b[0]=a:(d=Math.floor(Math.random()*(f+1)),b[f]=b[d],b[d]=a)});return b};b.sortBy=function(a,c,d){return b.pluck(b.map(a,function(a,b,g){return{value:a,criteria:c.call(d,a,b,g)}}).sort(function(a,b){var c=a.criteria,d=b.criteria;return c<d?-1:c>d?1:0}),"value")};b.groupBy=function(a,c){var d={},e=b.isFunction(c)?c:function(a){return a[c]};j(a,function(a,b){var c=e(a,b);(d[c]||(d[c]=[])).push(a)});return d};b.sortedIndex=function(a,
+c,d){d||(d=b.identity);for(var e=0,f=a.length;e<f;){var g=e+f>>1;d(a[g])<d(c)?e=g+1:f=g}return e};b.toArray=function(a){return!a?[]:a.toArray?a.toArray():b.isArray(a)?i.call(a):b.isArguments(a)?i.call(a):b.values(a)};b.size=function(a){return b.toArray(a).length};b.first=b.head=function(a,b,d){return b!=null&&!d?i.call(a,0,b):a[0]};b.initial=function(a,b,d){return i.call(a,0,a.length-(b==null||d?1:b))};b.last=function(a,b,d){return b!=null&&!d?i.call(a,Math.max(a.length-b,0)):a[a.length-1]};b.rest=
+b.tail=function(a,b,d){return i.call(a,b==null||d?1:b)};b.compact=function(a){return b.filter(a,function(a){return!!a})};b.flatten=function(a,c){return b.reduce(a,function(a,e){if(b.isArray(e))return a.concat(c?e:b.flatten(e));a[a.length]=e;return a},[])};b.without=function(a){return b.difference(a,i.call(arguments,1))};b.uniq=b.unique=function(a,c,d){var d=d?b.map(a,d):a,e=[];b.reduce(d,function(d,g,h){if(0==h||(c===true?b.last(d)!=g:!b.include(d,g)))d[d.length]=g,e[e.length]=a[h];return d},[]);
+return e};b.union=function(){return b.uniq(b.flatten(arguments,true))};b.intersection=b.intersect=function(a){var c=i.call(arguments,1);return b.filter(b.uniq(a),function(a){return b.every(c,function(c){return b.indexOf(c,a)>=0})})};b.difference=function(a){var c=b.flatten(i.call(arguments,1));return b.filter(a,function(a){return!b.include(c,a)})};b.zip=function(){for(var a=i.call(arguments),c=b.max(b.pluck(a,"length")),d=Array(c),e=0;e<c;e++)d[e]=b.pluck(a,""+e);return d};b.indexOf=function(a,c,
+d){if(a==null)return-1;var e;if(d)return d=b.sortedIndex(a,c),a[d]===c?d:-1;if(p&&a.indexOf===p)return a.indexOf(c);for(d=0,e=a.length;d<e;d++)if(d in a&&a[d]===c)return d;return-1};b.lastIndexOf=function(a,b){if(a==null)return-1;if(D&&a.lastIndexOf===D)return a.lastIndexOf(b);for(var d=a.length;d--;)if(d in a&&a[d]===b)return d;return-1};b.range=function(a,b,d){arguments.length<=1&&(b=a||0,a=0);for(var d=arguments[2]||1,e=Math.max(Math.ceil((b-a)/d),0),f=0,g=Array(e);f<e;)g[f++]=a,a+=d;return g};
+var F=function(){};b.bind=function(a,c){var d,e;if(a.bind===s&&s)return s.apply(a,i.call(arguments,1));if(!b.isFunction(a))throw new TypeError;e=i.call(arguments,2);return d=function(){if(!(this instanceof d))return a.apply(c,e.concat(i.call(arguments)));F.prototype=a.prototype;var b=new F,g=a.apply(b,e.concat(i.call(arguments)));return Object(g)===g?g:b}};b.bindAll=function(a){var c=i.call(arguments,1);c.length==0&&(c=b.functions(a));j(c,function(c){a[c]=b.bind(a[c],a)});return a};b.memoize=function(a,
+c){var d={};c||(c=b.identity);return function(){var e=c.apply(this,arguments);return b.has(d,e)?d[e]:d[e]=a.apply(this,arguments)}};b.delay=function(a,b){var d=i.call(arguments,2);return setTimeout(function(){return a.apply(a,d)},b)};b.defer=function(a){return b.delay.apply(b,[a,1].concat(i.call(arguments,1)))};b.throttle=function(a,c){var d,e,f,g,h,i=b.debounce(function(){h=g=false},c);return function(){d=this;e=arguments;var b;f||(f=setTimeout(function(){f=null;h&&a.apply(d,e);i()},c));g?h=true:
+a.apply(d,e);i();g=true}};b.debounce=function(a,b){var d;return function(){var e=this,f=arguments;clearTimeout(d);d=setTimeout(function(){d=null;a.apply(e,f)},b)}};b.once=function(a){var b=false,d;return function(){if(b)return d;b=true;return d=a.apply(this,arguments)}};b.wrap=function(a,b){return function(){var d=[a].concat(i.call(arguments,0));return b.apply(this,d)}};b.compose=function(){var a=arguments;return function(){for(var b=arguments,d=a.length-1;d>=0;d--)b=[a[d].apply(this,b)];return b[0]}};
+b.after=function(a,b){return a<=0?b():function(){if(--a<1)return b.apply(this,arguments)}};b.keys=J||function(a){if(a!==Object(a))throw new TypeError("Invalid object");var c=[],d;for(d in a)b.has(a,d)&&(c[c.length]=d);return c};b.values=function(a){return b.map(a,b.identity)};b.functions=b.methods=function(a){var c=[],d;for(d in a)b.isFunction(a[d])&&c.push(d);return c.sort()};b.extend=function(a){j(i.call(arguments,1),function(b){for(var d in b)a[d]=b[d]});return a};b.defaults=function(a){j(i.call(arguments,
+1),function(b){for(var d in b)a[d]==null&&(a[d]=b[d])});return a};b.clone=function(a){return!b.isObject(a)?a:b.isArray(a)?a.slice():b.extend({},a)};b.tap=function(a,b){b(a);return a};b.isEqual=function(a,b){return q(a,b,[])};b.isEmpty=function(a){if(b.isArray(a)||b.isString(a))return a.length===0;for(var c in a)if(b.has(a,c))return false;return true};b.isElement=function(a){return!!(a&&a.nodeType==1)};b.isArray=o||function(a){return l.call(a)=="[object Array]"};b.isObject=function(a){return a===Object(a)};
+b.isArguments=function(a){return l.call(a)=="[object Arguments]"};if(!b.isArguments(arguments))b.isArguments=function(a){return!(!a||!b.has(a,"callee"))};b.isFunction=function(a){return l.call(a)=="[object Function]"};b.isString=function(a){return l.call(a)=="[object String]"};b.isNumber=function(a){return l.call(a)=="[object Number]"};b.isNaN=function(a){return a!==a};b.isBoolean=function(a){return a===true||a===false||l.call(a)=="[object Boolean]"};b.isDate=function(a){return l.call(a)=="[object Date]"};
+b.isRegExp=function(a){return l.call(a)=="[object RegExp]"};b.isNull=function(a){return a===null};b.isUndefined=function(a){return a===void 0};b.has=function(a,b){return I.call(a,b)};b.noConflict=function(){r._=G;return this};b.identity=function(a){return a};b.times=function(a,b,d){for(var e=0;e<a;e++)b.call(d,e)};b.escape=function(a){return(""+a).replace(/&/g,"&amp;").replace(/</g,"&lt;").replace(/>/g,"&gt;").replace(/"/g,"&quot;").replace(/'/g,"&#x27;").replace(/\//g,"&#x2F;")};b.mixin=function(a){j(b.functions(a),
+function(c){K(c,b[c]=a[c])})};var L=0;b.uniqueId=function(a){var b=L++;return a?a+b:b};b.templateSettings={evaluate:/<%([\s\S]+?)%>/g,interpolate:/<%=([\s\S]+?)%>/g,escape:/<%-([\s\S]+?)%>/g};var t=/.^/,u=function(a){return a.replace(/\\\\/g,"\\").replace(/\\'/g,"'")};b.template=function(a,c){var d=b.templateSettings,d="var __p=[],print=function(){__p.push.apply(__p,arguments);};with(obj||{}){__p.push('"+a.replace(/\\/g,"\\\\").replace(/'/g,"\\'").replace(d.escape||t,function(a,b){return"',_.escape("+
+u(b)+"),'"}).replace(d.interpolate||t,function(a,b){return"',"+u(b)+",'"}).replace(d.evaluate||t,function(a,b){return"');"+u(b).replace(/[\r\n\t]/g," ")+";__p.push('"}).replace(/\r/g,"\\r").replace(/\n/g,"\\n").replace(/\t/g,"\\t")+"');}return __p.join('');",e=new Function("obj","_",d);return c?e(c,b):function(a){return e.call(this,a,b)}};b.chain=function(a){return b(a).chain()};var m=function(a){this._wrapped=a};b.prototype=m.prototype;var v=function(a,c){return c?b(a).chain():a},K=function(a,c){m.prototype[a]=
+function(){var a=i.call(arguments);H.call(a,this._wrapped);return v(c.apply(b,a),this._chain)}};b.mixin(b);j("pop,push,reverse,shift,sort,splice,unshift".split(","),function(a){var b=k[a];m.prototype[a]=function(){var d=this._wrapped;b.apply(d,arguments);var e=d.length;(a=="shift"||a=="splice")&&e===0&&delete d[0];return v(d,this._chain)}});j(["concat","join","slice"],function(a){var b=k[a];m.prototype[a]=function(){return v(b.apply(this._wrapped,arguments),this._chain)}});m.prototype.chain=function(){this._chain=
+true;return this};m.prototype.value=function(){return this._wrapped}}).call(this);

docs/_build/html/_static/websupport.js

@@ -0,0 +1,808 @@
+/*
+ * websupport.js
+ * ~~~~~~~~~~~~~
+ *
+ * sphinx.websupport utilities for all documentation.
+ *
+ * :copyright: Copyright 2007-2017 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+
+(function($) {
+  $.fn.autogrow = function() {
+    return this.each(function() {
+    var textarea = this;
+
+    $.fn.autogrow.resize(textarea);
+
+    $(textarea)
+      .focus(function() {
+        textarea.interval = setInterval(function() {
+          $.fn.autogrow.resize(textarea);
+        }, 500);
+      })
+      .blur(function() {
+        clearInterval(textarea.interval);
+      });
+    });
+  };
+
+  $.fn.autogrow.resize = function(textarea) {
+    var lineHeight = parseInt($(textarea).css('line-height'), 10);
+    var lines = textarea.value.split('\n');
+    var columns = textarea.cols;
+    var lineCount = 0;
+    $.each(lines, function() {
+      lineCount += Math.ceil(this.length / columns) || 1;
+    });
+    var height = lineHeight * (lineCount + 1);
+    $(textarea).css('height', height);
+  };
+})(jQuery);
+
+(function($) {
+  var comp, by;
+
+  function init() {
+    initEvents();
+    initComparator();
+  }
+
+  function initEvents() {
+    $(document).on("click", 'a.comment-close', function(event) {
+      event.preventDefault();
+      hide($(this).attr('id').substring(2));
+    });
+    $(document).on("click", 'a.vote', function(event) {
+      event.preventDefault();
+      handleVote($(this));
+    });
+    $(document).on("click", 'a.reply', function(event) {
+      event.preventDefault();
+      openReply($(this).attr('id').substring(2));
+    });
+    $(document).on("click", 'a.close-reply', function(event) {
+      event.preventDefault();
+      closeReply($(this).attr('id').substring(2));
+    });
+    $(document).on("click", 'a.sort-option', function(event) {
+      event.preventDefault();
+      handleReSort($(this));
+    });
+    $(document).on("click", 'a.show-proposal', function(event) {
+      event.preventDefault();
+      showProposal($(this).attr('id').substring(2));
+    });
+    $(document).on("click", 'a.hide-proposal', function(event) {
+      event.preventDefault();
+      hideProposal($(this).attr('id').substring(2));
+    });
+    $(document).on("click", 'a.show-propose-change', function(event) {
+      event.preventDefault();
+      showProposeChange($(this).attr('id').substring(2));
+    });
+    $(document).on("click", 'a.hide-propose-change', function(event) {
+      event.preventDefault();
+      hideProposeChange($(this).attr('id').substring(2));
+    });
+    $(document).on("click", 'a.accept-comment', function(event) {
+      event.preventDefault();
+      acceptComment($(this).attr('id').substring(2));
+    });
+    $(document).on("click", 'a.delete-comment', function(event) {
+      event.preventDefault();
+      deleteComment($(this).attr('id').substring(2));
+    });
+    $(document).on("click", 'a.comment-markup', function(event) {
+      event.preventDefault();
+      toggleCommentMarkupBox($(this).attr('id').substring(2));
+    });
+  }
+
+  /**
+   * Set comp, which is a comparator function used for sorting and
+   * inserting comments into the list.
+   */
+  function setComparator() {
+    // If the first three letters are "asc", sort in ascending order
+    // and remove the prefix.
+    if (by.substring(0,3) == 'asc') {
+      var i = by.substring(3);
+      comp = function(a, b) { return a[i] - b[i]; };
+    } else {
+      // Otherwise sort in descending order.
+      comp = function(a, b) { return b[by] - a[by]; };
+    }
+
+    // Reset link styles and format the selected sort option.
+    $('a.sel').attr('href', '#').removeClass('sel');
+    $('a.by' + by).removeAttr('href').addClass('sel');
+  }
+
+  /**
+   * Create a comp function. If the user has preferences stored in
+   * the sortBy cookie, use those, otherwise use the default.
+   */
+  function initComparator() {
+    by = 'rating'; // Default to sort by rating.
+    // If the sortBy cookie is set, use that instead.
+    if (document.cookie.length > 0) {
+      var start = document.cookie.indexOf('sortBy=');
+      if (start != -1) {
+        start = start + 7;
+        var end = document.cookie.indexOf(";", start);
+        if (end == -1) {
+          end = document.cookie.length;
+          by = unescape(document.cookie.substring(start, end));
+        }
+      }
+    }
+    setComparator();
+  }
+
+  /**
+   * Show a comment div.
+   */
+  function show(id) {
+    $('#ao' + id).hide();
+    $('#ah' + id).show();
+    var context = $.extend({id: id}, opts);
+    var popup = $(renderTemplate(popupTemplate, context)).hide();
+    popup.find('textarea[name="proposal"]').hide();
+    popup.find('a.by' + by).addClass('sel');
+    var form = popup.find('#cf' + id);
+    form.submit(function(event) {
+      event.preventDefault();
+      addComment(form);
+    });
+    $('#s' + id).after(popup);
+    popup.slideDown('fast', function() {
+      getComments(id);
+    });
+  }
+
+  /**
+   * Hide a comment div.
+   */
+  function hide(id) {
+    $('#ah' + id).hide();
+    $('#ao' + id).show();
+    var div = $('#sc' + id);
+    div.slideUp('fast', function() {
+      div.remove();
+    });
+  }
+
+  /**
+   * Perform an ajax request to get comments for a node
+   * and insert the comments into the comments tree.
+   */
+  function getComments(id) {
+    $.ajax({
+     type: 'GET',
+     url: opts.getCommentsURL,
+     data: {node: id},
+     success: function(data, textStatus, request) {
+       var ul = $('#cl' + id);
+       var speed = 100;
+       $('#cf' + id)
+         .find('textarea[name="proposal"]')
+         .data('source', data.source);
+
+       if (data.comments.length === 0) {
+         ul.html('<li>No comments yet.</li>');
+         ul.data('empty', true);
+       } else {
+         // If there are comments, sort them and put them in the list.
+         var comments = sortComments(data.comments);
+         speed = data.comments.length * 100;
+         appendComments(comments, ul);
+         ul.data('empty', false);
+       }
+       $('#cn' + id).slideUp(speed + 200);
+       ul.slideDown(speed);
+     },
+     error: function(request, textStatus, error) {
+       showError('Oops, there was a problem retrieving the comments.');
+     },
+     dataType: 'json'
+    });
+  }
+
+  /**
+   * Add a comment via ajax and insert the comment into the comment tree.
+   */
+  function addComment(form) {
+    var node_id = form.find('input[name="node"]').val();
+    var parent_id = form.find('input[name="parent"]').val();
+    var text = form.find('textarea[name="comment"]').val();
+    var proposal = form.find('textarea[name="proposal"]').val();
+
+    if (text == '') {
+      showError('Please enter a comment.');
+      return;
+    }
+
+    // Disable the form that is being submitted.
+    form.find('textarea,input').attr('disabled', 'disabled');
+
+    // Send the comment to the server.
+    $.ajax({
+      type: "POST",
+      url: opts.addCommentURL,
+      dataType: 'json',
+      data: {
+        node: node_id,
+        parent: parent_id,
+        text: text,
+        proposal: proposal
+      },
+      success: function(data, textStatus, error) {
+        // Reset the form.
+        if (node_id) {
+          hideProposeChange(node_id);
+        }
+        form.find('textarea')
+          .val('')
+          .add(form.find('input'))
+          .removeAttr('disabled');
+	var ul = $('#cl' + (node_id || parent_id));
+        if (ul.data('empty')) {
+          $(ul).empty();
+          ul.data('empty', false);
+        }
+        insertComment(data.comment);
+        var ao = $('#ao' + node_id);
+        ao.find('img').attr({'src': opts.commentBrightImage});
+        if (node_id) {
+          // if this was a "root" comment, remove the commenting box
+          // (the user can get it back by reopening the comment popup)
+          $('#ca' + node_id).slideUp();
+        }
+      },
+      error: function(request, textStatus, error) {
+        form.find('textarea,input').removeAttr('disabled');
+        showError('Oops, there was a problem adding the comment.');
+      }
+    });
+  }
+
+  /**
+   * Recursively append comments to the main comment list and children
+   * lists, creating the comment tree.
+   */
+  function appendComments(comments, ul) {
+    $.each(comments, function() {
+      var div = createCommentDiv(this);
+      ul.append($(document.createElement('li')).html(div));
+      appendComments(this.children, div.find('ul.comment-children'));
+      // To avoid stagnating data, don't store the comments children in data.
+      this.children = null;
+      div.data('comment', this);
+    });
+  }
+
+  /**
+   * After adding a new comment, it must be inserted in the correct
+   * location in the comment tree.
+   */
+  function insertComment(comment) {
+    var div = createCommentDiv(comment);
+
+    // To avoid stagnating data, don't store the comments children in data.
+    comment.children = null;
+    div.data('comment', comment);
+
+    var ul = $('#cl' + (comment.node || comment.parent));
+    var siblings = getChildren(ul);
+
+    var li = $(document.createElement('li'));
+    li.hide();
+
+    // Determine where in the parents children list to insert this comment.
+    for(i=0; i < siblings.length; i++) {
+      if (comp(comment, siblings[i]) <= 0) {
+        $('#cd' + siblings[i].id)
+          .parent()
+          .before(li.html(div));
+        li.slideDown('fast');
+        return;
+      }
+    }
+
+    // If we get here, this comment rates lower than all the others,
+    // or it is the only comment in the list.
+    ul.append(li.html(div));
+    li.slideDown('fast');
+  }
+
+  function acceptComment(id) {
+    $.ajax({
+      type: 'POST',
+      url: opts.acceptCommentURL,
+      data: {id: id},
+      success: function(data, textStatus, request) {
+        $('#cm' + id).fadeOut('fast');
+        $('#cd' + id).removeClass('moderate');
+      },
+      error: function(request, textStatus, error) {
+        showError('Oops, there was a problem accepting the comment.');
+      }
+    });
+  }
+
+  function deleteComment(id) {
+    $.ajax({
+      type: 'POST',
+      url: opts.deleteCommentURL,
+      data: {id: id},
+      success: function(data, textStatus, request) {
+        var div = $('#cd' + id);
+        if (data == 'delete') {
+          // Moderator mode: remove the comment and all children immediately
+          div.slideUp('fast', function() {
+            div.remove();
+          });
+          return;
+        }
+        // User mode: only mark the comment as deleted
+        div
+          .find('span.user-id:first')
+          .text('[deleted]').end()
+          .find('div.comment-text:first')
+          .text('[deleted]').end()
+          .find('#cm' + id + ', #dc' + id + ', #ac' + id + ', #rc' + id +
+                ', #sp' + id + ', #hp' + id + ', #cr' + id + ', #rl' + id)
+          .remove();
+        var comment = div.data('comment');
+        comment.username = '[deleted]';
+        comment.text = '[deleted]';
+        div.data('comment', comment);
+      },
+      error: function(request, textStatus, error) {
+        showError('Oops, there was a problem deleting the comment.');
+      }
+    });
+  }
+
+  function showProposal(id) {
+    $('#sp' + id).hide();
+    $('#hp' + id).show();
+    $('#pr' + id).slideDown('fast');
+  }
+
+  function hideProposal(id) {
+    $('#hp' + id).hide();
+    $('#sp' + id).show();
+    $('#pr' + id).slideUp('fast');
+  }
+
+  function showProposeChange(id) {
+    $('#pc' + id).hide();
+    $('#hc' + id).show();
+    var textarea = $('#pt' + id);
+    textarea.val(textarea.data('source'));
+    $.fn.autogrow.resize(textarea[0]);
+    textarea.slideDown('fast');
+  }
+
+  function hideProposeChange(id) {
+    $('#hc' + id).hide();
+    $('#pc' + id).show();
+    var textarea = $('#pt' + id);
+    textarea.val('').removeAttr('disabled');
+    textarea.slideUp('fast');
+  }
+
+  function toggleCommentMarkupBox(id) {
+    $('#mb' + id).toggle();
+  }
+
+  /** Handle when the user clicks on a sort by link. */
+  function handleReSort(link) {
+    var classes = link.attr('class').split(/\s+/);
+    for (var i=0; i<classes.length; i++) {
+      if (classes[i] != 'sort-option') {
+	by = classes[i].substring(2);
+      }
+    }
+    setComparator();
+    // Save/update the sortBy cookie.
+    var expiration = new Date();
+    expiration.setDate(expiration.getDate() + 365);
+    document.cookie= 'sortBy=' + escape(by) +
+                     ';expires=' + expiration.toUTCString();
+    $('ul.comment-ul').each(function(index, ul) {
+      var comments = getChildren($(ul), true);
+      comments = sortComments(comments);
+      appendComments(comments, $(ul).empty());
+    });
+  }
+
+  /**
+   * Function to process a vote when a user clicks an arrow.
+   */
+  function handleVote(link) {
+    if (!opts.voting) {
+      showError("You'll need to login to vote.");
+      return;
+    }
+
+    var id = link.attr('id');
+    if (!id) {
+      // Didn't click on one of the voting arrows.
+      return;
+    }
+    // If it is an unvote, the new vote value is 0,
+    // Otherwise it's 1 for an upvote, or -1 for a downvote.
+    var value = 0;
+    if (id.charAt(1) != 'u') {
+      value = id.charAt(0) == 'u' ? 1 : -1;
+    }
+    // The data to be sent to the server.
+    var d = {
+      comment_id: id.substring(2),
+      value: value
+    };
+
+    // Swap the vote and unvote links.
+    link.hide();
+    $('#' + id.charAt(0) + (id.charAt(1) == 'u' ? 'v' : 'u') + d.comment_id)
+      .show();
+
+    // The div the comment is displayed in.
+    var div = $('div#cd' + d.comment_id);
+    var data = div.data('comment');
+
+    // If this is not an unvote, and the other vote arrow has
+    // already been pressed, unpress it.
+    if ((d.value !== 0) && (data.vote === d.value * -1)) {
+      $('#' + (d.value == 1 ? 'd' : 'u') + 'u' + d.comment_id).hide();
+      $('#' + (d.value == 1 ? 'd' : 'u') + 'v' + d.comment_id).show();
+    }
+
+    // Update the comments rating in the local data.
+    data.rating += (data.vote === 0) ? d.value : (d.value - data.vote);
+    data.vote = d.value;
+    div.data('comment', data);
+
+    // Change the rating text.
+    div.find('.rating:first')
+      .text(data.rating + ' point' + (data.rating == 1 ? '' : 's'));
+
+    // Send the vote information to the server.
+    $.ajax({
+      type: "POST",
+      url: opts.processVoteURL,
+      data: d,
+      error: function(request, textStatus, error) {
+        showError('Oops, there was a problem casting that vote.');
+      }
+    });
+  }
+
+  /**
+   * Open a reply form used to reply to an existing comment.
+   */
+  function openReply(id) {
+    // Swap out the reply link for the hide link
+    $('#rl' + id).hide();
+    $('#cr' + id).show();
+
+    // Add the reply li to the children ul.
+    var div = $(renderTemplate(replyTemplate, {id: id})).hide();
+    $('#cl' + id)
+      .prepend(div)
+      // Setup the submit handler for the reply form.
+      .find('#rf' + id)
+      .submit(function(event) {
+        event.preventDefault();
+        addComment($('#rf' + id));
+        closeReply(id);
+      })
+      .find('input[type=button]')
+      .click(function() {
+        closeReply(id);
+      });
+    div.slideDown('fast', function() {
+      $('#rf' + id).find('textarea').focus();
+    });
+  }
+
+  /**
+   * Close the reply form opened with openReply.
+   */
+  function closeReply(id) {
+    // Remove the reply div from the DOM.
+    $('#rd' + id).slideUp('fast', function() {
+      $(this).remove();
+    });
+
+    // Swap out the hide link for the reply link
+    $('#cr' + id).hide();
+    $('#rl' + id).show();
+  }
+
+  /**
+   * Recursively sort a tree of comments using the comp comparator.
+   */
+  function sortComments(comments) {
+    comments.sort(comp);
+    $.each(comments, function() {
+      this.children = sortComments(this.children);
+    });
+    return comments;
+  }
+
+  /**
+   * Get the children comments from a ul. If recursive is true,
+   * recursively include childrens' children.
+   */
+  function getChildren(ul, recursive) {
+    var children = [];
+    ul.children().children("[id^='cd']")
+      .each(function() {
+        var comment = $(this).data('comment');
+        if (recursive)
+          comment.children = getChildren($(this).find('#cl' + comment.id), true);
+        children.push(comment);
+      });
+    return children;
+  }
+
+  /** Create a div to display a comment in. */
+  function createCommentDiv(comment) {
+    if (!comment.displayed && !opts.moderator) {
+      return $('<div class="moderate">Thank you!  Your comment will show up '
+               + 'once it is has been approved by a moderator.</div>');
+    }
+    // Prettify the comment rating.
+    comment.pretty_rating = comment.rating + ' point' +
+      (comment.rating == 1 ? '' : 's');
+    // Make a class (for displaying not yet moderated comments differently)
+    comment.css_class = comment.displayed ? '' : ' moderate';
+    // Create a div for this comment.
+    var context = $.extend({}, opts, comment);
+    var div = $(renderTemplate(commentTemplate, context));
+
+    // If the user has voted on this comment, highlight the correct arrow.
+    if (comment.vote) {
+      var direction = (comment.vote == 1) ? 'u' : 'd';
+      div.find('#' + direction + 'v' + comment.id).hide();
+      div.find('#' + direction + 'u' + comment.id).show();
+    }
+
+    if (opts.moderator || comment.text != '[deleted]') {
+      div.find('a.reply').show();
+      if (comment.proposal_diff)
+        div.find('#sp' + comment.id).show();
+      if (opts.moderator && !comment.displayed)
+        div.find('#cm' + comment.id).show();
+      if (opts.moderator || (opts.username == comment.username))
+        div.find('#dc' + comment.id).show();
+    }
+    return div;
+  }
+
+  /**
+   * A simple template renderer. Placeholders such as <%id%> are replaced
+   * by context['id'] with items being escaped. Placeholders such as <#id#>
+   * are not escaped.
+   */
+  function renderTemplate(template, context) {
+    var esc = $(document.createElement('div'));
+
+    function handle(ph, escape) {
+      var cur = context;
+      $.each(ph.split('.'), function() {
+        cur = cur[this];
+      });
+      return escape ? esc.text(cur || "").html() : cur;
+    }
+
+    return template.replace(/<([%#])([\w\.]*)\1>/g, function() {
+      return handle(arguments[2], arguments[1] == '%' ? true : false);
+    });
+  }
+
+  /** Flash an error message briefly. */
+  function showError(message) {
+    $(document.createElement('div')).attr({'class': 'popup-error'})
+      .append($(document.createElement('div'))
+               .attr({'class': 'error-message'}).text(message))
+      .appendTo('body')
+      .fadeIn("slow")
+      .delay(2000)
+      .fadeOut("slow");
+  }
+
+  /** Add a link the user uses to open the comments popup. */
+  $.fn.comment = function() {
+    return this.each(function() {
+      var id = $(this).attr('id').substring(1);
+      var count = COMMENT_METADATA[id];
+      var title = count + ' comment' + (count == 1 ? '' : 's');
+      var image = count > 0 ? opts.commentBrightImage : opts.commentImage;
+      var addcls = count == 0 ? ' nocomment' : '';
+      $(this)
+        .append(
+          $(document.createElement('a')).attr({
+            href: '#',
+            'class': 'sphinx-comment-open' + addcls,
+            id: 'ao' + id
+          })
+            .append($(document.createElement('img')).attr({
+              src: image,
+              alt: 'comment',
+              title: title
+            }))
+            .click(function(event) {
+              event.preventDefault();
+              show($(this).attr('id').substring(2));
+            })
+        )
+        .append(
+          $(document.createElement('a')).attr({
+            href: '#',
+            'class': 'sphinx-comment-close hidden',
+            id: 'ah' + id
+          })
+            .append($(document.createElement('img')).attr({
+              src: opts.closeCommentImage,
+              alt: 'close',
+              title: 'close'
+            }))
+            .click(function(event) {
+              event.preventDefault();
+              hide($(this).attr('id').substring(2));
+            })
+        );
+    });
+  };
+
+  var opts = {
+    processVoteURL: '/_process_vote',
+    addCommentURL: '/_add_comment',
+    getCommentsURL: '/_get_comments',
+    acceptCommentURL: '/_accept_comment',
+    deleteCommentURL: '/_delete_comment',
+    commentImage: '/static/_static/comment.png',
+    closeCommentImage: '/static/_static/comment-close.png',
+    loadingImage: '/static/_static/ajax-loader.gif',
+    commentBrightImage: '/static/_static/comment-bright.png',
+    upArrow: '/static/_static/up.png',
+    downArrow: '/static/_static/down.png',
+    upArrowPressed: '/static/_static/up-pressed.png',
+    downArrowPressed: '/static/_static/down-pressed.png',
+    voting: false,
+    moderator: false
+  };
+
+  if (typeof COMMENT_OPTIONS != "undefined") {
+    opts = jQuery.extend(opts, COMMENT_OPTIONS);
+  }
+
+  var popupTemplate = '\
+    <div class="sphinx-comments" id="sc<%id%>">\
+      <p class="sort-options">\
+        Sort by:\
+        <a href="#" class="sort-option byrating">best rated</a>\
+        <a href="#" class="sort-option byascage">newest</a>\
+        <a href="#" class="sort-option byage">oldest</a>\
+      </p>\
+      <div class="comment-header">Comments</div>\
+      <div class="comment-loading" id="cn<%id%>">\
+        loading comments... <img src="<%loadingImage%>" alt="" /></div>\
+      <ul id="cl<%id%>" class="comment-ul"></ul>\
+      <div id="ca<%id%>">\
+      <p class="add-a-comment">Add a comment\
+        (<a href="#" class="comment-markup" id="ab<%id%>">markup</a>):</p>\
+      <div class="comment-markup-box" id="mb<%id%>">\
+        reStructured text markup: <i>*emph*</i>, <b>**strong**</b>, \
+        <code>``code``</code>, \
+        code blocks: <code>::</code> and an indented block after blank line</div>\
+      <form method="post" id="cf<%id%>" class="comment-form" action="">\
+        <textarea name="comment" cols="80"></textarea>\
+        <p class="propose-button">\
+          <a href="#" id="pc<%id%>" class="show-propose-change">\
+            Propose a change &#9657;\
+          </a>\
+          <a href="#" id="hc<%id%>" class="hide-propose-change">\
+            Propose a change &#9663;\
+          </a>\
+        </p>\
+        <textarea name="proposal" id="pt<%id%>" cols="80"\
+                  spellcheck="false"></textarea>\
+        <input type="submit" value="Add comment" />\
+        <input type="hidden" name="node" value="<%id%>" />\
+        <input type="hidden" name="parent" value="" />\
+      </form>\
+      </div>\
+    </div>';
+
+  var commentTemplate = '\
+    <div id="cd<%id%>" class="sphinx-comment<%css_class%>">\
+      <div class="vote">\
+        <div class="arrow">\
+          <a href="#" id="uv<%id%>" class="vote" title="vote up">\
+            <img src="<%upArrow%>" />\
+          </a>\
+          <a href="#" id="uu<%id%>" class="un vote" title="vote up">\
+            <img src="<%upArrowPressed%>" />\
+          </a>\
+        </div>\
+        <div class="arrow">\
+          <a href="#" id="dv<%id%>" class="vote" title="vote down">\
+            <img src="<%downArrow%>" id="da<%id%>" />\
+          </a>\
+          <a href="#" id="du<%id%>" class="un vote" title="vote down">\
+            <img src="<%downArrowPressed%>" />\
+          </a>\
+        </div>\
+      </div>\
+      <div class="comment-content">\
+        <p class="tagline comment">\
+          <span class="user-id"><%username%></span>\
+          <span class="rating"><%pretty_rating%></span>\
+          <span class="delta"><%time.delta%></span>\
+        </p>\
+        <div class="comment-text comment"><#text#></div>\
+        <p class="comment-opts comment">\
+          <a href="#" class="reply hidden" id="rl<%id%>">reply &#9657;</a>\
+          <a href="#" class="close-reply" id="cr<%id%>">reply &#9663;</a>\
+          <a href="#" id="sp<%id%>" class="show-proposal">proposal &#9657;</a>\
+          <a href="#" id="hp<%id%>" class="hide-proposal">proposal &#9663;</a>\
+          <a href="#" id="dc<%id%>" class="delete-comment hidden">delete</a>\
+          <span id="cm<%id%>" class="moderation hidden">\
+            <a href="#" id="ac<%id%>" class="accept-comment">accept</a>\
+          </span>\
+        </p>\
+        <pre class="proposal" id="pr<%id%>">\
+<#proposal_diff#>\
+        </pre>\
+          <ul class="comment-children" id="cl<%id%>"></ul>\
+        </div>\
+        <div class="clearleft"></div>\
+      </div>\
+    </div>';
+
+  var replyTemplate = '\
+    <li>\
+      <div class="reply-div" id="rd<%id%>">\
+        <form id="rf<%id%>">\
+          <textarea name="comment" cols="80"></textarea>\
+          <input type="submit" value="Add reply" />\
+          <input type="button" value="Cancel" />\
+          <input type="hidden" name="parent" value="<%id%>" />\
+          <input type="hidden" name="node" value="" />\
+        </form>\
+      </div>\
+    </li>';
+
+  $(document).ready(function() {
+    init();
+  });
+})(jQuery);
+
+$(document).ready(function() {
+  // add comment anchors for all paragraphs that are commentable
+  $('.sphinx-has-comment').comment();
+
+  // highlight search words in search results
+  $("div.context").each(function() {
+    var params = $.getQueryParameters();
+    var terms = (params.q) ? params.q[0].split(/\s+/) : [];
+    var result = $(this);
+    $.each(terms, function() {
+      result.highlightText(this.toLowerCase(), 'highlighted');
+    });
+  });
+
+  // directly open comment window if requested
+  var anchor = document.location.hash;
+  if (anchor.substring(0, 9) == '#comment-') {
+    $('#ao' + anchor.substring(9)).click();
+    document.location.hash = '#s' + anchor.substring(9);
+  }
+});

docs/_build/html/bugreport.html

@@ -0,0 +1,136 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Reporting Bugs &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    './',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="_static/jquery.js"></script>
+    <script type="text/javascript" src="_static/underscore.js"></script>
+    <script type="text/javascript" src="_static/doctools.js"></script>
+    <link rel="index" title="Index" href="genindex.html" />
+    <link rel="search" title="Search" href="search.html" />
+    <link rel="prev" title="Transformation Functions" href="ref/transforms.html" />
+   
+  <link rel="stylesheet" href="_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="reporting-bugs">
+<h1>Reporting Bugs<a class="headerlink" href="#reporting-bugs" title="Permalink to this headline">¶</a></h1>
+<p>pgloader is a software and as such contains bugs. Most bugs are easy to
+solve and taken care of in a short delay. For this to be possible though,
+bug reports need to follow those recommandations:</p>
+<blockquote>
+<div><ul class="simple">
+<li>include pgloader version,</li>
+<li>include problematic input and output,</li>
+<li>include a description of the output you expected,</li>
+<li>explain the difference between the ouput you have and the one you expected,</li>
+<li>include a self-reproducing test-case</li>
+</ul>
+</div></blockquote>
+<div class="section" id="test-cases-to-reproduce-bugs">
+<h2>Test Cases to Reproduce Bugs<a class="headerlink" href="#test-cases-to-reproduce-bugs" title="Permalink to this headline">¶</a></h2>
+<p>Use the <em>inline</em> source type to help reproduce a bug, as in the pgloader tests:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>LOAD CSV
+   FROM INLINE
+   INTO postgresql://dim@localhost/pgloader?public.&quot;HS&quot;
+
+   WITH truncate,
+        fields terminated by &#39;\t&#39;,
+        fields not enclosed,
+        fields escaped by backslash-quote,
+        quote identifiers
+
+    SET work_mem to &#39;128MB&#39;,
+        standard_conforming_strings to &#39;on&#39;,
+        application_name to &#39;my app name&#39;
+
+ BEFORE LOAD DO
+  $$ create extension if not exists hstore; $$,
+  $$ drop table if exists &quot;HS&quot;; $$,
+  $$ CREATE TABLE &quot;HS&quot;
+     (
+        id     serial primary key,
+        kv     hstore
+     )
+  $$;
+
+
+1     email=&gt;foo@example.com,a=&gt;b
+2     test=&gt;value
+3     a=&gt;b,c=&gt;&quot;quoted hstore value&quot;,d=&gt;other
+4     baddata
+</pre></div>
+</div>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="index.html">Documentation overview</a><ul>
+      <li>Previous: <a href="ref/transforms.html" title="previous chapter">Transformation Functions</a></li>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="_sources/bugreport.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/genindex.html

@@ -0,0 +1,88 @@
+
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Index &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    './',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="_static/jquery.js"></script>
+    <script type="text/javascript" src="_static/underscore.js"></script>
+    <script type="text/javascript" src="_static/doctools.js"></script>
+    <link rel="index" title="Index" href="#" />
+    <link rel="search" title="Search" href="search.html" />
+   
+  <link rel="stylesheet" href="_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+
+<h1 id="index">Index</h1>
+
+<div class="genindex-jumpbox">
+ 
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="index.html">Documentation overview</a><ul>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/index.html

@@ -0,0 +1,198 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Welcome to pgloader’s documentation! &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    './',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="_static/jquery.js"></script>
+    <script type="text/javascript" src="_static/underscore.js"></script>
+    <script type="text/javascript" src="_static/doctools.js"></script>
+    <link rel="index" title="Index" href="genindex.html" />
+    <link rel="search" title="Search" href="search.html" />
+    <link rel="next" title="Introduction" href="intro.html" />
+   
+  <link rel="stylesheet" href="_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="welcome-to-pgloader-s-documentation">
+<h1>Welcome to pgloader’s documentation!<a class="headerlink" href="#welcome-to-pgloader-s-documentation" title="Permalink to this headline">¶</a></h1>
+<div class="toctree-wrapper compound">
+<p class="caption"><span class="caption-text">Table Of Contents:</span></p>
+<ul>
+<li class="toctree-l1"><a class="reference internal" href="intro.html">Introduction</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="intro.html#continuous-migration">Continuous Migration</a></li>
+<li class="toctree-l2"><a class="reference internal" href="intro.html#commands">Commands</a></li>
+<li class="toctree-l2"><a class="reference internal" href="intro.html#command-line">Command Line</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial/tutorial.html">PgLoader Tutorial</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="tutorial/tutorial.html#pgloader-quick-start">PgLoader Quick Start</a></li>
+<li class="toctree-l2"><a class="reference internal" href="tutorial/tutorial.html#loading-csv-data-with-pgloader">Loading CSV Data with pgloader</a></li>
+<li class="toctree-l2"><a class="reference internal" href="tutorial/tutorial.html#loading-fixed-width-data-file-with-pgloader">Loading Fixed Width Data File with pgloader</a></li>
+<li class="toctree-l2"><a class="reference internal" href="tutorial/tutorial.html#loading-maxmind-geolite-data-with-pgloader">Loading MaxMind Geolite Data with pgloader</a></li>
+<li class="toctree-l2"><a class="reference internal" href="tutorial/tutorial.html#loading-dbase-files-with-pgloader">Loading dBase files with pgloader</a></li>
+<li class="toctree-l2"><a class="reference internal" href="tutorial/tutorial.html#loading-sqlite-files-with-pgloader">Loading SQLite files with pgloader</a></li>
+<li class="toctree-l2"><a class="reference internal" href="tutorial/tutorial.html#migrating-from-mysql-to-postgresql">Migrating from MySQL to PostgreSQL</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="pgloader.html">PgLoader Reference Manual</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="pgloader.html#arguments">Arguments</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pgloader.html#options">Options</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pgloader.html#batches-and-retry-behaviour">Batches And Retry Behaviour</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pgloader.html#a-note-about-performance">A Note About Performance</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pgloader.html#a-note-about-parallelism">A Note About Parallelism</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pgloader.html#source-formats">Source Formats</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pgloader.html#pgloader-commands-syntax">Pgloader Commands Syntax</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pgloader.html#templating-with-mustache">Templating with Mustache</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pgloader.html#common-clauses">Common Clauses</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="ref/csv.html">Loading CSV data</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="ref/csv.html#csv-source-specification-from">CSV Source Specification: FROM</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/csv.html#fields-specifications">Fields Specifications</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/csv.html#csv-loading-options-with">CSV Loading Options: WITH</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="ref/fixed.html">Loading Fixed Cols File Formats</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="ref/fixed.html#fixed-file-format-source-specification-from">Fixed File Format Source Specification: FROM</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/fixed.html#fields-specifications">Fields Specifications</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/fixed.html#fixed-file-format-loading-options-with">Fixed File Format Loading Options: WITH</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="ref/copy.html">Loading COPY Formatted Files</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="ref/copy.html#copy-formatted-files-source-specification-from">COPY Formatted Files Source Specification: FROM</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/copy.html#copy-formatted-file-options-with">COPY Formatted File Options: WITH</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="ref/dbf.html">Loading DBF data</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="ref/dbf.html#dbf-source-specification-from">DBF Source Specification: FROM</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/dbf.html#dbf-loading-options-with">DBF Loading Options: WITH</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="ref/ixf.html">Loading IXF Data</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="ref/ixf.html#ixf-source-specification-from">IXF Source Specification: FROM</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/ixf.html#ixf-loading-options-with">IXF Loading Options: WITH</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="ref/archive.html">Loading From an Archive</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="ref/archive.html#archive-source-specification-from">Archive Source Specification: FROM</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/archive.html#archive-sub-commands">Archive Sub Commands</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/archive.html#archive-final-sql-commands">Archive Final SQL Commands</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="ref/mysql.html">Migrating a MySQL Database to PostgreSQL</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="ref/mysql.html#mysql-database-source-specification-from">MySQL Database Source Specification: FROM</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/mysql.html#mysql-database-migration-options-with">MySQL Database Migration Options: WITH</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/mysql.html#mysql-database-casting-rules">MySQL Database Casting Rules</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/mysql.html#mysql-views-support">MySQL Views Support</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/mysql.html#mysql-partial-migration">MySQL Partial Migration</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/mysql.html#mysql-encoding-support">MySQL Encoding Support</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/mysql.html#mysql-schema-transformations">MySQL Schema Transformations</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/mysql.html#mysql-migration-limitations">MySQL Migration: limitations</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/mysql.html#default-mysql-casting-rules">Default MySQL Casting Rules</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="ref/sqlite.html">Migrating a SQLite database to PostgreSQL</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="ref/sqlite.html#sqlite-database-source-specification-from">SQLite Database Source Specification: FROM</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/sqlite.html#sqlite-database-migration-options-with">SQLite Database Migration Options: WITH</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/sqlite.html#sqlite-database-casting-rules">SQLite Database Casting Rules</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/sqlite.html#sqlite-database-partial-migrations">SQlite Database Partial Migrations</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/sqlite.html#default-sqlite-casting-rules">Default SQLite Casting Rules</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="ref/mssql.html">Migrating a MS SQL Database to PostgreSQL</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="ref/mssql.html#ms-sql-database-source-specification-from">MS SQL Database Source Specification: FROM</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/mssql.html#ms-sql-database-migration-options-with">MS SQL Database Migration Options: WITH</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/mssql.html#ms-sql-database-casting-rules">MS SQL Database Casting Rules</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/mssql.html#ms-sql-partial-migration">MS SQL Partial Migration</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/mssql.html#ms-sql-schema-transformations">MS SQL Schema Transformations</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/mssql.html#ms-sql-driver-setup-and-encoding">MS SQL Driver setup and encoding</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ref/mssql.html#default-ms-sql-casting-rules">Default MS SQL Casting Rules</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="ref/transforms.html">Transformation Functions</a></li>
+<li class="toctree-l1"><a class="reference internal" href="bugreport.html">Reporting Bugs</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="bugreport.html#test-cases-to-reproduce-bugs">Test Cases to Reproduce Bugs</a></li>
+</ul>
+</li>
+</ul>
+</div>
+</div>
+<div class="section" id="indices-and-tables">
+<h1>Indices and tables<a class="headerlink" href="#indices-and-tables" title="Permalink to this headline">¶</a></h1>
+<ul class="simple">
+<li><a class="reference internal" href="genindex.html"><span class="std std-ref">Index</span></a></li>
+<li><a class="reference internal" href="py-modindex.html"><span class="std std-ref">Module Index</span></a></li>
+<li><a class="reference internal" href="search.html"><span class="std std-ref">Search Page</span></a></li>
+</ul>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="#">Documentation overview</a><ul>
+      <li>Next: <a href="intro.html" title="next chapter">Introduction</a></li>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="_sources/index.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/intro.html

@@ -0,0 +1,151 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Introduction &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    './',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="_static/jquery.js"></script>
+    <script type="text/javascript" src="_static/underscore.js"></script>
+    <script type="text/javascript" src="_static/doctools.js"></script>
+    <link rel="index" title="Index" href="genindex.html" />
+    <link rel="search" title="Search" href="search.html" />
+    <link rel="next" title="PgLoader Tutorial" href="tutorial/tutorial.html" />
+    <link rel="prev" title="Welcome to pgloader’s documentation!" href="index.html" />
+   
+  <link rel="stylesheet" href="_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="introduction">
+<h1>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h1>
+<p>pgloader loads data from various sources into PostgreSQL. It can
+transform the data it reads on the fly and submit raw SQL before and
+after the loading.  It uses the <cite>COPY</cite> PostgreSQL protocol to stream
+the data into the server, and manages errors by filling a pair of
+<em>reject.dat</em> and <em>reject.log</em> files.</p>
+<p>pgloader knows how to read data from different kind of sources:</p>
+<blockquote>
+<div><ul class="simple">
+<li>Files
+* CSV
+* Fixed Format
+* DBF</li>
+<li>Databases
+* SQLite
+* MySQL
+* MS SQL Server</li>
+</ul>
+</div></blockquote>
+<p>The level of automation provided by pgloader depends on the data source
+type. In the case of CSV and Fixed Format files, a full description of the
+expected input properties must be given to pgloader. In the case of a
+database, pgloader connects to the live service and knows how to fetch the
+metadata it needs directly from it.</p>
+<div class="section" id="continuous-migration">
+<h2>Continuous Migration<a class="headerlink" href="#continuous-migration" title="Permalink to this headline">¶</a></h2>
+<p>pgloader is meant to migrate a whole database in a single command line and
+without any manual intervention. The goal is to be able to setup a
+<em>Continuous Integration</em> environment as described in the <a class="reference external" href="http://mysqltopgsql.com/project/">Project
+Methodology</a> document of the <a class="reference external" href="http://mysqltopgsql.com/project/">MySQL to
+PostgreSQL</a> webpage.</p>
+<blockquote>
+<div><ol class="arabic simple">
+<li>Setup your target PostgreSQL Architecture</li>
+<li>Fork a Continuous Integration environment that uses PostgreSQL</li>
+<li>Migrate the data over and over again every night, from production</li>
+<li>As soon as the CI is all green using PostgreSQL, schedule the D-Day</li>
+<li>Migrate without suprise and enjoy!</li>
+</ol>
+</div></blockquote>
+<p>In order to be able to follow this great methodology, you need tooling to
+implement the third step in a fully automated way. That’s pgloader.</p>
+</div>
+<div class="section" id="commands">
+<h2>Commands<a class="headerlink" href="#commands" title="Permalink to this headline">¶</a></h2>
+<p>pgloader implements its own <em>Command Language</em>, a DSL that allows to specify
+every aspect of the data load and migration to implement. Some of the
+features provided in the language are only available for a specific source
+type.</p>
+</div>
+<div class="section" id="command-line">
+<h2>Command Line<a class="headerlink" href="#command-line" title="Permalink to this headline">¶</a></h2>
+<p>The pgloader command line accepts those two variants:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">pgloader</span> <span class="p">[</span><span class="o">&lt;</span><span class="n">options</span><span class="o">&gt;</span><span class="p">]</span> <span class="p">[</span><span class="o">&lt;</span><span class="n">command</span><span class="o">-</span><span class="n">file</span><span class="o">&gt;</span><span class="p">]</span><span class="o">...</span>
+<span class="n">pgloader</span> <span class="p">[</span><span class="o">&lt;</span><span class="n">options</span><span class="o">&gt;</span><span class="p">]</span> <span class="n">SOURCE</span> <span class="n">TARGET</span>
+</pre></div>
+</div>
+<p>Either you have a <em>command-file</em> containing migration specifications in the
+pgloader <em>Command Language</em>, or you can give a <em>Source</em> for the data and a
+PostgreSQL database connection <em>Target</em> where to load the data into.</p>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="index.html">Documentation overview</a><ul>
+      <li>Previous: <a href="index.html" title="previous chapter">Welcome to pgloader’s documentation!</a></li>
+      <li>Next: <a href="tutorial/tutorial.html" title="next chapter">PgLoader Tutorial</a></li>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="_sources/intro.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/pgloader-usage-examples.html

@@ -0,0 +1,237 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Pgloader Usage Examples &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    './',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="_static/jquery.js"></script>
+    <script type="text/javascript" src="_static/underscore.js"></script>
+    <script type="text/javascript" src="_static/doctools.js"></script>
+    <link rel="index" title="Index" href="genindex.html" />
+    <link rel="search" title="Search" href="search.html" />
+   
+  <link rel="stylesheet" href="_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="pgloader-usage-examples">
+<h1>Pgloader Usage Examples<a class="headerlink" href="#pgloader-usage-examples" title="Permalink to this headline">¶</a></h1>
+<p>Currently not included, because redundant with the tutorial.</p>
+<div class="section" id="usage-examples">
+<h2>Usage Examples<a class="headerlink" href="#usage-examples" title="Permalink to this headline">¶</a></h2>
+<p>Review the command line options and pgloader’s version:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">pgloader</span> <span class="o">--</span><span class="n">help</span>
+<span class="n">pgloader</span> <span class="o">--</span><span class="n">version</span>
+</pre></div>
+</div>
+<div class="section" id="loading-from-a-complex-command">
+<h3>Loading from a complex command<a class="headerlink" href="#loading-from-a-complex-command" title="Permalink to this headline">¶</a></h3>
+<p>Use the command file as the pgloader command argument, pgloader will parse
+that file and execute the commands found in it:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">pgloader</span> <span class="o">--</span><span class="n">verbose</span> <span class="o">./</span><span class="n">test</span><span class="o">/</span><span class="n">csv</span><span class="o">-</span><span class="n">districts</span><span class="o">.</span><span class="n">load</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="csv">
+<h3>CSV<a class="headerlink" href="#csv" title="Permalink to this headline">¶</a></h3>
+<p>Load data from a CSV file into a pre-existing table in your database, having
+pgloader guess the CSV properties (separator, quote and escape character):</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>pgloader ./test/data/matching-1.csv pgsql:///pgloader?tablename=matching
+</pre></div>
+</div>
+<p>Load data from a CSV file into a pre-existing table in your database, with
+expanded options:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>pgloader --type csv                                   \
+         --field id --field field                     \
+         --with truncate                              \
+         --with &quot;fields terminated by &#39;,&#39;&quot;            \
+         ./test/data/matching-1.csv                   \
+         postgres:///pgloader?tablename=matching
+</pre></div>
+</div>
+<p>In that example the whole loading is driven from the command line, bypassing
+the need for writing a command in the pgloader command syntax entirely. As
+there’s no command though, the extra inforamtion needed must be provided on
+the command line using the <cite>–type</cite> and <cite>–field</cite> and <cite>–with</cite> switches.</p>
+<p>For documentation about the available syntaxes for the <cite>–field</cite> and
+<cite>–with</cite> switches, please refer to the CSV section later in the man page.</p>
+<p>Note also that the PostgreSQL URI includes the target <em>tablename</em>.</p>
+</div>
+<div class="section" id="reading-from-stdin">
+<h3>Reading from STDIN<a class="headerlink" href="#reading-from-stdin" title="Permalink to this headline">¶</a></h3>
+<p>File based pgloader sources can be loaded from the standard input, as in the
+following example:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>pgloader --type csv                                         \
+         --field &quot;usps,geoid,aland,awater,aland_sqmi,awater_sqmi,intptlat,intptlong&quot; \
+         --with &quot;skip header = 1&quot;                          \
+         --with &quot;fields terminated by &#39;\t&#39;&quot;                \
+         -                                                 \
+         postgresql:///pgloader?districts_longlat          \
+         &lt; test/data/2013_Gaz_113CDs_national.txt
+</pre></div>
+</div>
+<p>The dash (<cite>-</cite>) character as a source is used to mean <em>standard input</em>, as
+usual in Unix command lines. It’s possible to stream compressed content to
+pgloader with this technique, using the Unix pipe:</p>
+<blockquote>
+<div>gunzip -c source.gz | pgloader –type csv … - pgsql:///target?foo</div></blockquote>
+</div>
+<div class="section" id="loading-from-csv-available-through-http">
+<h3>Loading from CSV available through HTTP<a class="headerlink" href="#loading-from-csv-available-through-http" title="Permalink to this headline">¶</a></h3>
+<p>The same command as just above can also be run if the CSV file happens to be
+found on a remote HTTP location:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>pgloader --type csv                                                     \
+         --field &quot;usps,geoid,aland,awater,aland_sqmi,awater_sqmi,intptlat,intptlong&quot; \
+         --with &quot;skip header = 1&quot;                                       \
+         --with &quot;fields terminated by &#39;\t&#39;&quot;                             \
+         http://pgsql.tapoueh.org/temp/2013_Gaz_113CDs_national.txt     \
+         postgresql:///pgloader?districts_longlat
+</pre></div>
+</div>
+<p>Some more options have to be used in that case, as the file contains a
+one-line header (most commonly that’s column names, could be a copyright
+notice). Also, in that case, we specify all the fields right into a single
+<cite>–field</cite> option argument.</p>
+<p>Again, the PostgreSQL target connection string must contain the <em>tablename</em>
+option and you have to ensure that the target table exists and may fit the
+data. Here’s the SQL command used in that example in case you want to try it
+yourself:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">create</span> <span class="n">table</span> <span class="n">districts_longlat</span>
+<span class="p">(</span>
+         <span class="n">usps</span>        <span class="n">text</span><span class="p">,</span>
+         <span class="n">geoid</span>       <span class="n">text</span><span class="p">,</span>
+         <span class="n">aland</span>       <span class="n">bigint</span><span class="p">,</span>
+         <span class="n">awater</span>      <span class="n">bigint</span><span class="p">,</span>
+         <span class="n">aland_sqmi</span>  <span class="n">double</span> <span class="n">precision</span><span class="p">,</span>
+         <span class="n">awater_sqmi</span> <span class="n">double</span> <span class="n">precision</span><span class="p">,</span>
+         <span class="n">intptlat</span>    <span class="n">double</span> <span class="n">precision</span><span class="p">,</span>
+         <span class="n">intptlong</span>   <span class="n">double</span> <span class="n">precision</span>
+<span class="p">);</span>
+</pre></div>
+</div>
+<p>Also notice that the same command will work against an archived version of
+the same data, e.g.
+<a class="reference external" href="http://pgsql.tapoueh.org/temp/2013_Gaz_113CDs_national.txt.gz">http://pgsql.tapoueh.org/temp/2013_Gaz_113CDs_national.txt.gz</a>.</p>
+<p>Finally, it’s important to note that pgloader first fetches the content from
+the HTTP URL it to a local file, then expand the archive when it’s
+recognized to be one, and only then processes the locally expanded file.</p>
+<p>In some cases, either because pgloader has no direct support for your
+archive format or maybe because expanding the archive is not feasible in
+your environment, you might want to <em>stream</em> the content straight from its
+remote location into PostgreSQL. Here’s how to do that, using the old battle
+tested Unix Pipes trick:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>curl http://pgsql.tapoueh.org/temp/2013_Gaz_113CDs_national.txt.gz \
+| gunzip -c                                                        \
+| pgloader --type csv                                              \
+           --field &quot;usps,geoid,aland,awater,aland_sqmi,awater_sqmi,intptlat,intptlong&quot;
+           --with &quot;skip header = 1&quot;                                \
+           --with &quot;fields terminated by &#39;\t&#39;&quot;                      \
+           -                                                       \
+           postgresql:///pgloader?districts_longlat
+</pre></div>
+</div>
+<p>Now the OS will take care of the streaming and buffering between the network
+and the commands and pgloader will take care of streaming the data down to
+PostgreSQL.</p>
+</div>
+<div class="section" id="migrating-from-sqlite">
+<h3>Migrating from SQLite<a class="headerlink" href="#migrating-from-sqlite" title="Permalink to this headline">¶</a></h3>
+<p>The following command will open the SQLite database, discover its tables
+definitions including indexes and foreign keys, migrate those definitions
+while <em>casting</em> the data type specifications to their PostgreSQL equivalent
+and then migrate the data over:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">createdb</span> <span class="n">newdb</span>
+<span class="n">pgloader</span> <span class="o">./</span><span class="n">test</span><span class="o">/</span><span class="n">sqlite</span><span class="o">/</span><span class="n">sqlite</span><span class="o">.</span><span class="n">db</span> <span class="n">postgresql</span><span class="p">:</span><span class="o">///</span><span class="n">newdb</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="migrating-from-mysql">
+<h3>Migrating from MySQL<a class="headerlink" href="#migrating-from-mysql" title="Permalink to this headline">¶</a></h3>
+<p>Just create a database where to host the MySQL data and definitions and have
+pgloader do the migration for you in a single command line:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">createdb</span> <span class="n">pagila</span>
+<span class="n">pgloader</span> <span class="n">mysql</span><span class="p">:</span><span class="o">//</span><span class="n">user</span><span class="nd">@localhost</span><span class="o">/</span><span class="n">sakila</span> <span class="n">postgresql</span><span class="p">:</span><span class="o">///</span><span class="n">pagila</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="fetching-an-archived-dbf-file-from-a-http-remote-location">
+<h3>Fetching an archived DBF file from a HTTP remote location<a class="headerlink" href="#fetching-an-archived-dbf-file-from-a-http-remote-location" title="Permalink to this headline">¶</a></h3>
+<p>It’s possible for pgloader to download a file from HTTP, unarchive it, and
+only then open it to discover the schema then load the data:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">createdb</span> <span class="n">foo</span>
+<span class="n">pgloader</span> <span class="o">--</span><span class="nb">type</span> <span class="n">dbf</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">www</span><span class="o">.</span><span class="n">insee</span><span class="o">.</span><span class="n">fr</span><span class="o">/</span><span class="n">fr</span><span class="o">/</span><span class="n">methodes</span><span class="o">/</span><span class="n">nomenclatures</span><span class="o">/</span><span class="n">cog</span><span class="o">/</span><span class="n">telechargement</span><span class="o">/</span><span class="mi">2013</span><span class="o">/</span><span class="n">dbf</span><span class="o">/</span><span class="n">historiq2013</span><span class="o">.</span><span class="n">zip</span> <span class="n">postgresql</span><span class="p">:</span><span class="o">///</span><span class="n">foo</span>
+</pre></div>
+</div>
+<p>Here it’s not possible for pgloader to guess the kind of data source it’s
+being given, so it’s necessary to use the <cite>–type</cite> command line switch.</p>
+</div>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="index.html">Documentation overview</a><ul>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="_sources/pgloader-usage-examples.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/ref/archive.html

@@ -0,0 +1,191 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Loading From an Archive &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="../_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="index" title="Index" href="../genindex.html" />
+    <link rel="search" title="Search" href="../search.html" />
+    <link rel="next" title="Migrating a MySQL Database to PostgreSQL" href="mysql.html" />
+    <link rel="prev" title="Loading IXF Data" href="ixf.html" />
+   
+  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="loading-from-an-archive">
+<h1>Loading From an Archive<a class="headerlink" href="#loading-from-an-archive" title="Permalink to this headline">¶</a></h1>
+<p>This command instructs pgloader to load data from one or more files contained
+in an archive. Currently the only supported archive format is <em>ZIP</em>, and the
+archive might be downloaded from an <em>HTTP</em> URL.</p>
+<p>Here’s an example:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>LOAD ARCHIVE
+   FROM /Users/dim/Downloads/GeoLiteCity-latest.zip
+   INTO postgresql:///ip4r
+
+   BEFORE LOAD
+     DO $$ create extension if not exists ip4r; $$,
+        $$ create schema if not exists geolite; $$,
+
+     EXECUTE &#39;geolite.sql&#39;
+
+   LOAD CSV
+        FROM FILENAME MATCHING ~/GeoLiteCity-Location.csv/
+             WITH ENCODING iso-8859-1
+             (
+                locId,
+                country,
+                region     null if blanks,
+                city       null if blanks,
+                postalCode null if blanks,
+                latitude,
+                longitude,
+                metroCode  null if blanks,
+                areaCode   null if blanks
+             )
+        INTO postgresql:///ip4r?geolite.location
+             (
+                locid,country,region,city,postalCode,
+                location point using (format nil &quot;(~a,~a)&quot; longitude latitude),
+                metroCode,areaCode
+             )
+        WITH skip header = 2,
+             fields optionally enclosed by &#39;&quot;&#39;,
+             fields escaped by double-quote,
+             fields terminated by &#39;,&#39;
+
+  AND LOAD CSV
+        FROM FILENAME MATCHING ~/GeoLiteCity-Blocks.csv/
+             WITH ENCODING iso-8859-1
+             (
+                startIpNum, endIpNum, locId
+             )
+        INTO postgresql:///ip4r?geolite.blocks
+             (
+                iprange ip4r using (ip-range startIpNum endIpNum),
+                locId
+             )
+        WITH skip header = 2,
+             fields optionally enclosed by &#39;&quot;&#39;,
+             fields escaped by double-quote,
+             fields terminated by &#39;,&#39;
+
+   FINALLY DO
+     $$ create index blocks_ip4r_idx on geolite.blocks using gist(iprange); $$;
+</pre></div>
+</div>
+<p>The <cite>archive</cite> command accepts the following clauses and options.</p>
+<div class="section" id="archive-source-specification-from">
+<h2>Archive Source Specification: FROM<a class="headerlink" href="#archive-source-specification-from" title="Permalink to this headline">¶</a></h2>
+<p>Filename or HTTP URI where to load the data from. When given an HTTP URL the
+linked file will get downloaded locally before processing.</p>
+<p>If the file is a <cite>zip</cite> file, the command line utility <cite>unzip</cite> is used to
+expand the archive into files in <cite>$TMPDIR</cite>, or <cite>/tmp</cite> if <cite>$TMPDIR</cite> is unset
+or set to a non-existing directory.</p>
+<p>Then the following commands are used from the top level directory where the
+archive has been expanded.</p>
+</div>
+<div class="section" id="archive-sub-commands">
+<h2>Archive Sub Commands<a class="headerlink" href="#archive-sub-commands" title="Permalink to this headline">¶</a></h2>
+<blockquote>
+<div><ul>
+<li><p class="first">command [ <em>AND</em> command … ]</p>
+<p>A series of commands against the contents of the archive, at the moment
+only <cite>CSV</cite>,`’FIXED` and <cite>DBF</cite> commands are supported.</p>
+<p>Note that commands are supporting the clause <em>FROM FILENAME MATCHING</em>
+which allows the pgloader command not to depend on the exact names of
+the archive directories.</p>
+<p>The same clause can also be applied to several files with using the
+spelling <em>FROM ALL FILENAMES MATCHING</em> and a regular expression.</p>
+<p>The whole <em>matching</em> clause must follow the following rule:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">FROM</span> <span class="p">[</span> <span class="n">ALL</span> <span class="n">FILENAMES</span> <span class="o">|</span> <span class="p">[</span> <span class="n">FIRST</span> <span class="p">]</span> <span class="n">FILENAME</span> <span class="p">]</span> <span class="n">MATCHING</span>
+</pre></div>
+</div>
+</li>
+</ul>
+</div></blockquote>
+</div>
+<div class="section" id="archive-final-sql-commands">
+<h2>Archive Final SQL Commands<a class="headerlink" href="#archive-final-sql-commands" title="Permalink to this headline">¶</a></h2>
+<blockquote>
+<div><ul>
+<li><p class="first"><em>FINALLY DO</em></p>
+<p>SQL Queries to run once the data is loaded, such as <cite>CREATE INDEX</cite>.</p>
+</li>
+</ul>
+</div></blockquote>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="../index.html">Documentation overview</a><ul>
+      <li>Previous: <a href="ixf.html" title="previous chapter">Loading IXF Data</a></li>
+      <li>Next: <a href="mysql.html" title="next chapter">Migrating a MySQL Database to PostgreSQL</a></li>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="../_sources/ref/archive.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/ref/copy.html

@@ -0,0 +1,196 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Loading COPY Formatted Files &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="../_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="index" title="Index" href="../genindex.html" />
+    <link rel="search" title="Search" href="../search.html" />
+    <link rel="next" title="Loading DBF data" href="dbf.html" />
+    <link rel="prev" title="Loading Fixed Cols File Formats" href="fixed.html" />
+   
+  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="loading-copy-formatted-files">
+<h1>Loading COPY Formatted Files<a class="headerlink" href="#loading-copy-formatted-files" title="Permalink to this headline">¶</a></h1>
+<p>This commands instructs pgloader to load from a file containing COPY TEXT
+data as described in the PostgreSQL documentation. Here’s an example:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>LOAD COPY
+     FROM copy://./data/track.copy
+          (
+            trackid, track, album, media, genre, composer,
+            milliseconds, bytes, unitprice
+          )
+     INTO postgresql:///pgloader
+   TARGET TABLE track_full
+
+     WITH truncate
+
+      SET work_mem to &#39;14MB&#39;,
+          standard_conforming_strings to &#39;on&#39;
+
+BEFORE LOAD DO
+     $$ drop table if exists track_full; $$,
+     $$ create table track_full (
+          trackid      bigserial,
+          track        text,
+          album        text,
+          media        text,
+          genre        text,
+          composer     text,
+          milliseconds bigint,
+          bytes        bigint,
+          unitprice    numeric
+        );
+     $$;
+</pre></div>
+</div>
+<p>The <cite>COPY</cite> format command accepts the following clauses and options.</p>
+<div class="section" id="copy-formatted-files-source-specification-from">
+<h2>COPY Formatted Files Source Specification: FROM<a class="headerlink" href="#copy-formatted-files-source-specification-from" title="Permalink to this headline">¶</a></h2>
+<p>Filename where to load the data from. This support local files, HTTP URLs
+and zip files containing a single dbf file of the same name. Fetch such a
+zip file from an HTTP address is of course supported.</p>
+<blockquote>
+<div><ul>
+<li><p class="first"><em>inline</em></p>
+<p>The data is found after the end of the parsed commands. Any number of
+empty lines between the end of the commands and the beginning of the
+data is accepted.</p>
+</li>
+<li><p class="first"><em>stdin</em></p>
+<p>Reads the data from the standard input stream.</p>
+</li>
+<li><p class="first"><em>FILENAMES MATCHING</em></p>
+<p>The whole <em>matching</em> clause must follow the following rule:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="p">[</span> <span class="n">ALL</span> <span class="n">FILENAMES</span> <span class="o">|</span> <span class="p">[</span> <span class="n">FIRST</span> <span class="p">]</span> <span class="n">FILENAME</span> <span class="p">]</span>
+<span class="n">MATCHING</span> <span class="n">regexp</span>
+<span class="p">[</span> <span class="n">IN</span> <span class="n">DIRECTORY</span> <span class="s1">&#39;...&#39;</span> <span class="p">]</span>
+</pre></div>
+</div>
+<p>The <em>matching</em> clause applies given <em>regular expression</em> (see above for
+exact syntax, several options can be used here) to filenames. It’s then
+possible to load data from only the first match of all of them.</p>
+<p>The optional <em>IN DIRECTORY</em> clause allows specifying which directory to
+walk for finding the data files, and can be either relative to where the
+command file is read from, or absolute. The given directory must exists.</p>
+</li>
+</ul>
+</div></blockquote>
+</div>
+<div class="section" id="copy-formatted-file-options-with">
+<h2>COPY Formatted File Options: WITH<a class="headerlink" href="#copy-formatted-file-options-with" title="Permalink to this headline">¶</a></h2>
+<p>When loading from a <cite>COPY</cite> file, the following options are supported:</p>
+<blockquote>
+<div><ul>
+<li><p class="first"><em>delimiter</em></p>
+<p>Takes a single character as argument, which must be found inside single
+quotes, and might be given as the printable character itself, the
+special value t to denote a tabulation character, or <cite>0x</cite> then an
+hexadecimal value read as the ASCII code for the character.</p>
+<p>This character is used as the <em>delimiter</em> when reading the data, in a
+similar way to the PostgreSQL <cite>COPY</cite> option.</p>
+</li>
+<li><p class="first"><em>null</em></p>
+<p>Takes a quoted string as an argument (quotes can be either double quotes
+or single quotes) and uses that string as the <cite>NULL</cite> representation in
+the data.</p>
+<p>This is similar to the <em>null</em> <cite>COPY</cite> option in PostgreSQL.</p>
+</li>
+<li><p class="first"><em>truncate</em></p>
+<p>When this option is listed, pgloader issues a <cite>TRUNCATE</cite> command against
+the PostgreSQL target table before reading the data file.</p>
+</li>
+<li><p class="first"><em>disable triggers</em></p>
+<p>When this option is listed, pgloader issues an <cite>ALTER TABLE … DISABLE
+TRIGGER ALL</cite> command against the PostgreSQL target table before copying
+the data, then the command <cite>ALTER TABLE … ENABLE TRIGGER ALL</cite> once the
+<cite>COPY</cite> is done.</p>
+<p>This option allows loading data into a pre-existing table ignoring the
+<em>foreign key constraints</em> and user defined triggers and may result in
+invalid <em>foreign key constraints</em> once the data is loaded. Use with
+care.</p>
+</li>
+<li><p class="first"><em>skip header</em></p>
+<p>Takes a numeric value as argument. Instruct pgloader to skip that many
+lines at the beginning of the input file.</p>
+</li>
+</ul>
+</div></blockquote>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="../index.html">Documentation overview</a><ul>
+      <li>Previous: <a href="fixed.html" title="previous chapter">Loading Fixed Cols File Formats</a></li>
+      <li>Next: <a href="dbf.html" title="next chapter">Loading DBF data</a></li>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="../_sources/ref/copy.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/ref/csv.html

@@ -0,0 +1,310 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Loading CSV data &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="../_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="index" title="Index" href="../genindex.html" />
+    <link rel="search" title="Search" href="../search.html" />
+    <link rel="next" title="Loading Fixed Cols File Formats" href="fixed.html" />
+    <link rel="prev" title="PgLoader Reference Manual" href="../pgloader.html" />
+   
+  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="loading-csv-data">
+<h1>Loading CSV data<a class="headerlink" href="#loading-csv-data" title="Permalink to this headline">¶</a></h1>
+<p>This command instructs pgloader to load data from a <cite>CSV</cite> file. Here’s an
+example:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">LOAD</span> <span class="n">CSV</span>
+   <span class="n">FROM</span> <span class="s1">&#39;GeoLiteCity-Blocks.csv&#39;</span> <span class="n">WITH</span> <span class="n">ENCODING</span> <span class="n">iso</span><span class="o">-</span><span class="mi">646</span><span class="o">-</span><span class="n">us</span>
+        <span class="n">HAVING</span> <span class="n">FIELDS</span>
+        <span class="p">(</span>
+           <span class="n">startIpNum</span><span class="p">,</span> <span class="n">endIpNum</span><span class="p">,</span> <span class="n">locId</span>
+        <span class="p">)</span>
+   <span class="n">INTO</span> <span class="n">postgresql</span><span class="p">:</span><span class="o">//</span><span class="n">user</span><span class="nd">@localhost</span><span class="p">:</span><span class="mi">54393</span><span class="o">/</span><span class="n">dbname</span>
+        <span class="n">TARGET</span> <span class="n">TABLE</span> <span class="n">geolite</span><span class="o">.</span><span class="n">blocks</span>
+        <span class="n">TARGET</span> <span class="n">COLUMNS</span>
+        <span class="p">(</span>
+           <span class="n">iprange</span> <span class="n">ip4r</span> <span class="n">using</span> <span class="p">(</span><span class="n">ip</span><span class="o">-</span><span class="nb">range</span> <span class="n">startIpNum</span> <span class="n">endIpNum</span><span class="p">),</span>
+           <span class="n">locId</span>
+        <span class="p">)</span>
+   <span class="n">WITH</span> <span class="n">truncate</span><span class="p">,</span>
+        <span class="n">skip</span> <span class="n">header</span> <span class="o">=</span> <span class="mi">2</span><span class="p">,</span>
+        <span class="n">fields</span> <span class="n">optionally</span> <span class="n">enclosed</span> <span class="n">by</span> <span class="s1">&#39;&quot;&#39;</span><span class="p">,</span>
+        <span class="n">fields</span> <span class="n">escaped</span> <span class="n">by</span> <span class="n">backslash</span><span class="o">-</span><span class="n">quote</span><span class="p">,</span>
+        <span class="n">fields</span> <span class="n">terminated</span> <span class="n">by</span> <span class="s1">&#39;</span><span class="se">\t</span><span class="s1">&#39;</span>
+
+    <span class="n">SET</span> <span class="n">work_mem</span> <span class="n">to</span> <span class="s1">&#39;32 MB&#39;</span><span class="p">,</span> <span class="n">maintenance_work_mem</span> <span class="n">to</span> <span class="s1">&#39;64 MB&#39;</span><span class="p">;</span>
+</pre></div>
+</div>
+<p>The <cite>csv</cite> format command accepts the following clauses and options.</p>
+<div class="section" id="csv-source-specification-from">
+<h2>CSV Source Specification: FROM<a class="headerlink" href="#csv-source-specification-from" title="Permalink to this headline">¶</a></h2>
+<p>Filename where to load the data from. Accepts an <em>ENCODING</em> option. Use the
+<cite>–list-encodings</cite> option to know which encoding names are supported.</p>
+<p>The filename may be enclosed by single quotes, and could be one of the
+following special values:</p>
+<blockquote>
+<div><ul>
+<li><p class="first"><em>inline</em></p>
+<p>The data is found after the end of the parsed commands. Any number
+of empty lines between the end of the commands and the beginning of
+the data is accepted.</p>
+</li>
+<li><p class="first"><em>stdin</em></p>
+<p>Reads the data from the standard input stream.</p>
+</li>
+<li><p class="first"><em>FILENAMES MATCHING</em></p>
+<p>The whole <em>matching</em> clause must follow the following rule:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="p">[</span> <span class="n">ALL</span> <span class="n">FILENAMES</span> <span class="o">|</span> <span class="p">[</span> <span class="n">FIRST</span> <span class="p">]</span> <span class="n">FILENAME</span> <span class="p">]</span>
+<span class="n">MATCHING</span> <span class="n">regexp</span>
+<span class="p">[</span> <span class="n">IN</span> <span class="n">DIRECTORY</span> <span class="s1">&#39;...&#39;</span> <span class="p">]</span>
+</pre></div>
+</div>
+<p>The <em>matching</em> clause applies given <em>regular expression</em> (see above
+for exact syntax, several options can be used here) to filenames.
+It’s then possible to load data from only the first match of all of
+them.</p>
+<p>The optional <em>IN DIRECTORY</em> clause allows specifying which directory
+to walk for finding the data files, and can be either relative to
+where the command file is read from, or absolute. The given
+directory must exists.</p>
+</li>
+</ul>
+</div></blockquote>
+</div>
+<div class="section" id="fields-specifications">
+<h2>Fields Specifications<a class="headerlink" href="#fields-specifications" title="Permalink to this headline">¶</a></h2>
+<p>The <em>FROM</em> option also supports an optional comma separated list of <em>field</em>
+names describing what is expected in the <cite>CSV</cite> data file, optionally
+introduced by the clause <cite>HAVING FIELDS</cite>.</p>
+<p>Each field name can be either only one name or a name following with
+specific reader options for that field, enclosed in square brackets and
+comma-separated. Supported per-field reader options are:</p>
+<blockquote>
+<div><ul>
+<li><p class="first"><em>terminated by</em></p>
+<blockquote>
+<div><p>See the description of <em>field terminated by</em> below.</p>
+<p>The processing of this option is not currently implemented.</p>
+</div></blockquote>
+</li>
+<li><p class="first"><em>date format</em></p>
+<p>When the field is expected of the date type, then this option allows
+to specify the date format used in the file.</p>
+<p>Date format string are template strings modeled against the
+PostgreSQL <cite>to_char</cite> template strings support, limited to the
+following patterns:</p>
+<blockquote>
+<div><ul class="simple">
+<li>YYYY, YYY, YY for the year part</li>
+<li>MM for the numeric month part</li>
+<li>DD for the numeric day part</li>
+<li>HH, HH12, HH24 for the hour part</li>
+<li>am, AM, a.m., A.M.</li>
+<li>pm, PM, p.m., P.M.</li>
+<li>MI for the minutes part</li>
+<li>SS for the seconds part</li>
+<li>MS for the milliseconds part (4 digits)</li>
+<li>US for the microseconds part (6 digits)</li>
+<li>unparsed punctuation signs: - . * # &#64; T / and space</li>
+</ul>
+</div></blockquote>
+<p>Here’s an example of a <em>date format</em> specification:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">column</span><span class="o">-</span><span class="n">name</span> <span class="p">[</span><span class="n">date</span> <span class="nb">format</span> <span class="s1">&#39;YYYY-MM-DD HH24-MI-SS.US&#39;</span><span class="p">]</span>
+</pre></div>
+</div>
+</li>
+<li><p class="first"><em>null if</em></p>
+<blockquote>
+<div><p>This option takes an argument which is either the keyword <em>blanks</em>
+or a double-quoted string.</p>
+<p>When <em>blanks</em> is used and the field value that is read contains
+only space characters, then it’s automatically converted to an SQL
+<cite>NULL</cite> value.</p>
+<p>When a double-quoted string is used and that string is read as the
+field value, then the field value is automatically converted to an
+SQL <cite>NULL</cite> value.</p>
+</div></blockquote>
+</li>
+<li><p class="first"><em>trim both whitespace</em>, <em>trim left whitespace</em>, <em>trim right whitespace</em></p>
+<p>This option allows to trim whitespaces in the read data, either from
+both sides of the data, or only the whitespace characters found on
+the left of the streaing, or only those on the right of the string.</p>
+</li>
+</ul>
+</div></blockquote>
+</div>
+<div class="section" id="csv-loading-options-with">
+<h2>CSV Loading Options: WITH<a class="headerlink" href="#csv-loading-options-with" title="Permalink to this headline">¶</a></h2>
+<p>When loading from a <cite>CSV</cite> file, the following options are supported:</p>
+<blockquote>
+<div><ul>
+<li><p class="first"><em>truncate</em></p>
+<blockquote>
+<div><p>When this option is listed, pgloader issues a <cite>TRUNCATE</cite> command
+against the PostgreSQL target table before reading the data file.</p>
+</div></blockquote>
+</li>
+<li><p class="first"><em>drop indexes</em></p>
+<p>When this option is listed, pgloader issues <cite>DROP INDEX</cite> commands
+against all the indexes defined on the target table before copying
+the data, then <cite>CREATE INDEX</cite> commands once the <cite>COPY</cite> is done.</p>
+<p>In order to get the best performance possible, all the indexes are
+created in parallel and when done the primary keys are built again
+from the unique indexes just created. This two step process allows
+creating the primary key index in parallel with the other indexes,
+as only the <cite>ALTER TABLE</cite> command needs an <em>access exclusive lock</em>
+on the target table.</p>
+</li>
+<li><p class="first"><em>disable triggers</em></p>
+<p>When this option is listed, pgloader issues an <cite>ALTER TABLE …
+DISABLE TRIGGER ALL</cite> command against the PostgreSQL target table
+before copying the data, then the command <cite>ALTER TABLE … ENABLE
+TRIGGER ALL</cite> once the <cite>COPY</cite> is done.</p>
+<p>This option allows loading data into a pre-existing table ignoring
+the <em>foreign key constraints</em> and user defined triggers and may
+result in invalid <em>foreign key constraints</em> once the data is loaded.
+Use with care.</p>
+</li>
+<li><p class="first"><em>skip header</em></p>
+<p>Takes a numeric value as argument. Instruct pgloader to skip that
+many lines at the beginning of the input file.</p>
+</li>
+<li><p class="first"><em>csv header</em></p>
+<p>Use the first line read after <em>skip header</em> as the list of csv field
+names to be found in the CSV file, using the same CSV parameters as
+for the CSV data.</p>
+</li>
+<li><p class="first"><em>trim unquoted blanks</em></p>
+<p>When reading unquoted values in the <cite>CSV</cite> file, remove the blanks
+found in between the separator and the value. That behaviour is the
+default.</p>
+</li>
+<li><p class="first"><em>keep unquoted blanks</em></p>
+<p>When reading unquoted values in the <cite>CSV</cite> file, keep blanks found in
+between the separator and the value.</p>
+</li>
+<li><p class="first"><em>fields optionally enclosed by</em></p>
+<p>Takes a single character as argument, which must be found inside
+single quotes, and might be given as the printable character itself,
+the special value t to denote a tabulation character, or <cite>0x</cite> then
+an hexadecimal value read as the ASCII code for the character.</p>
+<p>This character is used as the quoting character in the <cite>CSV</cite> file,
+and defaults to double-quote.</p>
+</li>
+<li><p class="first"><em>fields not enclosed</em></p>
+<p>By default, pgloader will use the double-quote character as the
+enclosing character. If you have a CSV file where fields are not
+enclosed and are using double-quote as an expected ordinary
+character, then use the option <em>fields not enclosed</em> for the CSV
+parser to accept those values.</p>
+</li>
+<li><p class="first"><em>fields escaped by</em></p>
+<p>Takes either the special value <em>backslash-quote</em> or <em>double-quote</em>,
+or any value supported by the <em>fields terminated by</em> option (see
+below). This value is used to recognize escaped field separators
+when they are to be found within the data fields themselves.
+Defaults to <em>double-quote</em>.</p>
+</li>
+<li><p class="first"><em>csv escape mode</em></p>
+<p>Takes either the special value <em>quote</em> (the default) or <em>following</em>
+and allows the CSV parser to parse either only escaped field
+separator or any character (including CSV data) when using the
+<em>following</em> value.</p>
+</li>
+<li><p class="first"><em>fields terminated by</em></p>
+<p>Takes a single character as argument, which must be found inside
+single quotes, and might be given as the printable character itself,
+the special value t to denote a tabulation character, or <cite>0x</cite> then
+an hexadecimal value read as the ASCII code for the character.</p>
+<p>This character is used as the <em>field separator</em> when reading the
+<cite>CSV</cite> data.</p>
+</li>
+<li><p class="first"><em>lines terminated by</em></p>
+<p>Takes a single character as argument, which must be found inside
+single quotes, and might be given as the printable character itself,
+the special value t to denote a tabulation character, or <cite>0x</cite> then
+an hexadecimal value read as the ASCII code for the character.</p>
+<p>This character is used to recognize <em>end-of-line</em> condition when
+reading the <cite>CSV</cite> data.</p>
+</li>
+</ul>
+</div></blockquote>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="../index.html">Documentation overview</a><ul>
+      <li>Previous: <a href="../pgloader.html" title="previous chapter">PgLoader Reference Manual</a></li>
+      <li>Next: <a href="fixed.html" title="next chapter">Loading Fixed Cols File Formats</a></li>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="../_sources/ref/csv.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/ref/dbf.html

@@ -0,0 +1,138 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Loading DBF data &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="../_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="index" title="Index" href="../genindex.html" />
+    <link rel="search" title="Search" href="../search.html" />
+    <link rel="next" title="Loading IXF Data" href="ixf.html" />
+    <link rel="prev" title="Loading COPY Formatted Files" href="copy.html" />
+   
+  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="loading-dbf-data">
+<h1>Loading DBF data<a class="headerlink" href="#loading-dbf-data" title="Permalink to this headline">¶</a></h1>
+<p>This command instructs pgloader to load data from a <cite>DBF</cite> file. Here’s an
+example:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">LOAD</span> <span class="n">DBF</span>
+        <span class="n">FROM</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">www</span><span class="o">.</span><span class="n">insee</span><span class="o">.</span><span class="n">fr</span><span class="o">/</span><span class="n">fr</span><span class="o">/</span><span class="n">methodes</span><span class="o">/</span><span class="n">nomenclatures</span><span class="o">/</span><span class="n">cog</span><span class="o">/</span><span class="n">telechargement</span><span class="o">/</span><span class="mi">2013</span><span class="o">/</span><span class="n">dbf</span><span class="o">/</span><span class="n">reg2013</span><span class="o">.</span><span class="n">dbf</span>
+    <span class="n">INTO</span> <span class="n">postgresql</span><span class="p">:</span><span class="o">//</span><span class="n">user</span><span class="nd">@localhost</span><span class="o">/</span><span class="n">dbname</span>
+    <span class="n">WITH</span> <span class="n">truncate</span><span class="p">,</span> <span class="n">create</span> <span class="n">table</span><span class="p">;</span>
+</pre></div>
+</div>
+<p>The <cite>dbf</cite> format command accepts the following clauses and options.</p>
+<div class="section" id="dbf-source-specification-from">
+<h2>DBF Source Specification: FROM<a class="headerlink" href="#dbf-source-specification-from" title="Permalink to this headline">¶</a></h2>
+<p>Filename where to load the data from. This support local files, HTTP URLs
+and zip files containing a single dbf file of the same name. Fetch such a
+zip file from an HTTP address is of course supported.</p>
+</div>
+<div class="section" id="dbf-loading-options-with">
+<h2>DBF Loading Options: WITH<a class="headerlink" href="#dbf-loading-options-with" title="Permalink to this headline">¶</a></h2>
+<p>When loading from a <cite>DBF</cite> file, the following options are supported:</p>
+<blockquote>
+<div><ul>
+<li><p class="first"><em>truncate</em></p>
+<p>When this option is listed, pgloader issues a <cite>TRUNCATE</cite> command against
+the PostgreSQL target table before reading the data file.</p>
+</li>
+<li><p class="first"><em>disable triggers</em></p>
+<p>When this option is listed, pgloader issues an <cite>ALTER TABLE … DISABLE
+TRIGGER ALL</cite> command against the PostgreSQL target table before copying
+the data, then the command <cite>ALTER TABLE … ENABLE TRIGGER ALL</cite> once the
+<cite>COPY</cite> is done.</p>
+<p>This option allows loading data into a pre-existing table ignoring the
+<em>foreign key constraints</em> and user defined triggers and may result in
+invalid <em>foreign key constraints</em> once the data is loaded. Use with
+care.</p>
+</li>
+<li><p class="first"><em>create table</em></p>
+<p>When this option is listed, pgloader creates the table using the meta
+data found in the <cite>DBF</cite> file, which must contain a list of fields with
+their data type. A standard data type conversion from DBF to PostgreSQL
+is done.</p>
+</li>
+<li><p class="first"><em>table name</em></p>
+<p>This options expects as its value the possibly qualified name of the
+table to create.</p>
+</li>
+</ul>
+</div></blockquote>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="../index.html">Documentation overview</a><ul>
+      <li>Previous: <a href="copy.html" title="previous chapter">Loading COPY Formatted Files</a></li>
+      <li>Next: <a href="ixf.html" title="next chapter">Loading IXF Data</a></li>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="../_sources/ref/dbf.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/ref/fixed.html

@@ -0,0 +1,270 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Loading Fixed Cols File Formats &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="../_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="index" title="Index" href="../genindex.html" />
+    <link rel="search" title="Search" href="../search.html" />
+    <link rel="next" title="Loading COPY Formatted Files" href="copy.html" />
+    <link rel="prev" title="Loading CSV data" href="csv.html" />
+   
+  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="loading-fixed-cols-file-formats">
+<h1>Loading Fixed Cols File Formats<a class="headerlink" href="#loading-fixed-cols-file-formats" title="Permalink to this headline">¶</a></h1>
+<p>This command instructs pgloader to load data from a text file containing
+columns arranged in a <em>fixed size</em> manner. Here’s an example:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>LOAD FIXED
+     FROM inline
+          (
+           a from  0 for 10,
+           b from 10 for  8,
+           c from 18 for  8,
+           d from 26 for 17 [null if blanks, trim right whitespace]
+          )
+     INTO postgresql:///pgloader
+   TARGET TABLE fixed
+          (
+             a, b,
+             c time using (time-with-no-separator c),
+             d
+          )
+
+     WITH truncate
+
+      SET work_mem to &#39;14MB&#39;,
+          standard_conforming_strings to &#39;on&#39;
+
+BEFORE LOAD DO
+     $$ drop table if exists fixed; $$,
+     $$ create table fixed (
+         a integer,
+         b date,
+         c time,
+         d text
+        );
+     $$;
+
+ 01234567892008052011431250firstline
+    01234562008052115182300left blank-padded
+ 12345678902008052208231560another line
+  2345609872014092914371500
+  2345678902014092914371520
+</pre></div>
+</div>
+<p>The <cite>fixed</cite> format command accepts the following clauses and options.</p>
+<div class="section" id="fixed-file-format-source-specification-from">
+<h2>Fixed File Format Source Specification: FROM<a class="headerlink" href="#fixed-file-format-source-specification-from" title="Permalink to this headline">¶</a></h2>
+<p>Filename where to load the data from. Accepts an <em>ENCODING</em> option. Use the
+<cite>–list-encodings</cite> option to know which encoding names are supported.</p>
+<p>The filename may be enclosed by single quotes, and could be one of the
+following special values:</p>
+<blockquote>
+<div><ul>
+<li><p class="first"><em>inline</em></p>
+<blockquote>
+<div><p>The data is found after the end of the parsed commands. Any number
+of empty lines between the end of the commands and the beginning of
+the data is accepted.</p>
+</div></blockquote>
+</li>
+<li><p class="first"><em>stdin</em></p>
+<blockquote>
+<div><p>Reads the data from the standard input stream.</p>
+</div></blockquote>
+</li>
+<li><p class="first"><em>FILENAMES MATCHING</em></p>
+<p>The whole <em>matching</em> clause must follow the following rule:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="p">[</span> <span class="n">ALL</span> <span class="n">FILENAMES</span> <span class="o">|</span> <span class="p">[</span> <span class="n">FIRST</span> <span class="p">]</span> <span class="n">FILENAME</span> <span class="p">]</span>
+<span class="n">MATCHING</span> <span class="n">regexp</span>
+<span class="p">[</span> <span class="n">IN</span> <span class="n">DIRECTORY</span> <span class="s1">&#39;...&#39;</span> <span class="p">]</span>
+</pre></div>
+</div>
+<p>The <em>matching</em> clause applies given <em>regular expression</em> (see above
+for exact syntax, several options can be used here) to filenames.
+It’s then possible to load data from only the first match of all of
+them.</p>
+<p>The optional <em>IN DIRECTORY</em> clause allows specifying which directory
+to walk for finding the data files, and can be either relative to
+where the command file is read from, or absolute. The given
+directory must exists.</p>
+</li>
+</ul>
+</div></blockquote>
+</div>
+<div class="section" id="fields-specifications">
+<h2>Fields Specifications<a class="headerlink" href="#fields-specifications" title="Permalink to this headline">¶</a></h2>
+<p>The <em>FROM</em> option also supports an optional comma separated list of <em>field</em>
+names describing what is expected in the <cite>FIXED</cite> data file.</p>
+<p>Each field name is composed of the field name followed with specific reader
+options for that field. Supported per-field reader options are the
+following, where only <em>start</em> and <em>length</em> are required.</p>
+<blockquote>
+<div><ul>
+<li><p class="first"><em>start</em></p>
+<p>Position in the line where to start reading that field’s value. Can
+be entered with decimal digits or <cite>0x</cite> then hexadecimal digits.</p>
+</li>
+<li><p class="first"><em>length</em></p>
+<p>How many bytes to read from the <em>start</em> position to read that
+field’s value. Same format as <em>start</em>.</p>
+</li>
+</ul>
+</div></blockquote>
+<p>Those optional parameters must be enclosed in square brackets and
+comma-separated:</p>
+<blockquote>
+<div><ul>
+<li><p class="first"><em>terminated by</em></p>
+<blockquote>
+<div><p>See the description of <em>field terminated by</em> below.</p>
+<p>The processing of this option is not currently implemented.</p>
+</div></blockquote>
+</li>
+<li><p class="first"><em>date format</em></p>
+<p>When the field is expected of the date type, then this option allows
+to specify the date format used in the file.</p>
+<p>Date format string are template strings modeled against the
+PostgreSQL <cite>to_char</cite> template strings support, limited to the
+following patterns:</p>
+<blockquote>
+<div><ul class="simple">
+<li>YYYY, YYY, YY for the year part</li>
+<li>MM for the numeric month part</li>
+<li>DD for the numeric day part</li>
+<li>HH, HH12, HH24 for the hour part</li>
+<li>am, AM, a.m., A.M.</li>
+<li>pm, PM, p.m., P.M.</li>
+<li>MI for the minutes part</li>
+<li>SS for the seconds part</li>
+<li>MS for the milliseconds part (4 digits)</li>
+<li>US for the microseconds part (6 digits)</li>
+<li>unparsed punctuation signs: - . * # &#64; T / and space</li>
+</ul>
+</div></blockquote>
+<p>Here’s an example of a <em>date format</em> specification:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">column</span><span class="o">-</span><span class="n">name</span> <span class="p">[</span><span class="n">date</span> <span class="nb">format</span> <span class="s1">&#39;YYYY-MM-DD HH24-MI-SS.US&#39;</span><span class="p">]</span>
+</pre></div>
+</div>
+</li>
+<li><p class="first"><em>null if</em></p>
+<p>This option takes an argument which is either the keyword <em>blanks</em>
+or a double-quoted string.</p>
+<p>When <em>blanks</em> is used and the field value that is read contains only
+space characters, then it’s automatically converted to an SQL <cite>NULL</cite>
+value.</p>
+<p>When a double-quoted string is used and that string is read as the
+field value, then the field value is automatically converted to an
+SQL <cite>NULL</cite> value.</p>
+</li>
+<li><p class="first"><em>trim both whitespace</em>, <em>trim left whitespace</em>, <em>trim right whitespace</em></p>
+<p>This option allows to trim whitespaces in the read data, either from
+both sides of the data, or only the whitespace characters found on
+the left of the streaing, or only those on the right of the string.</p>
+</li>
+</ul>
+</div></blockquote>
+</div>
+<div class="section" id="fixed-file-format-loading-options-with">
+<h2>Fixed File Format Loading Options: WITH<a class="headerlink" href="#fixed-file-format-loading-options-with" title="Permalink to this headline">¶</a></h2>
+<p>When loading from a <cite>FIXED</cite> file, the following options are supported:</p>
+<blockquote>
+<div><ul>
+<li><p class="first"><em>truncate</em></p>
+<p>When this option is listed, pgloader issues a <cite>TRUNCATE</cite> command
+against the PostgreSQL target table before reading the data file.</p>
+</li>
+<li><p class="first"><em>disable triggers</em></p>
+<p>When this option is listed, pgloader issues an <cite>ALTER TABLE …
+DISABLE TRIGGER ALL</cite> command against the PostgreSQL target table
+before copying the data, then the command <cite>ALTER TABLE … ENABLE
+TRIGGER ALL</cite> once the <cite>COPY</cite> is done.</p>
+<p>This option allows loading data into a pre-existing table ignoring
+the <em>foreign key constraints</em> and user defined triggers and may
+result in invalid <em>foreign key constraints</em> once the data is loaded.
+Use with care.</p>
+</li>
+<li><p class="first"><em>skip header</em></p>
+<p>Takes a numeric value as argument. Instruct pgloader to skip that
+many lines at the beginning of the input file.</p>
+</li>
+</ul>
+</div></blockquote>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="../index.html">Documentation overview</a><ul>
+      <li>Previous: <a href="csv.html" title="previous chapter">Loading CSV data</a></li>
+      <li>Next: <a href="copy.html" title="next chapter">Loading COPY Formatted Files</a></li>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="../_sources/ref/fixed.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/ref/ixf.html

@@ -0,0 +1,149 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Loading IXF Data &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="../_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="index" title="Index" href="../genindex.html" />
+    <link rel="search" title="Search" href="../search.html" />
+    <link rel="next" title="Loading From an Archive" href="archive.html" />
+    <link rel="prev" title="Loading DBF data" href="dbf.html" />
+   
+  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="loading-ixf-data">
+<h1>Loading IXF Data<a class="headerlink" href="#loading-ixf-data" title="Permalink to this headline">¶</a></h1>
+<p>This command instructs pgloader to load data from an IBM <cite>IXF</cite> file. Here’s
+an example:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>LOAD IXF
+    FROM data/nsitra.test1.ixf
+    INTO postgresql:///pgloader
+  TARGET TABLE nsitra.test1
+    WITH truncate, create table, timezone UTC
+
+  BEFORE LOAD DO
+   $$ create schema if not exists nsitra; $$,
+   $$ drop table if exists nsitra.test1; $$;
+</pre></div>
+</div>
+<p>The <cite>ixf</cite> format command accepts the following clauses and options.</p>
+<div class="section" id="ixf-source-specification-from">
+<h2>IXF Source Specification: FROM<a class="headerlink" href="#ixf-source-specification-from" title="Permalink to this headline">¶</a></h2>
+<p>Filename where to load the data from. This support local files, HTTP URLs
+and zip files containing a single ixf file of the same name. Fetch such a
+zip file from an HTTP address is of course supported.</p>
+</div>
+<div class="section" id="ixf-loading-options-with">
+<h2>IXF Loading Options: WITH<a class="headerlink" href="#ixf-loading-options-with" title="Permalink to this headline">¶</a></h2>
+<p>When loading from a <cite>IXF</cite> file, the following options are supported:</p>
+<blockquote>
+<div><ul>
+<li><p class="first"><em>truncate</em></p>
+<p>When this option is listed, pgloader issues a <cite>TRUNCATE</cite> command against
+the PostgreSQL target table before reading the data file.</p>
+</li>
+<li><p class="first"><em>disable triggers</em></p>
+<p>When this option is listed, pgloader issues an <cite>ALTER TABLE … DISABLE
+TRIGGER ALL</cite> command against the PostgreSQL target table before copying
+the data, then the command <cite>ALTER TABLE … ENABLE TRIGGER ALL</cite> once the
+<cite>COPY</cite> is done.</p>
+<p>This option allows loading data into a pre-existing table ignoring the
+<em>foreign key constraints</em> and user defined triggers and may result in
+invalid <em>foreign key constraints</em> once the data is loaded. Use with
+care.</p>
+</li>
+<li><p class="first"><em>create table</em></p>
+<p>When this option is listed, pgloader creates the table using the meta
+data found in the <cite>DBF</cite> file, which must contain a list of fields with
+their data type. A standard data type conversion from DBF to PostgreSQL
+is done.</p>
+</li>
+<li><p class="first"><em>table name</em></p>
+<p>This options expects as its value the possibly qualified name of the
+table to create.</p>
+</li>
+<li><p class="first"><em>timezone</em></p>
+<p>This options allows to specify which timezone is used when parsing
+timestamps from an IXF file, and defaults to <em>UTC</em>. Expected values are
+either <cite>UTC</cite>, <cite>GMT</cite> or a single quoted location name such as
+<cite>‘Universal’</cite> or <cite>‘Europe/Paris’</cite>.</p>
+</li>
+</ul>
+</div></blockquote>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="../index.html">Documentation overview</a><ul>
+      <li>Previous: <a href="dbf.html" title="previous chapter">Loading DBF data</a></li>
+      <li>Next: <a href="archive.html" title="next chapter">Loading From an Archive</a></li>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="../_sources/ref/ixf.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/ref/mssql.html

@@ -0,0 +1,236 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Migrating a MS SQL Database to PostgreSQL &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="../_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="index" title="Index" href="../genindex.html" />
+    <link rel="search" title="Search" href="../search.html" />
+    <link rel="next" title="Transformation Functions" href="transforms.html" />
+    <link rel="prev" title="Migrating a SQLite database to PostgreSQL" href="sqlite.html" />
+   
+  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="migrating-a-ms-sql-database-to-postgresql">
+<h1>Migrating a MS SQL Database to PostgreSQL<a class="headerlink" href="#migrating-a-ms-sql-database-to-postgresql" title="Permalink to this headline">¶</a></h1>
+<p>This command instructs pgloader to load data from a MS SQL database.
+Automatic discovery of the schema is supported, including build of the
+indexes, primary and foreign keys constraints.</p>
+<p>Here’s an example:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>load database
+     from mssql://user@host/dbname
+     into postgresql:///dbname
+
+including only table names like &#39;GlobalAccount&#39; in schema &#39;dbo&#39;
+
+set work_mem to &#39;16MB&#39;, maintenance_work_mem to &#39;512 MB&#39;
+
+before load do $$ drop schema if exists dbo cascade; $$;
+</pre></div>
+</div>
+<p>The <cite>mssql</cite> command accepts the following clauses and options.</p>
+<div class="section" id="ms-sql-database-source-specification-from">
+<h2>MS SQL Database Source Specification: FROM<a class="headerlink" href="#ms-sql-database-source-specification-from" title="Permalink to this headline">¶</a></h2>
+<p>Connection string to an existing MS SQL database server that listens and
+welcome external TCP/IP connection. As pgloader currently piggybacks on the
+FreeTDS driver, to change the port of the server please export the <cite>TDSPORT</cite>
+environment variable.</p>
+</div>
+<div class="section" id="ms-sql-database-migration-options-with">
+<h2>MS SQL Database Migration Options: WITH<a class="headerlink" href="#ms-sql-database-migration-options-with" title="Permalink to this headline">¶</a></h2>
+<p>When loading from a <cite>MS SQL</cite> database, the same options as when loading a
+<cite>MySQL</cite> database are supported. Please refer to the MySQL section. The
+following options are added:</p>
+<blockquote>
+<div><ul>
+<li><p class="first"><em>create schemas</em></p>
+<p>When this option is listed, pgloader creates the same schemas as found
+on the MS SQL instance. This is the default.</p>
+</li>
+<li><p class="first"><em>create no schemas</em></p>
+<p>When this option is listed, pgloader refrains from creating any schemas
+at all, you must then ensure that the target schema do exist.</p>
+</li>
+</ul>
+</div></blockquote>
+</div>
+<div class="section" id="ms-sql-database-casting-rules">
+<h2>MS SQL Database Casting Rules<a class="headerlink" href="#ms-sql-database-casting-rules" title="Permalink to this headline">¶</a></h2>
+<div class="section" id="cast">
+<h3>CAST<a class="headerlink" href="#cast" title="Permalink to this headline">¶</a></h3>
+<p>The cast clause allows to specify custom casting rules, either to overload
+the default casting rules or to amend them with special cases.</p>
+<p>Please refer to the MySQL CAST clause for details.</p>
+</div>
+</div>
+<div class="section" id="ms-sql-partial-migration">
+<h2>MS SQL Partial Migration<a class="headerlink" href="#ms-sql-partial-migration" title="Permalink to this headline">¶</a></h2>
+<div class="section" id="including-only-table-names-like">
+<h3>INCLUDING ONLY TABLE NAMES LIKE<a class="headerlink" href="#including-only-table-names-like" title="Permalink to this headline">¶</a></h3>
+<p>Introduce a comma separated list of table name patterns used to limit the
+tables to migrate to a sublist. More than one such clause may be used, they
+will be accumulated together.</p>
+<p>Example:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">including</span> <span class="n">only</span> <span class="n">table</span> <span class="n">names</span> <span class="n">lile</span> <span class="s1">&#39;GlobalAccount&#39;</span> <span class="ow">in</span> <span class="n">schema</span> <span class="s1">&#39;dbo&#39;</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="excluding-table-names-like">
+<h3>EXCLUDING TABLE NAMES LIKE<a class="headerlink" href="#excluding-table-names-like" title="Permalink to this headline">¶</a></h3>
+<p>Introduce a comma separated list of table name patterns used to exclude
+table names from the migration. This filter only applies to the result of
+the <em>INCLUDING</em> filter.</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">excluding</span> <span class="n">table</span> <span class="n">names</span> <span class="n">matching</span> <span class="s1">&#39;LocalAccount&#39;</span> <span class="ow">in</span> <span class="n">schema</span> <span class="s1">&#39;dbo&#39;</span>
+</pre></div>
+</div>
+</div>
+</div>
+<div class="section" id="ms-sql-schema-transformations">
+<h2>MS SQL Schema Transformations<a class="headerlink" href="#ms-sql-schema-transformations" title="Permalink to this headline">¶</a></h2>
+<div class="section" id="alter-schema-rename-to">
+<h3>ALTER SCHEMA ‘…’ RENAME TO ‘…’<a class="headerlink" href="#alter-schema-rename-to" title="Permalink to this headline">¶</a></h3>
+<p>Allows to rename a schema on the flight, so that for instance the tables
+found in the schema ‘dbo’ in your source database will get migrated into the
+schema ‘public’ in the target database with this command:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">alter</span> <span class="n">schema</span> <span class="s1">&#39;dbo&#39;</span> <span class="n">rename</span> <span class="n">to</span> <span class="s1">&#39;public&#39;</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="alter-table-names-matching-in-schema">
+<h3>ALTER TABLE NAMES MATCHING … IN SCHEMA ‘…’<a class="headerlink" href="#alter-table-names-matching-in-schema" title="Permalink to this headline">¶</a></h3>
+<p>See the MySQL explanation for this clause above. It works the same in the
+context of migrating from MS SQL, only with the added option to specify the
+name of the schema where to find the definition of the target tables.</p>
+<p>The matching is done in pgloader itself, with a Common Lisp regular
+expression lib, so doesn’t depend on the <em>LIKE</em> implementation of MS SQL,
+nor on the lack of support for regular expressions in the engine.</p>
+</div>
+</div>
+<div class="section" id="ms-sql-driver-setup-and-encoding">
+<h2>MS SQL Driver setup and encoding<a class="headerlink" href="#ms-sql-driver-setup-and-encoding" title="Permalink to this headline">¶</a></h2>
+<p>pgloader is using the <cite>FreeTDS</cite> driver, and internally expects the data to
+be sent in utf-8. To achieve that, you can configure the FreeTDS driver with
+those defaults, in the file <cite>~/.freetds.conf</cite>:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="p">[</span><span class="k">global</span><span class="p">]</span>
+    <span class="n">tds</span> <span class="n">version</span> <span class="o">=</span> <span class="mf">7.4</span>
+    <span class="n">client</span> <span class="n">charset</span> <span class="o">=</span> <span class="n">UTF</span><span class="o">-</span><span class="mi">8</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="default-ms-sql-casting-rules">
+<h2>Default MS SQL Casting Rules<a class="headerlink" href="#default-ms-sql-casting-rules" title="Permalink to this headline">¶</a></h2>
+<p>When migrating from MS SQL the following Casting Rules are provided:</p>
+<p>Numbers:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="nb">type</span> <span class="n">tinyint</span> <span class="n">to</span> <span class="n">smallint</span>
+
+<span class="nb">type</span> <span class="nb">float</span> <span class="n">to</span> <span class="nb">float</span>   <span class="n">using</span> <span class="nb">float</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">string</span>
+<span class="nb">type</span> <span class="n">real</span> <span class="n">to</span> <span class="n">real</span>     <span class="n">using</span> <span class="nb">float</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">string</span>
+<span class="nb">type</span> <span class="n">double</span> <span class="n">to</span> <span class="n">double</span> <span class="n">precision</span>     <span class="n">using</span> <span class="nb">float</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">string</span>
+<span class="nb">type</span> <span class="n">numeric</span> <span class="n">to</span> <span class="n">numeric</span>     <span class="n">using</span> <span class="nb">float</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">string</span>
+<span class="nb">type</span> <span class="n">decimal</span> <span class="n">to</span> <span class="n">numeric</span>     <span class="n">using</span> <span class="nb">float</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">string</span>
+<span class="nb">type</span> <span class="n">money</span> <span class="n">to</span> <span class="n">numeric</span>     <span class="n">using</span> <span class="nb">float</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">string</span>
+<span class="nb">type</span> <span class="n">smallmoney</span> <span class="n">to</span> <span class="n">numeric</span>     <span class="n">using</span> <span class="nb">float</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">string</span>
+</pre></div>
+</div>
+<p>Texts:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="nb">type</span> <span class="n">char</span>      <span class="n">to</span> <span class="n">text</span> <span class="n">drop</span> <span class="n">typemod</span>
+<span class="nb">type</span> <span class="n">nchat</span>     <span class="n">to</span> <span class="n">text</span> <span class="n">drop</span> <span class="n">typemod</span>
+<span class="nb">type</span> <span class="n">varchar</span>   <span class="n">to</span> <span class="n">text</span> <span class="n">drop</span> <span class="n">typemod</span>
+<span class="nb">type</span> <span class="n">nvarchar</span>  <span class="n">to</span> <span class="n">text</span> <span class="n">drop</span> <span class="n">typemod</span>
+<span class="nb">type</span> <span class="n">xml</span>       <span class="n">to</span> <span class="n">text</span> <span class="n">drop</span> <span class="n">typemod</span>
+</pre></div>
+</div>
+<p>Binary:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="nb">type</span> <span class="n">binary</span>    <span class="n">to</span> <span class="n">bytea</span> <span class="n">using</span> <span class="n">byte</span><span class="o">-</span><span class="n">vector</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">bytea</span>
+<span class="nb">type</span> <span class="n">varbinary</span> <span class="n">to</span> <span class="n">bytea</span> <span class="n">using</span> <span class="n">byte</span><span class="o">-</span><span class="n">vector</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">bytea</span>
+</pre></div>
+</div>
+<p>Date:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="nb">type</span> <span class="n">datetime</span>    <span class="n">to</span> <span class="n">timestamptz</span>
+<span class="nb">type</span> <span class="n">datetime2</span>   <span class="n">to</span> <span class="n">timestamptz</span>
+</pre></div>
+</div>
+<p>Others:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="nb">type</span> <span class="n">bit</span> <span class="n">to</span> <span class="n">boolean</span>
+<span class="nb">type</span> <span class="n">hierarchyid</span> <span class="n">to</span> <span class="n">bytea</span>
+<span class="nb">type</span> <span class="n">geography</span> <span class="n">to</span> <span class="n">bytea</span>
+<span class="nb">type</span> <span class="n">uniqueidentifier</span> <span class="n">to</span> <span class="n">uuid</span> <span class="n">using</span> <span class="n">sql</span><span class="o">-</span><span class="n">server</span><span class="o">-</span><span class="n">uniqueidentifier</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">uuid</span>
+</pre></div>
+</div>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="../index.html">Documentation overview</a><ul>
+      <li>Previous: <a href="sqlite.html" title="previous chapter">Migrating a SQLite database to PostgreSQL</a></li>
+      <li>Next: <a href="transforms.html" title="next chapter">Transformation Functions</a></li>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="../_sources/ref/mssql.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/ref/mysql.html

@@ -0,0 +1,639 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Migrating a MySQL Database to PostgreSQL &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="../_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="index" title="Index" href="../genindex.html" />
+    <link rel="search" title="Search" href="../search.html" />
+    <link rel="next" title="Migrating a SQLite database to PostgreSQL" href="sqlite.html" />
+    <link rel="prev" title="Loading From an Archive" href="archive.html" />
+   
+  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="migrating-a-mysql-database-to-postgresql">
+<h1>Migrating a MySQL Database to PostgreSQL<a class="headerlink" href="#migrating-a-mysql-database-to-postgresql" title="Permalink to this headline">¶</a></h1>
+<p>This command instructs pgloader to load data from a database connection. The
+only supported database source is currently <em>MySQL</em>, and pgloader supports
+dynamically converting the schema of the source database and the indexes
+building.</p>
+<p>A default set of casting rules are provided and might be overloaded and
+appended to by the command.</p>
+<p>Here’s an example using as many options as possible, some of them even being
+defaults. Chances are you don’t need that complex a setup, don’t copy and
+paste it, use it only as a reference!</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>LOAD DATABASE
+     FROM      mysql://root@localhost/sakila
+     INTO postgresql://localhost:54393/sakila
+
+ WITH include drop, create tables, create indexes, reset sequences,
+      workers = 8, concurrency = 1,
+      multiple readers per thread, rows per range = 50000
+
+  SET PostgreSQL PARAMETERS
+      maintenance_work_mem to &#39;128MB&#39;,
+      work_mem to &#39;12MB&#39;,
+      search_path to &#39;sakila, public, &quot;$user&quot;&#39;
+
+  SET MySQL PARAMETERS
+      net_read_timeout  = &#39;120&#39;,
+      net_write_timeout = &#39;120&#39;
+
+ CAST type bigint when (= precision 20) to bigserial drop typemod,
+      type date drop not null drop default using zero-dates-to-null,
+      -- type tinyint to boolean using tinyint-to-boolean,
+      type year to integer
+
+ MATERIALIZE VIEWS film_list, staff_list
+
+ -- INCLUDING ONLY TABLE NAMES MATCHING ~/film/, &#39;actor&#39;
+ -- EXCLUDING TABLE NAMES MATCHING ~&lt;ory&gt;
+ -- DECODING TABLE NAMES MATCHING ~/messed/, ~/encoding/ AS utf8
+ -- ALTER TABLE NAMES MATCHING &#39;film&#39; RENAME TO &#39;films&#39;
+ -- ALTER TABLE NAMES MATCHING ~/_list$/ SET SCHEMA &#39;mv&#39;
+
+ ALTER TABLE NAMES MATCHING ~/_list$/, &#39;sales_by_store&#39;, ~/sales_by/
+  SET SCHEMA &#39;mv&#39;
+
+ ALTER TABLE NAMES MATCHING &#39;film&#39; RENAME TO &#39;films&#39;
+ ALTER TABLE NAMES MATCHING ~/./ SET (fillfactor=&#39;40&#39;)
+
+ ALTER SCHEMA &#39;sakila&#39; RENAME TO &#39;pagila&#39;
+
+ BEFORE LOAD DO
+   $$ create schema if not exists pagila; $$,
+   $$ create schema if not exists mv;     $$,
+   $$ alter database sakila set search_path to pagila, mv, public; $$;
+</pre></div>
+</div>
+<p>The <cite>database</cite> command accepts the following clauses and options.</p>
+<div class="section" id="mysql-database-source-specification-from">
+<h2>MySQL Database Source Specification: FROM<a class="headerlink" href="#mysql-database-source-specification-from" title="Permalink to this headline">¶</a></h2>
+<p>Must be a connection URL pointing to a MySQL database.</p>
+<p>If the connection URI contains a table name, then only this table is
+migrated from MySQL to PostgreSQL.</p>
+<p>See the <cite>SOURCE CONNECTION STRING</cite> section above for details on how to write
+the connection string. The MySQL connection string accepts the same
+parameter <em>sslmode</em> as the PostgreSQL connection string, but the <em>verify</em>
+mode is not implemented (yet).</p>
+<p>Environment variables described in
+&lt;<a class="reference external" href="http://dev.mysql.com/doc/refman/5.0/en/environment-variables.html">http://dev.mysql.com/doc/refman/5.0/en/environment-variables.html</a>&gt; can be
+used as default values too. If the user is not provided, then it defaults to
+<cite>USER</cite> environment variable value. The password can be provided with the
+environment variable <cite>MYSQL_PWD</cite>. The host can be provided with the
+environment variable <cite>MYSQL_HOST</cite> and otherwise defaults to <cite>localhost</cite>. The
+port can be provided with the environment variable <cite>MYSQL_TCP_PORT</cite> and
+otherwise defaults to <cite>3306</cite>.</p>
+</div>
+<div class="section" id="mysql-database-migration-options-with">
+<h2>MySQL Database Migration Options: WITH<a class="headerlink" href="#mysql-database-migration-options-with" title="Permalink to this headline">¶</a></h2>
+<p>When loading from a <cite>MySQL</cite> database, the following options are supported,
+and the default <em>WITH</em> clause is: <em>no truncate</em>, <em>create schema</em>, <em>create
+tables</em>, <em>include drop</em>, <em>create indexes</em>, <em>reset sequences</em>, <em>foreign
+keys</em>, <em>downcase identifiers</em>, <em>uniquify index names</em>.</p>
+<blockquote>
+<div><ul>
+<li><p class="first"><em>include drop</em></p>
+<p>When this option is listed, pgloader drops all the tables in the target
+PostgreSQL database whose names appear in the MySQL database. This
+option allows for using the same command several times in a row until
+you figure out all the options, starting automatically from a clean
+environment. Please note that <cite>CASCADE</cite> is used to ensure that tables
+are dropped even if there are foreign keys pointing to them. This is
+precisely what <cite>include drop</cite> is intended to do: drop all target tables
+and recreate them.</p>
+<p>Great care needs to be taken when using <cite>include drop</cite>, as it will
+cascade to <em>all</em> objects referencing the target tables, possibly
+including other tables that are not being loaded from the source DB.</p>
+</li>
+<li><p class="first"><em>include no drop</em></p>
+<p>When this option is listed, pgloader will not include any <cite>DROP</cite>
+statement when loading the data.</p>
+</li>
+<li><p class="first"><em>truncate</em></p>
+<p>When this option is listed, pgloader issue the <cite>TRUNCATE</cite> command
+against each PostgreSQL table just before loading data into it.</p>
+</li>
+<li><p class="first"><em>no truncate</em></p>
+<p>When this option is listed, pgloader issues no <cite>TRUNCATE</cite> command.</p>
+</li>
+<li><p class="first"><em>disable triggers</em></p>
+<p>When this option is listed, pgloader issues an <cite>ALTER TABLE … DISABLE
+TRIGGER ALL</cite> command against the PostgreSQL target table before copying
+the data, then the command <cite>ALTER TABLE … ENABLE TRIGGER ALL</cite> once the
+<cite>COPY</cite> is done.</p>
+<p>This option allows loading data into a pre-existing table ignoring the
+<em>foreign key constraints</em> and user defined triggers and may result in
+invalid <em>foreign key constraints</em> once the data is loaded. Use with
+care.</p>
+</li>
+<li><p class="first"><em>create tables</em></p>
+<p>When this option is listed, pgloader creates the table using the meta
+data found in the <cite>MySQL</cite> file, which must contain a list of fields with
+their data type. A standard data type conversion from DBF to PostgreSQL
+is done.</p>
+</li>
+<li><p class="first"><em>create no tables</em></p>
+<p>When this option is listed, pgloader skips the creation of table before
+loading data, target tables must then already exist.</p>
+<p>Also, when using <em>create no tables</em> pgloader fetches the metadata from
+the current target database and checks type casting, then will remove
+constraints and indexes prior to loading the data and install them back
+again once the loading is done.</p>
+</li>
+<li><p class="first"><em>create indexes</em></p>
+<p>When this option is listed, pgloader gets the definitions of all the
+indexes found in the MySQL database and create the same set of index
+definitions against the PostgreSQL database.</p>
+</li>
+<li><p class="first"><em>create no indexes</em></p>
+<p>When this option is listed, pgloader skips the creating indexes.</p>
+</li>
+<li><p class="first"><em>drop indexes</em></p>
+<p>When this option is listed, pgloader drops the indexes in the target
+database before loading the data, and creates them again at the end
+of the data copy.</p>
+</li>
+<li><p class="first"><em>uniquify index names</em>, <em>preserve index names</em></p>
+<p>MySQL index names are unique per-table whereas in PostgreSQL index names
+have to be unique per-schema. The default for pgloader is to change the
+index name by prefixing it with <cite>idx_OID</cite> where <cite>OID</cite> is the internal
+numeric identifier of the table the index is built against.</p>
+<p>In somes cases like when the DDL are entirely left to a framework it
+might be sensible for pgloader to refrain from handling index unique
+names, that is achieved by using the <em>preserve index names</em> option.</p>
+<p>The default is to <em>uniquify index names</em>.</p>
+<p>Even when using the option <em>preserve index names</em>, MySQL primary key
+indexes named “PRIMARY” will get their names uniquified. Failing to do
+so would prevent the primary keys to be created again in PostgreSQL
+where the index names must be unique per schema.</p>
+</li>
+<li><p class="first"><em>drop schema</em></p>
+<p>When this option is listed, pgloader drops the target schema in the
+target PostgreSQL database before creating it again and all the objects
+it contains. The default behavior doesn’t drop the target schemas.</p>
+</li>
+<li><p class="first"><em>foreign keys</em></p>
+<p>When this option is listed, pgloader gets the definitions of all the
+foreign keys found in the MySQL database and create the same set of
+foreign key definitions against the PostgreSQL database.</p>
+</li>
+<li><p class="first"><em>no foreign keys</em></p>
+<p>When this option is listed, pgloader skips creating foreign keys.</p>
+</li>
+<li><p class="first"><em>reset sequences</em></p>
+<p>When this option is listed, at the end of the data loading and after the
+indexes have all been created, pgloader resets all the PostgreSQL
+sequences created to the current maximum value of the column they are
+attached to.</p>
+<p>The options <em>schema only</em> and <em>data only</em> have no effects on this
+option.</p>
+</li>
+<li><p class="first"><em>reset no sequences</em></p>
+<p>When this option is listed, pgloader skips resetting sequences after the
+load.</p>
+<p>The options <em>schema only</em> and <em>data only</em> have no effects on this
+option.</p>
+</li>
+<li><p class="first"><em>downcase identifiers</em></p>
+<p>When this option is listed, pgloader converts all MySQL identifiers
+(table names, index names, column names) to <em>downcase</em>, except for
+PostgreSQL <em>reserved</em> keywords.</p>
+<p>The PostgreSQL <em>reserved</em> keywords are determined dynamically by using
+the system function <cite>pg_get_keywords()</cite>.</p>
+</li>
+<li><p class="first"><em>quote identifiers</em></p>
+<p>When this option is listed, pgloader quotes all MySQL identifiers so
+that their case is respected. Note that you will then have to do the
+same thing in your application code queries.</p>
+</li>
+<li><p class="first"><em>schema only</em></p>
+<p>When this option is listed pgloader refrains from migrating the data
+over. Note that the schema in this context includes the indexes when the
+option <em>create indexes</em> has been listed.</p>
+</li>
+<li><p class="first"><em>data only</em></p>
+<p>When this option is listed pgloader only issues the <cite>COPY</cite> statements,
+without doing any other processing.</p>
+</li>
+<li><p class="first"><em>single reader per thread</em>, <em>multiple readers per thread</em></p>
+<p>The default is <em>single reader per thread</em> and it means that each
+MySQL table is read by a single thread as a whole, with a single
+<cite>SELECT</cite> statement using no <cite>WHERE</cite> clause.</p>
+<p>When using <em>multiple readers per thread</em> pgloader may be able to
+divide the reading work into several threads, as many as the
+<em>concurrency</em> setting, which needs to be greater than 1 for this
+option to kick be activated.</p>
+<p>For each source table, pgloader searches for a primary key over a
+single numeric column, or a multiple-column primary key index for
+which the first column is of a numeric data type (one of <cite>integer</cite>
+or <cite>bigint</cite>). When such an index exists, pgloader runs a query to
+find the <em>min</em> and <em>max</em> values on this column, and then split that
+range into many ranges containing a maximum of <em>rows per range</em>.</p>
+<p>When the range list we then obtain contains at least as many ranges
+than our concurrency setting, then we distribute those ranges to
+each reader thread.</p>
+<p>So when all the conditions are met, pgloader then starts as many
+reader thread as the <em>concurrency</em> setting, and each reader thread
+issues several queries with a <cite>WHERE id &gt;= x AND id &lt; y</cite>, where <cite>y -
+x = rows per range</cite> or less (for the last range, depending on the
+max value just obtained.</p>
+</li>
+<li><p class="first"><em>rows per range</em></p>
+<p>How many rows are fetched per <cite>SELECT</cite> query when using <em>multiple
+readers per thread</em>, see above for details.</p>
+</li>
+<li><p class="first"><em>SET MySQL PARAMETERS</em></p>
+<p>The <em>SET MySQL PARAMETERS</em> allows setting MySQL parameters using the
+MySQL <cite>SET</cite> command each time pgloader connects to it.</p>
+</li>
+</ul>
+</div></blockquote>
+</div>
+<div class="section" id="mysql-database-casting-rules">
+<h2>MySQL Database Casting Rules<a class="headerlink" href="#mysql-database-casting-rules" title="Permalink to this headline">¶</a></h2>
+<p>The command <em>CAST</em> introduces user-defined casting rules.</p>
+<p>The cast clause allows to specify custom casting rules, either to overload
+the default casting rules or to amend them with special cases.</p>
+<p>A casting rule is expected to follow one of the forms:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="nb">type</span> <span class="o">&lt;</span><span class="n">mysql</span><span class="o">-</span><span class="nb">type</span><span class="o">-</span><span class="n">name</span><span class="o">&gt;</span> <span class="p">[</span> <span class="o">&lt;</span><span class="n">guard</span><span class="o">&gt;</span> <span class="o">...</span> <span class="p">]</span> <span class="n">to</span> <span class="o">&lt;</span><span class="n">pgsql</span><span class="o">-</span><span class="nb">type</span><span class="o">-</span><span class="n">name</span><span class="o">&gt;</span> <span class="p">[</span> <span class="o">&lt;</span><span class="n">option</span><span class="o">&gt;</span> <span class="o">...</span> <span class="p">]</span>
+<span class="n">column</span> <span class="o">&lt;</span><span class="n">table</span><span class="o">-</span><span class="n">name</span><span class="o">&gt;.&lt;</span><span class="n">column</span><span class="o">-</span><span class="n">name</span><span class="o">&gt;</span> <span class="p">[</span> <span class="o">&lt;</span><span class="n">guards</span><span class="o">&gt;</span> <span class="p">]</span> <span class="n">to</span> <span class="o">...</span>
+</pre></div>
+</div>
+<p>It’s possible for a <em>casting rule</em> to either match against a MySQL data type
+or against a given <em>column name</em> in a given <em>table name</em>. That flexibility
+allows to cope with cases where the type <cite>tinyint</cite> might have been used as a
+<cite>boolean</cite> in some cases but as a <cite>smallint</cite> in others.</p>
+<p>The <em>casting rules</em> are applied in order, the first match prevents following
+rules to be applied, and user defined rules are evaluated first.</p>
+<p>The supported guards are:</p>
+<blockquote>
+<div><ul>
+<li><p class="first"><em>when unsigned</em></p>
+<p>The casting rule is only applied against MySQL columns of the source
+type that have the keyword <em>unsigned</em> in their data type definition.</p>
+<p>Example of a casting rule using a <em>unsigned</em> guard:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="nb">type</span> <span class="n">smallint</span> <span class="n">when</span> <span class="n">unsigned</span> <span class="n">to</span> <span class="n">integer</span> <span class="n">drop</span> <span class="n">typemod</span>
+</pre></div>
+</div>
+</li>
+<li><p class="first"><em>when default ‘value’</em></p>
+<p>The casting rule is only applied against MySQL columns of the source
+type that have given <em>value</em>, which must be a single-quoted or a
+double-quoted string.</p>
+</li>
+<li><p class="first"><em>when typemod expression</em></p>
+<p>The casting rule is only applied against MySQL columns of the source
+type that have a <em>typemod</em> value matching the given <em>typemod
+expression</em>. The <em>typemod</em> is separated into its <em>precision</em> and <em>scale</em>
+components.</p>
+<p>Example of a cast rule using a <em>typemod</em> guard:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="nb">type</span> <span class="n">char</span> <span class="n">when</span> <span class="p">(</span><span class="o">=</span> <span class="n">precision</span> <span class="mi">1</span><span class="p">)</span> <span class="n">to</span> <span class="n">char</span> <span class="n">keep</span> <span class="n">typemod</span>
+</pre></div>
+</div>
+<p>This expression casts MySQL <cite>char(1)</cite> column to a PostgreSQL column of
+type <cite>char(1)</cite> while allowing for the general case <cite>char(N)</cite> will be
+converted by the default cast rule into a PostgreSQL type <cite>varchar(N)</cite>.</p>
+</li>
+<li><p class="first"><em>with extra auto_increment</em></p>
+<p>The casting rule is only applied against MySQL columns having the
+<em>extra</em> column <cite>auto_increment</cite> option set, so that it’s possible to
+target e.g. <cite>serial</cite> rather than <cite>integer</cite>.</p>
+<p>The default matching behavior, when this option isn’t set, is to match
+both columns with the extra definition and without.</p>
+<p>This means that if you want to implement a casting rule that target
+either <cite>serial</cite> or <cite>integer</cite> from a <cite>smallint</cite> definition depending on
+the <em>auto_increment</em> extra bit of information from MySQL, then you need
+to spell out two casting rules as following:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="nb">type</span> <span class="n">smallint</span>  <span class="k">with</span> <span class="n">extra</span> <span class="n">auto_increment</span>
+  <span class="n">to</span> <span class="n">serial</span> <span class="n">drop</span> <span class="n">typemod</span> <span class="n">keep</span> <span class="n">default</span> <span class="n">keep</span> <span class="ow">not</span> <span class="n">null</span><span class="p">,</span>
+
+<span class="nb">type</span> <span class="n">smallint</span>
+  <span class="n">to</span> <span class="n">integer</span> <span class="n">drop</span> <span class="n">typemod</span> <span class="n">keep</span> <span class="n">default</span> <span class="n">keep</span> <span class="ow">not</span> <span class="n">null</span>
+</pre></div>
+</div>
+</li>
+</ul>
+</div></blockquote>
+<p>The supported casting options are:</p>
+<blockquote>
+<div><ul>
+<li><p class="first"><em>drop default</em>, <em>keep default</em></p>
+<p>When the option <em>drop default</em> is listed, pgloader drops any
+existing default expression in the MySQL database for columns of the
+source type from the <cite>CREATE TABLE</cite> statement it generates.</p>
+<p>The spelling <em>keep default</em> explicitly prevents that behaviour and
+can be used to overload the default casting rules.</p>
+</li>
+<li><p class="first"><em>drop not null</em>, <em>keep not null</em>, <em>set not null</em></p>
+<p>When the option <em>drop not null</em> is listed, pgloader drops any
+existing <cite>NOT NULL</cite> constraint associated with the given source
+MySQL datatype when it creates the tables in the PostgreSQL
+database.</p>
+<p>The spelling <em>keep not null</em> explicitly prevents that behaviour and
+can be used to overload the default casting rules.</p>
+<p>When the option <em>set not null</em> is listed, pgloader sets a <cite>NOT NULL</cite>
+constraint on the target column regardless whether it has been set
+in the source MySQL column.</p>
+</li>
+<li><p class="first"><em>drop typemod</em>, <em>keep typemod</em></p>
+<p>When the option <em>drop typemod</em> is listed, pgloader drops any
+existing <em>typemod</em> definition (e.g. <em>precision</em> and <em>scale</em>) from
+the datatype definition found in the MySQL columns of the source
+type when it created the tables in the PostgreSQL database.</p>
+<p>The spelling <em>keep typemod</em> explicitly prevents that behaviour and
+can be used to overload the default casting rules.</p>
+</li>
+<li><p class="first"><em>using</em></p>
+<p>This option takes as its single argument the name of a function to
+be found in the <cite>pgloader.transforms</cite> Common Lisp package. See above
+for details.</p>
+<p>It’s possible to augment a default cast rule (such as one that
+applies against <cite>ENUM</cite> data type for example) with a <em>transformation
+function</em> by omitting entirely the <cite>type</cite> parts of the casting rule,
+as in the following example:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">column</span> <span class="nb">enumerate</span><span class="o">.</span><span class="n">foo</span> <span class="n">using</span> <span class="n">empty</span><span class="o">-</span><span class="n">string</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">null</span>
+</pre></div>
+</div>
+</li>
+</ul>
+</div></blockquote>
+</div>
+<div class="section" id="mysql-views-support">
+<h2>MySQL Views Support<a class="headerlink" href="#mysql-views-support" title="Permalink to this headline">¶</a></h2>
+<p>MySQL views support allows pgloader to migrate view as if they were base
+tables. This feature then allows for on-the-fly transformation from MySQL to
+PostgreSQL, as the view definition is used rather than the base data.</p>
+<div class="section" id="materialize-views">
+<h3>MATERIALIZE VIEWS<a class="headerlink" href="#materialize-views" title="Permalink to this headline">¶</a></h3>
+<p>This clause allows you to implement custom data processing at the data
+source by providing a <em>view definition</em> against which pgloader will query
+the data. It’s not possible to just allow for plain <cite>SQL</cite> because we want to
+know a lot about the exact data types of each column involved in the query
+output.</p>
+<p>This clause expect a comma separated list of view definitions, each one
+being either the name of an existing view in your database or the following
+expression:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>*name* `AS` `$$` *sql query* `$$`
+</pre></div>
+</div>
+<p>The <em>name</em> and the <em>sql query</em> will be used in a <cite>CREATE VIEW</cite> statement at
+the beginning of the data loading, and the resulting view will then be
+dropped at the end of the data loading.</p>
+</div>
+<div class="section" id="materialize-all-views">
+<h3>MATERIALIZE ALL VIEWS<a class="headerlink" href="#materialize-all-views" title="Permalink to this headline">¶</a></h3>
+<p>Same behaviour as <em>MATERIALIZE VIEWS</em> using the dynamic list of views as
+returned by MySQL rather than asking the user to specify the list.</p>
+</div>
+</div>
+<div class="section" id="mysql-partial-migration">
+<h2>MySQL Partial Migration<a class="headerlink" href="#mysql-partial-migration" title="Permalink to this headline">¶</a></h2>
+<div class="section" id="including-only-table-names-matching">
+<h3>INCLUDING ONLY TABLE NAMES MATCHING<a class="headerlink" href="#including-only-table-names-matching" title="Permalink to this headline">¶</a></h3>
+<p>Introduce a comma separated list of table names or <em>regular expression</em> used
+to limit the tables to migrate to a sublist.</p>
+<p>Example:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">including</span> <span class="n">only</span> <span class="n">table</span> <span class="n">names</span> <span class="n">matching</span> <span class="o">~/</span><span class="n">film</span><span class="o">/</span><span class="p">,</span> <span class="s1">&#39;actor&#39;</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="excluding-table-names-matching">
+<h3>EXCLUDING TABLE NAMES MATCHING<a class="headerlink" href="#excluding-table-names-matching" title="Permalink to this headline">¶</a></h3>
+<p>Introduce a comma separated list of table names or <em>regular expression</em> used
+to exclude table names from the migration. This filter only applies to the
+result of the <em>INCLUDING</em> filter.</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">excluding</span> <span class="n">table</span> <span class="n">names</span> <span class="n">matching</span> <span class="o">~&lt;</span><span class="n">ory</span><span class="o">&gt;</span>
+</pre></div>
+</div>
+</div>
+</div>
+<div class="section" id="mysql-encoding-support">
+<h2>MySQL Encoding Support<a class="headerlink" href="#mysql-encoding-support" title="Permalink to this headline">¶</a></h2>
+<div class="section" id="decoding-table-names-matching">
+<h3>DECODING TABLE NAMES MATCHING<a class="headerlink" href="#decoding-table-names-matching" title="Permalink to this headline">¶</a></h3>
+<p>Introduce a comma separated list of table names or <em>regular expressions</em>
+used to force the encoding to use when processing data from MySQL. If the
+data encoding known to you is different from MySQL’s idea about it, this is
+the option to use.</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">decoding</span> <span class="n">table</span> <span class="n">names</span> <span class="n">matching</span> <span class="o">~/</span><span class="n">messed</span><span class="o">/</span><span class="p">,</span> <span class="o">~/</span><span class="n">encoding</span><span class="o">/</span> <span class="n">AS</span> <span class="n">utf8</span>
+</pre></div>
+</div>
+<p>You can use as many such rules as you need, all with possibly different
+encodings.</p>
+</div>
+</div>
+<div class="section" id="mysql-schema-transformations">
+<h2>MySQL Schema Transformations<a class="headerlink" href="#mysql-schema-transformations" title="Permalink to this headline">¶</a></h2>
+<div class="section" id="alter-table-names-matching">
+<h3>ALTER TABLE NAMES MATCHING<a class="headerlink" href="#alter-table-names-matching" title="Permalink to this headline">¶</a></h3>
+<p>Introduce a comma separated list of table names or <em>regular expressions</em>
+that you want to target in the pgloader <em>ALTER TABLE</em> command. The only two
+available actions are <em>SET SCHEMA</em> and <em>RENAME TO</em>, both take a quoted
+string as parameter:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>ALTER TABLE NAMES MATCHING ~/_list$/, &#39;sales_by_store&#39;, ~/sales_by/
+ SET SCHEMA &#39;mv&#39;
+
+ALTER TABLE NAMES MATCHING &#39;film&#39; RENAME TO &#39;films&#39;
+
+ALTER TABLE NAMES MATCHING ~/./ SET (fillfactor=&#39;40&#39;)
+</pre></div>
+</div>
+<p>You can use as many such rules as you need. The list of tables to be
+migrated is searched in pgloader memory against the <em>ALTER TABLE</em> matching
+rules, and for each command pgloader stops at the first matching criteria
+(regexp or string).</p>
+<p>No <em>ALTER TABLE</em> command is sent to PostgreSQL, the modification happens at
+the level of the pgloader in-memory representation of your source database
+schema. In case of a name change, the mapping is kept and reused in the
+<em>foreign key</em> and <em>index</em> support.</p>
+<p>The <em>SET ()</em> action takes effect as a <em>WITH</em> clause for the <cite>CREATE TABLE</cite>
+command that pgloader will run when it has to create a table.</p>
+</div>
+</div>
+<div class="section" id="mysql-migration-limitations">
+<h2>MySQL Migration: limitations<a class="headerlink" href="#mysql-migration-limitations" title="Permalink to this headline">¶</a></h2>
+<p>The <cite>database</cite> command currently only supports MySQL source database and has
+the following limitations:</p>
+<blockquote>
+<div><ul>
+<li><p class="first">Views are not migrated,</p>
+<p>Supporting views might require implementing a full SQL parser for the
+MySQL dialect with a porting engine to rewrite the SQL against
+PostgreSQL, including renaming functions and changing some constructs.</p>
+<p>While it’s not theoretically impossible, don’t hold your breath.</p>
+</li>
+<li><p class="first">Triggers are not migrated</p>
+<p>The difficulty of doing so is not yet assessed.</p>
+</li>
+<li><p class="first">Of the geometric datatypes, only the <cite>POINT</cite> database has been covered.
+The other ones should be easy enough to implement now, it’s just not
+done yet.</p>
+</li>
+</ul>
+</div></blockquote>
+</div>
+<div class="section" id="default-mysql-casting-rules">
+<h2>Default MySQL Casting Rules<a class="headerlink" href="#default-mysql-casting-rules" title="Permalink to this headline">¶</a></h2>
+<p>When migrating from MySQL the following Casting Rules are provided:</p>
+<p>Numbers:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="nb">type</span> <span class="nb">int</span> <span class="k">with</span> <span class="n">extra</span> <span class="n">auto_increment</span> <span class="n">to</span> <span class="n">serial</span> <span class="n">when</span> <span class="p">(</span><span class="o">&lt;</span> <span class="n">precision</span> <span class="mi">10</span><span class="p">)</span>
+<span class="nb">type</span> <span class="nb">int</span> <span class="k">with</span> <span class="n">extra</span> <span class="n">auto_increment</span> <span class="n">to</span> <span class="n">bigserial</span> <span class="n">when</span> <span class="p">(</span><span class="o">&lt;=</span> <span class="mi">10</span> <span class="n">precision</span><span class="p">)</span>
+<span class="nb">type</span> <span class="nb">int</span> <span class="n">to</span> <span class="nb">int</span>       <span class="n">when</span>  <span class="p">(</span><span class="o">&lt;</span> <span class="n">precision</span> <span class="mi">10</span><span class="p">)</span>
+<span class="nb">type</span> <span class="nb">int</span> <span class="n">to</span> <span class="n">bigint</span>    <span class="n">when</span>  <span class="p">(</span><span class="o">&lt;=</span> <span class="mi">10</span> <span class="n">precision</span><span class="p">)</span>
+<span class="nb">type</span> <span class="n">tinyint</span>   <span class="k">with</span> <span class="n">extra</span> <span class="n">auto_increment</span> <span class="n">to</span> <span class="n">serial</span>
+<span class="nb">type</span> <span class="n">smallint</span>  <span class="k">with</span> <span class="n">extra</span> <span class="n">auto_increment</span> <span class="n">to</span> <span class="n">serial</span>
+<span class="nb">type</span> <span class="n">mediumint</span> <span class="k">with</span> <span class="n">extra</span> <span class="n">auto_increment</span> <span class="n">to</span> <span class="n">serial</span>
+<span class="nb">type</span> <span class="n">bigint</span>    <span class="k">with</span> <span class="n">extra</span> <span class="n">auto_increment</span> <span class="n">to</span> <span class="n">bigserial</span>
+
+<span class="nb">type</span> <span class="n">tinyint</span> <span class="n">to</span> <span class="n">boolean</span> <span class="n">when</span> <span class="p">(</span><span class="o">=</span> <span class="mi">1</span> <span class="n">precision</span><span class="p">)</span> <span class="n">using</span> <span class="n">tinyint</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">boolean</span>
+
+<span class="nb">type</span> <span class="n">tinyint</span> <span class="n">when</span> <span class="n">unsigned</span> <span class="n">to</span> <span class="n">smallint</span>   <span class="n">drop</span> <span class="n">typemod</span>
+<span class="nb">type</span> <span class="n">smallint</span> <span class="n">when</span> <span class="n">unsigned</span> <span class="n">to</span> <span class="n">integer</span>  <span class="n">drop</span> <span class="n">typemod</span>
+<span class="nb">type</span> <span class="n">mediumint</span> <span class="n">when</span> <span class="n">unsigned</span> <span class="n">to</span> <span class="n">integer</span>  <span class="n">drop</span> <span class="n">typemod</span>
+<span class="nb">type</span> <span class="n">integer</span> <span class="n">when</span> <span class="n">unsigned</span> <span class="n">to</span> <span class="n">bigint</span>    <span class="n">drop</span> <span class="n">typemod</span>
+
+<span class="nb">type</span> <span class="n">tinyint</span> <span class="n">to</span> <span class="n">smallint</span>   <span class="n">drop</span> <span class="n">typemod</span>
+<span class="nb">type</span> <span class="n">smallint</span> <span class="n">to</span> <span class="n">smallint</span>  <span class="n">drop</span> <span class="n">typemod</span>
+<span class="nb">type</span> <span class="n">mediumint</span> <span class="n">to</span> <span class="n">integer</span>  <span class="n">drop</span> <span class="n">typemod</span>
+<span class="nb">type</span> <span class="n">integer</span> <span class="n">to</span> <span class="n">integer</span>    <span class="n">drop</span> <span class="n">typemod</span>
+<span class="nb">type</span> <span class="n">bigint</span> <span class="n">to</span> <span class="n">bigint</span>      <span class="n">drop</span> <span class="n">typemod</span>
+
+<span class="nb">type</span> <span class="nb">float</span> <span class="n">to</span> <span class="nb">float</span>        <span class="n">drop</span> <span class="n">typemod</span>
+<span class="nb">type</span> <span class="n">double</span> <span class="n">to</span> <span class="n">double</span> <span class="n">precision</span> <span class="n">drop</span> <span class="n">typemod</span>
+
+<span class="nb">type</span> <span class="n">numeric</span> <span class="n">to</span> <span class="n">numeric</span> <span class="n">keep</span> <span class="n">typemod</span>
+<span class="nb">type</span> <span class="n">decimal</span> <span class="n">to</span> <span class="n">decimal</span> <span class="n">keep</span> <span class="n">typemod</span>
+</pre></div>
+</div>
+<p>Texts:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="nb">type</span> <span class="n">char</span>       <span class="n">to</span> <span class="n">char</span> <span class="n">keep</span> <span class="n">typemod</span> <span class="n">using</span> <span class="n">remove</span><span class="o">-</span><span class="n">null</span><span class="o">-</span><span class="n">characters</span>
+<span class="nb">type</span> <span class="n">varchar</span>    <span class="n">to</span> <span class="n">varchar</span> <span class="n">keep</span> <span class="n">typemod</span> <span class="n">using</span> <span class="n">remove</span><span class="o">-</span><span class="n">null</span><span class="o">-</span><span class="n">characters</span>
+<span class="nb">type</span> <span class="n">tinytext</span>   <span class="n">to</span> <span class="n">text</span> <span class="n">using</span> <span class="n">remove</span><span class="o">-</span><span class="n">null</span><span class="o">-</span><span class="n">characters</span>
+<span class="nb">type</span> <span class="n">text</span>       <span class="n">to</span> <span class="n">text</span> <span class="n">using</span> <span class="n">remove</span><span class="o">-</span><span class="n">null</span><span class="o">-</span><span class="n">characters</span>
+<span class="nb">type</span> <span class="n">mediumtext</span> <span class="n">to</span> <span class="n">text</span> <span class="n">using</span> <span class="n">remove</span><span class="o">-</span><span class="n">null</span><span class="o">-</span><span class="n">characters</span>
+<span class="nb">type</span> <span class="n">longtext</span>   <span class="n">to</span> <span class="n">text</span> <span class="n">using</span> <span class="n">remove</span><span class="o">-</span><span class="n">null</span><span class="o">-</span><span class="n">characters</span>
+</pre></div>
+</div>
+<p>Binary:</p>
+<blockquote>
+<div>type binary     to bytea
+type varbinary  to bytea
+type tinyblob   to bytea
+type blob       to bytea
+type mediumblob to bytea
+type longblob   to bytea</div></blockquote>
+<p>Date:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="nb">type</span> <span class="n">datetime</span> <span class="n">when</span> <span class="n">default</span> <span class="s2">&quot;0000-00-00 00:00:00&quot;</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">null</span>
+  <span class="n">to</span> <span class="n">timestamptz</span> <span class="n">drop</span> <span class="ow">not</span> <span class="n">null</span> <span class="n">drop</span> <span class="n">default</span>
+      <span class="n">using</span> <span class="n">zero</span><span class="o">-</span><span class="n">dates</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">null</span>
+
+<span class="nb">type</span> <span class="n">datetime</span> <span class="n">when</span> <span class="n">default</span> <span class="s2">&quot;0000-00-00 00:00:00&quot;</span>
+  <span class="n">to</span> <span class="n">timestamptz</span> <span class="n">drop</span> <span class="n">default</span>
+      <span class="n">using</span> <span class="n">zero</span><span class="o">-</span><span class="n">dates</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">null</span>
+
+<span class="nb">type</span> <span class="n">timestamp</span> <span class="n">when</span> <span class="n">default</span> <span class="s2">&quot;0000-00-00 00:00:00&quot;</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">null</span>
+  <span class="n">to</span> <span class="n">timestamptz</span> <span class="n">drop</span> <span class="ow">not</span> <span class="n">null</span> <span class="n">drop</span> <span class="n">default</span>
+      <span class="n">using</span> <span class="n">zero</span><span class="o">-</span><span class="n">dates</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">null</span>
+
+<span class="nb">type</span> <span class="n">timestamp</span> <span class="n">when</span> <span class="n">default</span> <span class="s2">&quot;0000-00-00 00:00:00&quot;</span>
+  <span class="n">to</span> <span class="n">timestamptz</span> <span class="n">drop</span> <span class="n">default</span>
+      <span class="n">using</span> <span class="n">zero</span><span class="o">-</span><span class="n">dates</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">null</span>
+
+<span class="nb">type</span> <span class="n">date</span> <span class="n">when</span> <span class="n">default</span> <span class="s2">&quot;0000-00-00&quot;</span> <span class="n">to</span> <span class="n">date</span> <span class="n">drop</span> <span class="n">default</span>
+      <span class="n">using</span> <span class="n">zero</span><span class="o">-</span><span class="n">dates</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">null</span>
+
+<span class="nb">type</span> <span class="n">date</span> <span class="n">to</span> <span class="n">date</span>
+<span class="nb">type</span> <span class="n">datetime</span> <span class="n">to</span> <span class="n">timestamptz</span>
+<span class="nb">type</span> <span class="n">timestamp</span> <span class="n">to</span> <span class="n">timestamptz</span>
+<span class="nb">type</span> <span class="n">year</span> <span class="n">to</span> <span class="n">integer</span> <span class="n">drop</span> <span class="n">typemod</span>
+</pre></div>
+</div>
+<p>Geometric:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="nb">type</span> <span class="n">point</span> <span class="n">to</span> <span class="n">point</span> <span class="n">using</span> <span class="n">pgloader</span><span class="o">.</span><span class="n">transforms</span><span class="p">::</span><span class="n">convert</span><span class="o">-</span><span class="n">mysql</span><span class="o">-</span><span class="n">point</span>
+</pre></div>
+</div>
+<p>Enum types are declared inline in MySQL and separately with a <cite>CREATE TYPE</cite>
+command in PostgreSQL, so each column of Enum Type is converted to a type
+named after the table and column names defined with the same labels in the
+same order.</p>
+<p>When the source type definition is not matched in the default casting rules
+nor in the casting rules provided in the command, then the type name with
+the typemod is used.</p>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="../index.html">Documentation overview</a><ul>
+      <li>Previous: <a href="archive.html" title="previous chapter">Loading From an Archive</a></li>
+      <li>Next: <a href="sqlite.html" title="next chapter">Migrating a SQLite database to PostgreSQL</a></li>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="../_sources/ref/mysql.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/ref/sqlite.html

@@ -0,0 +1,268 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Migrating a SQLite database to PostgreSQL &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="../_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="index" title="Index" href="../genindex.html" />
+    <link rel="search" title="Search" href="../search.html" />
+    <link rel="next" title="Migrating a MS SQL Database to PostgreSQL" href="mssql.html" />
+    <link rel="prev" title="Migrating a MySQL Database to PostgreSQL" href="mysql.html" />
+   
+  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="migrating-a-sqlite-database-to-postgresql">
+<h1>Migrating a SQLite database to PostgreSQL<a class="headerlink" href="#migrating-a-sqlite-database-to-postgresql" title="Permalink to this headline">¶</a></h1>
+<p>This command instructs pgloader to load data from a SQLite file. Automatic
+discovery of the schema is supported, including build of the indexes.</p>
+<p>Here’s an example:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">load</span> <span class="n">database</span>
+     <span class="kn">from</span> <span class="nn">sqlite</span><span class="p">:</span><span class="o">///</span><span class="n">Users</span><span class="o">/</span><span class="n">dim</span><span class="o">/</span><span class="n">Downloads</span><span class="o">/</span><span class="n">lastfm_tags</span><span class="o">.</span><span class="n">db</span>
+     <span class="n">into</span> <span class="n">postgresql</span><span class="p">:</span><span class="o">///</span><span class="n">tags</span>
+
+ <span class="k">with</span> <span class="n">include</span> <span class="n">drop</span><span class="p">,</span> <span class="n">create</span> <span class="n">tables</span><span class="p">,</span> <span class="n">create</span> <span class="n">indexes</span><span class="p">,</span> <span class="n">reset</span> <span class="n">sequences</span>
+
+  <span class="nb">set</span> <span class="n">work_mem</span> <span class="n">to</span> <span class="s1">&#39;16MB&#39;</span><span class="p">,</span> <span class="n">maintenance_work_mem</span> <span class="n">to</span> <span class="s1">&#39;512 MB&#39;</span><span class="p">;</span>
+</pre></div>
+</div>
+<p>The <cite>sqlite</cite> command accepts the following clauses and options.</p>
+<div class="section" id="sqlite-database-source-specification-from">
+<h2>SQLite Database Source Specification: FROM<a class="headerlink" href="#sqlite-database-source-specification-from" title="Permalink to this headline">¶</a></h2>
+<p>Path or HTTP URL to a SQLite file, might be a <cite>.zip</cite> file.</p>
+</div>
+<div class="section" id="sqlite-database-migration-options-with">
+<h2>SQLite Database Migration Options: WITH<a class="headerlink" href="#sqlite-database-migration-options-with" title="Permalink to this headline">¶</a></h2>
+<p>When loading from a <cite>SQLite</cite> database, the following options are
+supported:</p>
+<p>When loading from a <cite>SQLite</cite> database, the following options are
+supported, and the default <em>WITH</em> clause is: <em>no truncate</em>, <em>create
+tables</em>, <em>include drop</em>, <em>create indexes</em>, <em>reset sequences</em>, <em>downcase
+identifiers</em>, <em>encoding ‘utf-8’</em>.</p>
+<blockquote>
+<div><ul>
+<li><p class="first"><em>include drop</em></p>
+<p>When this option is listed, pgloader drops all the tables in the target
+PostgreSQL database whose names appear in the SQLite database. This
+option allows for using the same command several times in a row until
+you figure out all the options, starting automatically from a clean
+environment. Please note that <cite>CASCADE</cite> is used to ensure that tables
+are dropped even if there are foreign keys pointing to them. This is
+precisely what <cite>include drop</cite> is intended to do: drop all target tables
+and recreate them.</p>
+<p>Great care needs to be taken when using <cite>include drop</cite>, as it will
+cascade to <em>all</em> objects referencing the target tables, possibly
+including other tables that are not being loaded from the source DB.</p>
+</li>
+<li><p class="first"><em>include no drop</em></p>
+<p>When this option is listed, pgloader will not include any <cite>DROP</cite>
+statement when loading the data.</p>
+</li>
+<li><p class="first"><em>truncate</em></p>
+<p>When this option is listed, pgloader issue the <cite>TRUNCATE</cite> command
+against each PostgreSQL table just before loading data into it.</p>
+</li>
+<li><p class="first"><em>no truncate</em></p>
+<p>When this option is listed, pgloader issues no <cite>TRUNCATE</cite> command.</p>
+</li>
+<li><p class="first"><em>disable triggers</em></p>
+<p>When this option is listed, pgloader issues an <cite>ALTER TABLE … DISABLE
+TRIGGER ALL</cite> command against the PostgreSQL target table before copying
+the data, then the command <cite>ALTER TABLE … ENABLE TRIGGER ALL</cite> once the
+<cite>COPY</cite> is done.</p>
+<p>This option allows loading data into a pre-existing table ignoring
+the <em>foreign key constraints</em> and user defined triggers and may
+result in invalid <em>foreign key constraints</em> once the data is loaded.
+Use with care.</p>
+</li>
+<li><p class="first"><em>create tables</em></p>
+<p>When this option is listed, pgloader creates the table using the meta
+data found in the <cite>SQLite</cite> file, which must contain a list of fields
+with their data type. A standard data type conversion from SQLite to
+PostgreSQL is done.</p>
+</li>
+<li><p class="first"><em>create no tables</em></p>
+<p>When this option is listed, pgloader skips the creation of table before
+loading data, target tables must then already exist.</p>
+<p>Also, when using <em>create no tables</em> pgloader fetches the metadata
+from the current target database and checks type casting, then will
+remove constraints and indexes prior to loading the data and install
+them back again once the loading is done.</p>
+</li>
+<li><p class="first"><em>create indexes</em></p>
+<p>When this option is listed, pgloader gets the definitions of all the
+indexes found in the SQLite database and create the same set of index
+definitions against the PostgreSQL database.</p>
+</li>
+<li><p class="first"><em>create no indexes</em></p>
+<p>When this option is listed, pgloader skips the creating indexes.</p>
+</li>
+<li><p class="first"><em>drop indexes</em></p>
+<p>When this option is listed, pgloader drops the indexes in the target
+database before loading the data, and creates them again at the end
+of the data copy.</p>
+</li>
+<li><p class="first"><em>reset sequences</em></p>
+<p>When this option is listed, at the end of the data loading and after
+the indexes have all been created, pgloader resets all the
+PostgreSQL sequences created to the current maximum value of the
+column they are attached to.</p>
+</li>
+<li><p class="first"><em>reset no sequences</em></p>
+<p>When this option is listed, pgloader skips resetting sequences after the
+load.</p>
+<p>The options <em>schema only</em> and <em>data only</em> have no effects on this
+option.</p>
+</li>
+<li><p class="first"><em>schema only</em></p>
+<p>When this option is listed pgloader will refrain from migrating the data
+over. Note that the schema in this context includes the indexes when the
+option <em>create indexes</em> has been listed.</p>
+</li>
+<li><p class="first"><em>data only</em></p>
+<p>When this option is listed pgloader only issues the <cite>COPY</cite> statements,
+without doing any other processing.</p>
+</li>
+<li><p class="first"><em>encoding</em></p>
+<p>This option allows to control which encoding to parse the SQLite text
+data with. Defaults to UTF-8.</p>
+</li>
+</ul>
+</div></blockquote>
+</div>
+<div class="section" id="sqlite-database-casting-rules">
+<h2>SQLite Database Casting Rules<a class="headerlink" href="#sqlite-database-casting-rules" title="Permalink to this headline">¶</a></h2>
+<p>The command <em>CAST</em> introduces user-defined casting rules.</p>
+<p>The cast clause allows to specify custom casting rules, either to overload
+the default casting rules or to amend them with special cases.</p>
+</div>
+<div class="section" id="sqlite-database-partial-migrations">
+<h2>SQlite Database Partial Migrations<a class="headerlink" href="#sqlite-database-partial-migrations" title="Permalink to this headline">¶</a></h2>
+<div class="section" id="including-only-table-names-like">
+<h3>INCLUDING ONLY TABLE NAMES LIKE<a class="headerlink" href="#including-only-table-names-like" title="Permalink to this headline">¶</a></h3>
+<p>Introduce a comma separated list of table name patterns used to limit the
+tables to migrate to a sublist.</p>
+<p>Example:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">including</span> <span class="n">only</span> <span class="n">table</span> <span class="n">names</span> <span class="n">like</span> <span class="s1">&#39;Invoice%&#39;</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="excluding-table-names-like">
+<h3>EXCLUDING TABLE NAMES LIKE<a class="headerlink" href="#excluding-table-names-like" title="Permalink to this headline">¶</a></h3>
+<p>Introduce a comma separated list of table name patterns used to exclude
+table names from the migration. This filter only applies to the result of
+the <em>INCLUDING</em> filter.</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">excluding</span> <span class="n">table</span> <span class="n">names</span> <span class="n">like</span> <span class="s1">&#39;appointments&#39;</span>
+</pre></div>
+</div>
+</div>
+</div>
+<div class="section" id="default-sqlite-casting-rules">
+<h2>Default SQLite Casting Rules<a class="headerlink" href="#default-sqlite-casting-rules" title="Permalink to this headline">¶</a></h2>
+<p>When migrating from SQLite the following Casting Rules are provided:</p>
+<p>Numbers:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="nb">type</span> <span class="n">tinyint</span> <span class="n">to</span> <span class="n">smallint</span> <span class="n">using</span> <span class="n">integer</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">string</span>
+<span class="nb">type</span> <span class="n">integer</span> <span class="n">to</span> <span class="n">bigint</span>   <span class="n">using</span> <span class="n">integer</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">string</span>
+
+<span class="nb">type</span> <span class="nb">float</span> <span class="n">to</span> <span class="nb">float</span>   <span class="n">using</span> <span class="nb">float</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">string</span>
+<span class="nb">type</span> <span class="n">real</span> <span class="n">to</span> <span class="n">real</span>     <span class="n">using</span> <span class="nb">float</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">string</span>
+<span class="nb">type</span> <span class="n">double</span> <span class="n">to</span> <span class="n">double</span> <span class="n">precision</span>     <span class="n">using</span> <span class="nb">float</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">string</span>
+<span class="nb">type</span> <span class="n">numeric</span> <span class="n">to</span> <span class="n">numeric</span>     <span class="n">using</span> <span class="nb">float</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">string</span>
+</pre></div>
+</div>
+<p>Texts:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="nb">type</span> <span class="n">character</span>  <span class="n">to</span> <span class="n">text</span> <span class="n">drop</span> <span class="n">typemod</span>
+<span class="nb">type</span> <span class="n">varchar</span>    <span class="n">to</span> <span class="n">text</span> <span class="n">drop</span> <span class="n">typemod</span>
+<span class="nb">type</span> <span class="n">nvarchar</span>   <span class="n">to</span> <span class="n">text</span> <span class="n">drop</span> <span class="n">typemod</span>
+<span class="nb">type</span> <span class="n">char</span>       <span class="n">to</span> <span class="n">text</span> <span class="n">drop</span> <span class="n">typemod</span>
+<span class="nb">type</span> <span class="n">nchar</span>      <span class="n">to</span> <span class="n">text</span> <span class="n">drop</span> <span class="n">typemod</span>
+<span class="nb">type</span> <span class="n">nvarchar</span>   <span class="n">to</span> <span class="n">text</span> <span class="n">drop</span> <span class="n">typemod</span>
+<span class="nb">type</span> <span class="n">clob</span>       <span class="n">to</span> <span class="n">text</span> <span class="n">drop</span> <span class="n">typemod</span>
+</pre></div>
+</div>
+<p>Binary:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="nb">type</span> <span class="n">blob</span>       <span class="n">to</span> <span class="n">bytea</span>
+</pre></div>
+</div>
+<p>Date:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="nb">type</span> <span class="n">datetime</span>    <span class="n">to</span> <span class="n">timestamptz</span> <span class="n">using</span> <span class="n">sqlite</span><span class="o">-</span><span class="n">timestamp</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">timestamp</span>
+<span class="nb">type</span> <span class="n">timestamp</span>   <span class="n">to</span> <span class="n">timestamptz</span> <span class="n">using</span> <span class="n">sqlite</span><span class="o">-</span><span class="n">timestamp</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">timestamp</span>
+<span class="nb">type</span> <span class="n">timestamptz</span> <span class="n">to</span> <span class="n">timestamptz</span> <span class="n">using</span> <span class="n">sqlite</span><span class="o">-</span><span class="n">timestamp</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">timestamp</span>
+</pre></div>
+</div>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="../index.html">Documentation overview</a><ul>
+      <li>Previous: <a href="mysql.html" title="previous chapter">Migrating a MySQL Database to PostgreSQL</a></li>
+      <li>Next: <a href="mssql.html" title="next chapter">Migrating a MS SQL Database to PostgreSQL</a></li>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="../_sources/ref/sqlite.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/ref/transforms.html

@@ -0,0 +1,214 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Transformation Functions &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="../_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="index" title="Index" href="../genindex.html" />
+    <link rel="search" title="Search" href="../search.html" />
+    <link rel="next" title="Reporting Bugs" href="../bugreport.html" />
+    <link rel="prev" title="Migrating a MS SQL Database to PostgreSQL" href="mssql.html" />
+   
+  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="transformation-functions">
+<h1>Transformation Functions<a class="headerlink" href="#transformation-functions" title="Permalink to this headline">¶</a></h1>
+<p>Some data types are implemented in a different enough way that a
+transformation function is necessary. This function must be written in
+<cite>Common lisp</cite> and is searched in the <cite>pgloader.transforms</cite> package.</p>
+<p>Some default transformation function are provided with pgloader, and you can
+use the <cite>–load</cite> command line option to load and compile your own lisp file
+into pgloader at runtime. For your functions to be found, remember to begin
+your lisp file with the following form:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="p">(</span><span class="ow">in</span><span class="o">-</span><span class="n">package</span> <span class="c1">#:pgloader.transforms)</span>
+</pre></div>
+</div>
+<p>The provided transformation functions are:</p>
+<blockquote>
+<div><ul>
+<li><p class="first"><em>zero-dates-to-null</em></p>
+<p>When the input date is all zeroes, return <cite>nil</cite>, which gets loaded as a
+PostgreSQL <cite>NULL</cite> value.</p>
+</li>
+<li><p class="first"><em>date-with-no-separator</em></p>
+<p>Applies <em>zero-dates-to-null</em> then transform the given date into a format
+that PostgreSQL will actually process:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">In</span><span class="p">:</span>  <span class="s2">&quot;20041002152952&quot;</span>
+<span class="n">Out</span><span class="p">:</span> <span class="s2">&quot;2004-10-02 15:29:52&quot;</span>
+</pre></div>
+</div>
+</li>
+<li><p class="first"><em>time-with-no-separator</em></p>
+<p>Transform the given time into a format that PostgreSQL will actually
+process:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">In</span><span class="p">:</span>  <span class="s2">&quot;08231560&quot;</span>
+<span class="n">Out</span><span class="p">:</span> <span class="s2">&quot;08:23:15.60&quot;</span>
+</pre></div>
+</div>
+</li>
+<li><p class="first"><em>tinyint-to-boolean</em></p>
+<p>As MySQL lacks a proper boolean type, <em>tinyint</em> is often used to
+implement that. This function transforms <cite>0</cite> to <cite>‘false’</cite> and anything
+else to <cite>‘true</cite>’.</p>
+</li>
+<li><p class="first"><em>bits-to-boolean</em></p>
+<p>As MySQL lacks a proper boolean type, <em>BIT</em> is often used to implement
+that. This function transforms 1-bit bit vectors from <cite>0</cite> to <cite>f</cite> and any
+other value to <cite>t</cite>..</p>
+</li>
+<li><p class="first"><em>int-to-ip</em></p>
+<p>Convert an integer into a dotted representation of an ip4.</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">In</span><span class="p">:</span>  <span class="mi">18435761</span>
+<span class="n">Out</span><span class="p">:</span> <span class="s2">&quot;1.25.78.177&quot;</span>
+</pre></div>
+</div>
+</li>
+<li><p class="first"><em>ip-range</em></p>
+<p>Converts a couple of integers given as strings into a range of ip4.</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">In</span><span class="p">:</span>  <span class="s2">&quot;16825344&quot;</span> <span class="s2">&quot;16825599&quot;</span>
+<span class="n">Out</span><span class="p">:</span> <span class="s2">&quot;1.0.188.0-1.0.188.255&quot;</span>
+</pre></div>
+</div>
+</li>
+<li><p class="first"><em>convert-mysql-point</em></p>
+<p>Converts from the <cite>astext</cite> representation of points in MySQL to the
+PostgreSQL representation.</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">In</span><span class="p">:</span>  <span class="s2">&quot;POINT(48.5513589 7.6926827)&quot;</span>
+<span class="n">Out</span><span class="p">:</span> <span class="s2">&quot;(48.5513589,7.6926827)&quot;</span>
+</pre></div>
+</div>
+</li>
+<li><p class="first"><em>integer-to-string</em></p>
+<p>Converts a integer string or a Common Lisp integer into a string
+suitable for a PostgreSQL integer. Takes care of quoted integers.</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">In</span><span class="p">:</span>  <span class="s2">&quot;</span><span class="se">\&quot;</span><span class="s2">0</span><span class="se">\&quot;</span><span class="s2">&quot;</span>
+<span class="n">Out</span><span class="p">:</span> <span class="s2">&quot;0&quot;</span>
+</pre></div>
+</div>
+</li>
+<li><p class="first"><em>float-to-string</em></p>
+<p>Converts a Common Lisp float into a string suitable for a PostgreSQL float:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">In</span><span class="p">:</span>  <span class="mf">100.0</span><span class="n">d0</span>
+<span class="n">Out</span><span class="p">:</span> <span class="s2">&quot;100.0&quot;</span>
+</pre></div>
+</div>
+</li>
+<li><p class="first"><em>set-to-enum-array</em></p>
+<p>Converts a string representing a MySQL SET into a PostgreSQL Array of
+Enum values from the set.</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">In</span><span class="p">:</span> <span class="s2">&quot;foo,bar&quot;</span>
+<span class="n">Out</span><span class="p">:</span> <span class="s2">&quot;{foo,bar}&quot;</span>
+</pre></div>
+</div>
+</li>
+<li><p class="first"><em>empty-string-to-null</em></p>
+<p>Convert an empty string to a null.</p>
+</li>
+<li><p class="first"><em>right-trim</em></p>
+<p>Remove whitespace at end of string.</p>
+</li>
+<li><p class="first"><em>remove-null-characters</em></p>
+<p>Remove <cite>NUL</cite> characters (<cite>0x0</cite>) from given strings.</p>
+</li>
+<li><p class="first"><em>byte-vector-to-bytea</em></p>
+<p>Transform a simple array of unsigned bytes to the PostgreSQL bytea Hex
+Format representation as documented at
+<a class="reference external" href="http://www.postgresql.org/docs/9.3/interactive/datatype-binary.html">http://www.postgresql.org/docs/9.3/interactive/datatype-binary.html</a></p>
+</li>
+<li><p class="first"><em>sqlite-timestamp-to-timestamp</em></p>
+<p>SQLite type system is quite interesting, so cope with it here to produce
+timestamp literals as expected by PostgreSQL. That covers year only on 4
+digits, 0 dates to null, and proper date strings.</p>
+</li>
+<li><p class="first"><em>sql-server-uniqueidentifier-to-uuid</em></p>
+<p>The SQL Server driver receives data fo type uniqueidentifier as byte
+vector that we then need to convert to an UUID string for PostgreSQL
+COPY input format to process.</p>
+</li>
+<li><p class="first"><em>unix-timestamp-to-timestamptz</em></p>
+<p>Converts a unix timestamp (number of seconds elapsed since beginning of
+1970) into a proper PostgreSQL timestamp format.</p>
+</li>
+<li><p class="first"><em>varbinary-to-string</em></p>
+<p>Converts binary encoded string (such as a MySQL <cite>varbinary</cite> entry) to a
+decoded text, using the table’s encoding that may be overloaded with the
+<em>DECODING TABLE NAMES MATCHING</em> clause.</p>
+</li>
+</ul>
+</div></blockquote>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="../index.html">Documentation overview</a><ul>
+      <li>Previous: <a href="mssql.html" title="previous chapter">Migrating a MS SQL Database to PostgreSQL</a></li>
+      <li>Next: <a href="../bugreport.html" title="next chapter">Reporting Bugs</a></li>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="../_sources/ref/transforms.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/search.html

@@ -0,0 +1,101 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Search &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    './',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="_static/jquery.js"></script>
+    <script type="text/javascript" src="_static/underscore.js"></script>
+    <script type="text/javascript" src="_static/doctools.js"></script>
+    <script type="text/javascript" src="_static/searchtools.js"></script>
+    <link rel="index" title="Index" href="genindex.html" />
+    <link rel="search" title="Search" href="#" />
+  <script type="text/javascript">
+    jQuery(function() { Search.loadIndex("searchindex.js"); });
+  </script>
+  
+  <script type="text/javascript" id="searchindexloader"></script>
+  
+   
+  <link rel="stylesheet" href="_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <h1 id="search-documentation">Search</h1>
+  <div id="fallback" class="admonition warning">
+  <script type="text/javascript">$('#fallback').hide();</script>
+  <p>
+    Please activate JavaScript to enable the search
+    functionality.
+  </p>
+  </div>
+  <p>
+    From here you can search these documents. Enter your search
+    words into the box below and click "search". Note that the search
+    function will automatically search for all of the words. Pages
+    containing fewer words won't appear in the result list.
+  </p>
+  <form action="" method="get">
+    <input type="text" name="q" value="" />
+    <input type="submit" value="search" />
+    <span id="search-progress" style="padding-left: 10px"></span>
+  </form>
+  
+  <div id="search-results">
+  
+  </div>
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="index.html">Documentation overview</a><ul>
+  </ul></li>
+</ul>
+</div>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/tutorial/csv.html

@@ -0,0 +1,169 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Loading CSV Data with pgloader &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="../_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="index" title="Index" href="../genindex.html" />
+    <link rel="search" title="Search" href="../search.html" />
+   
+  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="loading-csv-data-with-pgloader">
+<h1>Loading CSV Data with pgloader<a class="headerlink" href="#loading-csv-data-with-pgloader" title="Permalink to this headline">¶</a></h1>
+<p>CSV means <em>comma separated values</em> and is often found with quite varying
+specifications. pgloader allows you to describe those specs in its command.</p>
+<div class="section" id="the-command">
+<h2>The Command<a class="headerlink" href="#the-command" title="Permalink to this headline">¶</a></h2>
+<p>To load data with [pgloader](<a class="reference external" href="http://pgloader.io/">http://pgloader.io/</a>) you need to define in a
+<em>command</em> the operations in some details. Here’s our example for loading CSV
+data:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>LOAD CSV
+     FROM &#39;path/to/file.csv&#39; (x, y, a, b, c, d)
+     INTO postgresql:///pgloader?csv (a, b, d, c)
+
+     WITH truncate,
+          skip header = 1,
+          fields optionally enclosed by &#39;&quot;&#39;,
+          fields escaped by double-quote,
+          fields terminated by &#39;,&#39;
+
+      SET client_encoding to &#39;latin1&#39;,
+          work_mem to &#39;12MB&#39;,
+          standard_conforming_strings to &#39;on&#39;
+
+   BEFORE LOAD DO
+    $$ drop table if exists csv; $$,
+    $$ create table csv (
+        a bigint,
+        b bigint,
+        c char(2),
+        d text
+       );
+  $$;
+</pre></div>
+</div>
+</div>
+<div class="section" id="the-data">
+<h2>The Data<a class="headerlink" href="#the-data" title="Permalink to this headline">¶</a></h2>
+<p>This command allows loading the following CSV file content:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>Header, with a © sign
+&quot;2.6.190.56&quot;,&quot;2.6.190.63&quot;,&quot;33996344&quot;,&quot;33996351&quot;,&quot;GB&quot;,&quot;United Kingdom&quot;
+&quot;3.0.0.0&quot;,&quot;4.17.135.31&quot;,&quot;50331648&quot;,&quot;68257567&quot;,&quot;US&quot;,&quot;United States&quot;
+&quot;4.17.135.32&quot;,&quot;4.17.135.63&quot;,&quot;68257568&quot;,&quot;68257599&quot;,&quot;CA&quot;,&quot;Canada&quot;
+&quot;4.17.135.64&quot;,&quot;4.17.142.255&quot;,&quot;68257600&quot;,&quot;68259583&quot;,&quot;US&quot;,&quot;United States&quot;
+&quot;4.17.143.0&quot;,&quot;4.17.143.15&quot;,&quot;68259584&quot;,&quot;68259599&quot;,&quot;CA&quot;,&quot;Canada&quot;
+&quot;4.17.143.16&quot;,&quot;4.18.32.71&quot;,&quot;68259600&quot;,&quot;68296775&quot;,&quot;US&quot;,&quot;United States&quot;
+</pre></div>
+</div>
+</div>
+<div class="section" id="loading-the-data">
+<h2>Loading the data<a class="headerlink" href="#loading-the-data" title="Permalink to this headline">¶</a></h2>
+<p>Here’s how to start loading the data. Note that the ouput here has been
+edited so as to facilitate its browsing online:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>$ pgloader csv.load
+... LOG Starting pgloader, log system is ready.
+... LOG Parsing commands from file &quot;/Users/dim/dev/pgloader/test/csv.load&quot;
+
+       table name       read   imported     errors            time
+-----------------  ---------  ---------  ---------  --------------
+      before load          2          2          0          0.039s
+-----------------  ---------  ---------  ---------  --------------
+              csv          6          6          0          0.019s
+-----------------  ---------  ---------  ---------  --------------
+Total import time          6          6          0          0.058s
+</pre></div>
+</div>
+</div>
+<div class="section" id="the-result">
+<h2>The result<a class="headerlink" href="#the-result" title="Permalink to this headline">¶</a></h2>
+<p>As you can see, the command described above is filtering the input and only
+importing some of the columns from the example data file. Here’s what gets
+loaded in the PostgreSQL database:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">pgloader</span><span class="c1"># table csv;</span>
+    <span class="n">a</span>     <span class="o">|</span>    <span class="n">b</span>     <span class="o">|</span> <span class="n">c</span>  <span class="o">|</span>       <span class="n">d</span>
+<span class="o">----------+----------+----+----------------</span>
+ <span class="mi">33996344</span> <span class="o">|</span> <span class="mi">33996351</span> <span class="o">|</span> <span class="n">GB</span> <span class="o">|</span> <span class="n">United</span> <span class="n">Kingdom</span>
+ <span class="mi">50331648</span> <span class="o">|</span> <span class="mi">68257567</span> <span class="o">|</span> <span class="n">US</span> <span class="o">|</span> <span class="n">United</span> <span class="n">States</span>
+ <span class="mi">68257568</span> <span class="o">|</span> <span class="mi">68257599</span> <span class="o">|</span> <span class="n">CA</span> <span class="o">|</span> <span class="n">Canada</span>
+ <span class="mi">68257600</span> <span class="o">|</span> <span class="mi">68259583</span> <span class="o">|</span> <span class="n">US</span> <span class="o">|</span> <span class="n">United</span> <span class="n">States</span>
+ <span class="mi">68259584</span> <span class="o">|</span> <span class="mi">68259599</span> <span class="o">|</span> <span class="n">CA</span> <span class="o">|</span> <span class="n">Canada</span>
+ <span class="mi">68259600</span> <span class="o">|</span> <span class="mi">68296775</span> <span class="o">|</span> <span class="n">US</span> <span class="o">|</span> <span class="n">United</span> <span class="n">States</span>
+<span class="p">(</span><span class="mi">6</span> <span class="n">rows</span><span class="p">)</span>
+</pre></div>
+</div>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="../index.html">Documentation overview</a><ul>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="../_sources/tutorial/csv.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/tutorial/dBase.html

@@ -0,0 +1,134 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Loading dBase files with pgloader &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="../_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="index" title="Index" href="../genindex.html" />
+    <link rel="search" title="Search" href="../search.html" />
+   
+  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="loading-dbase-files-with-pgloader">
+<h1>Loading dBase files with pgloader<a class="headerlink" href="#loading-dbase-files-with-pgloader" title="Permalink to this headline">¶</a></h1>
+<p>The dBase format is still in use in some places as modern tools such as
+<em>Filemaker</em> and <em>Excel</em> offer some level of support for it. Speaking of
+support in modern tools, pgloader is right there on the list too!</p>
+<div class="section" id="the-command">
+<h2>The Command<a class="headerlink" href="#the-command" title="Permalink to this headline">¶</a></h2>
+<p>To load data with [pgloader](<a class="reference external" href="http://pgloader.io/">http://pgloader.io/</a>) you need to define in a
+<em>command</em> the operations in some details. Here’s our example for loading a
+dBase file, using a file provided by the french administration.</p>
+<p>You can find more files from them at the <a class="reference external" href="http://www.insee.fr/fr/methodes/nomenclatures/cog/telechargement.asp">Insee</a>
+website.</p>
+<p>Here’s our command:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">LOAD</span> <span class="n">DBF</span>
+    <span class="n">FROM</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">www</span><span class="o">.</span><span class="n">insee</span><span class="o">.</span><span class="n">fr</span><span class="o">/</span><span class="n">fr</span><span class="o">/</span><span class="n">methodes</span><span class="o">/</span><span class="n">nomenclatures</span><span class="o">/</span><span class="n">cog</span><span class="o">/</span><span class="n">telechargement</span><span class="o">/</span><span class="mi">2013</span><span class="o">/</span><span class="n">dbf</span><span class="o">/</span><span class="n">historiq2013</span><span class="o">.</span><span class="n">zip</span>
+    <span class="n">INTO</span> <span class="n">postgresql</span><span class="p">:</span><span class="o">///</span><span class="n">pgloader</span>
+    <span class="n">WITH</span> <span class="n">truncate</span><span class="p">,</span> <span class="n">create</span> <span class="n">table</span>
+     <span class="n">SET</span> <span class="n">client_encoding</span> <span class="n">TO</span> <span class="s1">&#39;latin1&#39;</span><span class="p">;</span>
+</pre></div>
+</div>
+<p>Note that here pgloader will benefit from the meta-data information found in
+the dBase file to create a PostgreSQL table capable of hosting the data as
+described, then load the data.</p>
+</div>
+<div class="section" id="loading-the-data">
+<h2>Loading the data<a class="headerlink" href="#loading-the-data" title="Permalink to this headline">¶</a></h2>
+<p>Let’s start the <cite>pgloader</cite> command with our <cite>dbf-zip.load</cite> command file:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>$ pgloader dbf-zip.load
+... LOG Starting pgloader, log system is ready.
+... LOG Parsing commands from file &quot;/Users/dim/dev/pgloader/test/dbf-zip.load&quot;
+... LOG Fetching &#39;http://www.insee.fr/fr/methodes/nomenclatures/cog/telechargement/2013/dbf/historiq2013.zip&#39;
+... LOG Extracting files from archive &#39;//private/var/folders/w7/9n8v8pw54t1gngfff0lj16040000gn/T/pgloader//historiq2013.zip&#39;
+
+       table name       read   imported     errors            time
+-----------------  ---------  ---------  ---------  --------------
+         download          0          0          0          0.167s
+          extract          0          0          0          1.010s
+ create, truncate          0          0          0          0.071s
+-----------------  ---------  ---------  ---------  --------------
+     historiq2013       9181       9181          0          0.658s
+-----------------  ---------  ---------  ---------  --------------
+Total import time       9181       9181          0          1.906s
+</pre></div>
+</div>
+<p>We can see that <a class="reference external" href="http://pgloader.io">pgloader</a> did download the file from
+its HTTP URL location then <em>unziped</em> it before the loading itself.</p>
+<p>Note that the output of the command has been edited to facilitate its
+browsing online.</p>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="../index.html">Documentation overview</a><ul>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="../_sources/tutorial/dBase.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/tutorial/fixed.html

@@ -0,0 +1,176 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Loading Fixed Width Data File with pgloader &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="../_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="index" title="Index" href="../genindex.html" />
+    <link rel="search" title="Search" href="../search.html" />
+   
+  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="loading-fixed-width-data-file-with-pgloader">
+<h1>Loading Fixed Width Data File with pgloader<a class="headerlink" href="#loading-fixed-width-data-file-with-pgloader" title="Permalink to this headline">¶</a></h1>
+<p>Some data providers still use a format where each column is specified with a
+starting index position and a given length. Usually the columns are
+blank-padded when the data is shorter than the full reserved range.</p>
+<div class="section" id="the-command">
+<h2>The Command<a class="headerlink" href="#the-command" title="Permalink to this headline">¶</a></h2>
+<p>To load data with [pgloader](<a class="reference external" href="http://pgloader.io/">http://pgloader.io/</a>) you need to define in a
+<em>command</em> the operations in some details. Here’s our example for loading
+Fixed Width Data, using a file provided by the US census.</p>
+<p>You can find more files from them at the
+[Census 2000 Gazetteer Files](<a class="reference external" href="http://www.census.gov/geo/maps-data/data/gazetteer2000.html">http://www.census.gov/geo/maps-data/data/gazetteer2000.html</a>).</p>
+<p>Here’s our command:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>LOAD ARCHIVE
+   FROM http://www.census.gov/geo/maps-data/data/docs/gazetteer/places2k.zip
+   INTO postgresql:///pgloader
+
+   BEFORE LOAD DO
+     $$ drop table if exists places; $$,
+     $$ create table places
+       (
+          usps      char(2)  not null,
+          fips      char(2)  not null,
+          fips_code char(5),
+          loc_name  varchar(64)
+       );
+     $$
+
+   LOAD FIXED
+        FROM FILENAME MATCHING ~/places2k.txt/
+             WITH ENCODING latin1
+             (
+                usps           from   0 for  2,
+                fips           from   2 for  2,
+                fips_code      from   4 for  5,
+                &quot;LocationName&quot; from   9 for 64 [trim right whitespace],
+                p              from  73 for  9,
+                h              from  82 for  9,
+                land           from  91 for 14,
+                water          from 105 for 14,
+                ldm            from 119 for 14,
+                wtm            from 131 for 14,
+                lat            from 143 for 10,
+                long           from 153 for 11
+             )
+        INTO postgresql:///pgloader?places
+             (
+            usps, fips, fips_code, &quot;LocationName&quot;
+             );
+</pre></div>
+</div>
+</div>
+<div class="section" id="the-data">
+<h2>The Data<a class="headerlink" href="#the-data" title="Permalink to this headline">¶</a></h2>
+<p>This command allows loading the following file content, where we are only
+showing the first couple of lines:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">AL0100124Abbeville</span> <span class="n">city</span>                                                       <span class="mi">2987</span>     <span class="mi">1353</span>      <span class="mi">40301945</span>        <span class="mi">120383</span>   <span class="mf">15.560669</span>    <span class="mf">0.046480</span> <span class="mf">31.566367</span> <span class="o">-</span><span class="mf">85.251300</span>
+<span class="n">AL0100460Adamsville</span> <span class="n">city</span>                                                      <span class="mi">4965</span>     <span class="mi">2042</span>      <span class="mi">50779330</span>         <span class="mi">14126</span>   <span class="mf">19.606010</span>    <span class="mf">0.005454</span> <span class="mf">33.590411</span> <span class="o">-</span><span class="mf">86.949166</span>
+<span class="n">AL0100484Addison</span> <span class="n">town</span>                                                          <span class="mi">723</span>      <span class="mi">339</span>       <span class="mi">9101325</span>             <span class="mi">0</span>    <span class="mf">3.514041</span>    <span class="mf">0.000000</span> <span class="mf">34.200042</span> <span class="o">-</span><span class="mf">87.177851</span>
+<span class="n">AL0100676Akron</span> <span class="n">town</span>                                                            <span class="mi">521</span>      <span class="mi">239</span>       <span class="mi">1436797</span>             <span class="mi">0</span>    <span class="mf">0.554750</span>    <span class="mf">0.000000</span> <span class="mf">32.876425</span> <span class="o">-</span><span class="mf">87.740978</span>
+<span class="n">AL0100820Alabaster</span> <span class="n">city</span>                                                      <span class="mi">22619</span>     <span class="mi">8594</span>      <span class="mi">53023800</span>        <span class="mi">141711</span>   <span class="mf">20.472605</span>    <span class="mf">0.054715</span> <span class="mf">33.231162</span> <span class="o">-</span><span class="mf">86.823829</span>
+<span class="n">AL0100988Albertville</span> <span class="n">city</span>                                                    <span class="mi">17247</span>     <span class="mi">7090</span>      <span class="mi">67212867</span>        <span class="mi">258738</span>   <span class="mf">25.951034</span>    <span class="mf">0.099899</span> <span class="mf">34.265362</span> <span class="o">-</span><span class="mf">86.211261</span>
+<span class="n">AL0101132Alexander</span> <span class="n">City</span> <span class="n">city</span>                                                 <span class="mi">15008</span>     <span class="mi">6855</span>     <span class="mi">100534344</span>        <span class="mi">433413</span>   <span class="mf">38.816529</span>    <span class="mf">0.167342</span> <span class="mf">32.933157</span> <span class="o">-</span><span class="mf">85.936008</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="loading-the-data">
+<h2>Loading the data<a class="headerlink" href="#loading-the-data" title="Permalink to this headline">¶</a></h2>
+<p>Let’s start the <cite>pgloader</cite> command with our <cite>census-places.load</cite> command file:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>$ pgloader census-places.load
+... LOG Starting pgloader, log system is ready.
+... LOG Parsing commands from file &quot;/Users/dim/dev/pgloader/test/census-places.load&quot;
+... LOG Fetching &#39;http://www.census.gov/geo/maps-data/data/docs/gazetteer/places2k.zip&#39;
+... LOG Extracting files from archive &#39;//private/var/folders/w7/9n8v8pw54t1gngfff0lj16040000gn/T/pgloader//places2k.zip&#39;
+
+       table name       read   imported     errors            time
+-----------------  ---------  ---------  ---------  --------------
+         download          0          0          0          1.494s
+          extract          0          0          0          1.013s
+      before load          2          2          0          0.013s
+-----------------  ---------  ---------  ---------  --------------
+           places      25375      25375          0          0.499s
+-----------------  ---------  ---------  ---------  --------------
+Total import time      25375      25375          0          3.019s
+</pre></div>
+</div>
+<p>We can see that pgloader did download the file from its HTTP URL location
+then <em>unziped</em> it before the loading itself.</p>
+<p>Note that the output of the command has been edited to facilitate its
+browsing online.</p>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="../index.html">Documentation overview</a><ul>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="../_sources/tutorial/fixed.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/tutorial/geolite.html

@@ -0,0 +1,239 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Loading MaxMind Geolite Data with pgloader &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="../_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="index" title="Index" href="../genindex.html" />
+    <link rel="search" title="Search" href="../search.html" />
+   
+  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="loading-maxmind-geolite-data-with-pgloader">
+<h1>Loading MaxMind Geolite Data with pgloader<a class="headerlink" href="#loading-maxmind-geolite-data-with-pgloader" title="Permalink to this headline">¶</a></h1>
+<p><a class="reference external" href="http://www.maxmind.com/">MaxMind</a> provides a free dataset for
+geolocation, which is quite popular. Using pgloader you can download the
+lastest version of it, extract the CSV files from the archive and load their
+content into your database directly.</p>
+<div class="section" id="the-command">
+<h2>The Command<a class="headerlink" href="#the-command" title="Permalink to this headline">¶</a></h2>
+<p>To load data with pgloader you need to define in a <em>command</em> the operations
+in some details. Here’s our example for loading the Geolite data:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>/*
+ * Loading from a ZIP archive containing CSV files. The full test can be
+ * done with using the archive found at
+ * http://geolite.maxmind.com/download/geoip/database/GeoLiteCity_CSV/GeoLiteCity-latest.zip
+ *
+ * And a very light version of this data set is found at
+ * http://pgsql.tapoueh.org/temp/foo.zip for quick testing.
+ */
+
+LOAD ARCHIVE
+   FROM http://geolite.maxmind.com/download/geoip/database/GeoLiteCity_CSV/GeoLiteCity-latest.zip
+   INTO postgresql:///ip4r
+
+   BEFORE LOAD DO
+     $$ create extension if not exists ip4r; $$,
+     $$ create schema if not exists geolite; $$,
+     $$ create table if not exists geolite.location
+       (
+          locid      integer primary key,
+          country    text,
+          region     text,
+          city       text,
+          postalcode text,
+          location   point,
+          metrocode  text,
+          areacode   text
+       );
+     $$,
+     $$ create table if not exists geolite.blocks
+       (
+          iprange    ip4r,
+          locid      integer
+       );
+     $$,
+     $$ drop index if exists geolite.blocks_ip4r_idx; $$,
+     $$ truncate table geolite.blocks, geolite.location cascade; $$
+
+   LOAD CSV
+        FROM FILENAME MATCHING ~/GeoLiteCity-Location.csv/
+             WITH ENCODING iso-8859-1
+             (
+                locId,
+                country,
+                region     null if blanks,
+                city       null if blanks,
+                postalCode null if blanks,
+                latitude,
+                longitude,
+                metroCode  null if blanks,
+                areaCode   null if blanks
+             )
+        INTO postgresql:///ip4r?geolite.location
+             (
+                locid,country,region,city,postalCode,
+                location point using (format nil &quot;(~a,~a)&quot; longitude latitude),
+                metroCode,areaCode
+             )
+        WITH skip header = 2,
+             fields optionally enclosed by &#39;&quot;&#39;,
+             fields escaped by double-quote,
+             fields terminated by &#39;,&#39;
+
+  AND LOAD CSV
+        FROM FILENAME MATCHING ~/GeoLiteCity-Blocks.csv/
+             WITH ENCODING iso-8859-1
+             (
+                startIpNum, endIpNum, locId
+             )
+        INTO postgresql:///ip4r?geolite.blocks
+             (
+                iprange ip4r using (ip-range startIpNum endIpNum),
+                locId
+             )
+        WITH skip header = 2,
+             fields optionally enclosed by &#39;&quot;&#39;,
+             fields escaped by double-quote,
+             fields terminated by &#39;,&#39;
+
+   FINALLY DO
+     $$ create index blocks_ip4r_idx on geolite.blocks using gist(iprange); $$;
+</pre></div>
+</div>
+<p>Note that while the <em>Geolite</em> data is using a pair of integers (<em>start</em>,
+<em>end</em>) to represent <em>ipv4</em> data, we use the very poweful <a class="reference external" href="https://github.com/RhodiumToad/ip4r">ip4r</a> PostgreSQL Extension instead.</p>
+<p>The transformation from a pair of integers into an IP is done dynamically by
+the pgloader process.</p>
+<p>Also, the location is given as a pair of <em>float</em> columns for the <em>longitude</em>
+and the <em>latitude</em> where PostgreSQL offers the
+<a class="reference external" href="http://www.postgresql.org/docs/9.3/interactive/functions-geometry.html">point</a>
+datatype, so the pgloader command here will actually transform the data on
+the fly to use the appropriate data type and its input representation.</p>
+</div>
+<div class="section" id="loading-the-data">
+<h2>Loading the data<a class="headerlink" href="#loading-the-data" title="Permalink to this headline">¶</a></h2>
+<p>Here’s how to start loading the data. Note that the ouput here has been
+edited so as to facilitate its browsing online:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>$ pgloader archive.load
+... LOG Starting pgloader, log system is ready.
+... LOG Parsing commands from file &quot;/Users/dim/dev/pgloader/test/archive.load&quot;
+... LOG Fetching &#39;http://geolite.maxmind.com/download/geoip/database/GeoLiteCity_CSV/GeoLiteCity-latest.zip&#39;
+... LOG Extracting files from archive &#39;//private/var/folders/w7/9n8v8pw54t1gngfff0lj16040000gn/T/pgloader//GeoLiteCity-latest.zip&#39;
+
+       table name       read   imported     errors            time
+-----------------  ---------  ---------  ---------  --------------
+         download          0          0          0         11.592s
+          extract          0          0          0          1.012s
+      before load          6          6          0          0.019s
+-----------------  ---------  ---------  ---------  --------------
+ geolite.location     470387     470387          0          7.743s
+   geolite.blocks    1903155    1903155          0         16.332s
+-----------------  ---------  ---------  ---------  --------------
+          finally          1          1          0         31.692s
+-----------------  ---------  ---------  ---------  --------------
+Total import time    2373542    2373542          0        1m8.390s
+</pre></div>
+</div>
+<p>The timing of course includes the transformation of the <em>1.9 million</em> pairs
+of integer into a single <em>ipv4 range</em> each. The <em>finally</em> step consists of
+creating the <em>GiST</em> specialized index as given in the main command:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">CREATE</span> <span class="n">INDEX</span> <span class="n">blocks_ip4r_idx</span> <span class="n">ON</span> <span class="n">geolite</span><span class="o">.</span><span class="n">blocks</span> <span class="n">USING</span> <span class="n">gist</span><span class="p">(</span><span class="n">iprange</span><span class="p">);</span>
+</pre></div>
+</div>
+<p>That index will then be used to speed up queries wanting to find which
+recorded geolocation contains a specific IP address:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">ip4r</span><span class="o">&gt;</span> <span class="n">select</span> <span class="o">*</span>
+        <span class="kn">from</span>      <span class="nn">geolite.location</span> <span class="n">l</span>
+             <span class="n">join</span> <span class="n">geolite</span><span class="o">.</span><span class="n">blocks</span> <span class="n">b</span> <span class="n">using</span><span class="p">(</span><span class="n">locid</span><span class="p">)</span>
+       <span class="n">where</span> <span class="n">iprange</span> <span class="o">&gt;&gt;=</span> <span class="s1">&#39;8.8.8.8&#39;</span><span class="p">;</span>
+
+<span class="o">-</span><span class="p">[</span> <span class="n">RECORD</span> <span class="mi">1</span> <span class="p">]</span><span class="o">------------------</span>
+<span class="n">locid</span>      <span class="o">|</span> <span class="mi">223</span>
+<span class="n">country</span>    <span class="o">|</span> <span class="n">US</span>
+<span class="n">region</span>     <span class="o">|</span>
+<span class="n">city</span>       <span class="o">|</span>
+<span class="n">postalcode</span> <span class="o">|</span>
+<span class="n">location</span>   <span class="o">|</span> <span class="p">(</span><span class="o">-</span><span class="mi">97</span><span class="p">,</span><span class="mi">38</span><span class="p">)</span>
+<span class="n">metrocode</span>  <span class="o">|</span>
+<span class="n">areacode</span>   <span class="o">|</span>
+<span class="n">iprange</span>    <span class="o">|</span> <span class="mf">8.8</span><span class="o">.</span><span class="mf">8.8</span><span class="o">-</span><span class="mf">8.8</span><span class="o">.</span><span class="mf">37.255</span>
+
+<span class="n">Time</span><span class="p">:</span> <span class="mf">0.747</span> <span class="n">ms</span>
+</pre></div>
+</div>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="../index.html">Documentation overview</a><ul>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="../_sources/tutorial/geolite.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/tutorial/mysql.html

@@ -0,0 +1,246 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Migrating from MySQL to PostgreSQL &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="../_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="index" title="Index" href="../genindex.html" />
+    <link rel="search" title="Search" href="../search.html" />
+   
+  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="migrating-from-mysql-to-postgresql">
+<h1>Migrating from MySQL to PostgreSQL<a class="headerlink" href="#migrating-from-mysql-to-postgresql" title="Permalink to this headline">¶</a></h1>
+<p>If you want to migrate your data over to <a class="reference external" href="http://www.postgresql.org">PostgreSQL</a> from MySQL then pgloader is the tool of
+choice!</p>
+<p>Most tools around are skipping the main problem with migrating from MySQL,
+which is to do with the type casting and data sanitizing that needs to be
+done. pgloader will not leave you alone on those topics.</p>
+<div class="section" id="in-a-single-command-line">
+<h2>In a Single Command Line<a class="headerlink" href="#in-a-single-command-line" title="Permalink to this headline">¶</a></h2>
+<p>As an example, we will use the f1db database from &lt;<a class="reference external" href="http://ergast.com/mrd/">http://ergast.com/mrd/</a>&gt;
+which which provides a historical record of motor racing data for
+non-commercial purposes. You can either use their API or download the whole
+database at <a class="reference external" href="http://ergast.com/downloads/f1db.sql.gz">http://ergast.com/downloads/f1db.sql.gz</a>. Once you’ve done that load the
+database in MySQL:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>$ mysql -u root
+&gt; create database f1db;
+&gt; source f1db.sql
+</pre></div>
+</div>
+<p>Now let’s migrate this database into PostgreSQL in a single command line:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>$ createdb f1db
+$ pgloader mysql://root@localhost/f1db pgsql:///f1db
+</pre></div>
+</div>
+<p>Done! All with schema, table definitions, constraints, indexes, primary
+keys, <em>auto_increment</em> columns turned into <em>bigserial</em> , foreign keys,
+comments, and if you had some MySQL default values such as <em>ON UPDATE
+CURRENT_TIMESTAMP</em> they would have been translated to a <a class="reference external" href="https://www.postgresql.org/docs/current/static/plpgsql-trigger.html">PostgreSQL before
+update trigger</a>
+automatically.</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>$ pgloader mysql://root@localhost/f1db pgsql:///f1db
+2017-06-16T08:56:14.064000+02:00 LOG Main logs in &#39;/private/tmp/pgloader/pgloader.log&#39;
+2017-06-16T08:56:14.068000+02:00 LOG Data errors in &#39;/private/tmp/pgloader/&#39;
+2017-06-16T08:56:19.542000+02:00 LOG report summary reset
+               table name       read   imported     errors      total time
+-------------------------  ---------  ---------  ---------  --------------
+          fetch meta data         33         33          0          0.365s
+           Create Schemas          0          0          0          0.007s
+         Create SQL Types          0          0          0          0.006s
+            Create tables         26         26          0          0.068s
+           Set Table OIDs         13         13          0          0.012s
+-------------------------  ---------  ---------  ---------  --------------
+  f1db.constructorresults      11011      11011          0          0.205s
+            f1db.circuits         73         73          0          0.150s
+        f1db.constructors        208        208          0          0.059s
+f1db.constructorstandings      11766      11766          0          0.365s
+             f1db.drivers        841        841          0          0.268s
+            f1db.laptimes     413578     413578          0          2.892s
+     f1db.driverstandings      31420      31420          0          0.583s
+            f1db.pitstops       5796       5796          0          2.154s
+               f1db.races        976        976          0          0.227s
+          f1db.qualifying       7257       7257          0          0.228s
+             f1db.seasons         68         68          0          0.527s
+             f1db.results      23514      23514          0          0.658s
+              f1db.status        133        133          0          0.130s
+-------------------------  ---------  ---------  ---------  --------------
+  COPY Threads Completion         39         39          0          4.303s
+           Create Indexes         20         20          0          1.497s
+   Index Build Completion         20         20          0          0.214s
+          Reset Sequences          0         10          0          0.058s
+             Primary Keys         13         13          0          0.012s
+      Create Foreign Keys          0          0          0          0.000s
+          Create Triggers          0          0          0          0.001s
+         Install Comments          0          0          0          0.000s
+-------------------------  ---------  ---------  ---------  --------------
+        Total import time     506641     506641          0          5.547s
+</pre></div>
+</div>
+<p>You may need to have special cases to take care of tho, or views that you
+want to materialize while doing the migration. In advanced case you can use
+the pgloader command.</p>
+</div>
+<div class="section" id="the-command">
+<h2>The Command<a class="headerlink" href="#the-command" title="Permalink to this headline">¶</a></h2>
+<p>To load data with pgloader you need to define in a <em>command</em> the operations
+in some details. Here’s our example for loading the <a class="reference external" href="http://dev.mysql.com/doc/sakila/en/">MySQL Sakila Sample
+Database</a>.</p>
+<p>Here’s our command:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>load database
+     from      mysql://root@localhost/sakila
+     into postgresql:///sakila
+
+ WITH include drop, create tables, no truncate,
+      create indexes, reset sequences, foreign keys
+
+  SET maintenance_work_mem to &#39;128MB&#39;, work_mem to &#39;12MB&#39;, search_path to &#39;sakila&#39;
+
+ CAST type datetime to timestamptz
+                drop default drop not null using zero-dates-to-null,
+      type date drop not null drop default using zero-dates-to-null
+
+ MATERIALIZE VIEWS film_list, staff_list
+
+ -- INCLUDING ONLY TABLE NAMES MATCHING ~/film/, &#39;actor&#39;
+ -- EXCLUDING TABLE NAMES MATCHING ~&lt;ory&gt;
+
+ BEFORE LOAD DO
+ $$ create schema if not exists sakila; $$;
+</pre></div>
+</div>
+<p>Note that here pgloader will benefit from the meta-data information found in
+the MySQL database to create a PostgreSQL database capable of hosting the
+data as described, then load the data.</p>
+<p>In particular, some specific <em>casting rules</em> are given here, to cope with
+date values such as <cite>0000-00-00</cite> that MySQL allows and PostgreSQL rejects
+for not existing in our calendar. It’s possible to add per-column casting
+rules too, which is useful is some of your <cite>tinyint</cite> are in fact <cite>smallint</cite>
+while some others are in fact <cite>boolean</cite> values.</p>
+<p>Finaly note that we are using the <em>MATERIALIZE VIEWS</em> clause of pgloader:
+the selected views here will be migrated over to PostgreSQL <em>with their
+contents</em>.</p>
+<p>It’s possible to use the <em>MATERIALIZE VIEWS</em> clause and give both the name
+and the SQL (in MySQL dialect) definition of view, then pgloader creates the
+view before loading the data, then drops it again at the end.</p>
+<p>## Loading the data</p>
+<p>Let’s start the <cite>pgloader</cite> command with our <cite>sakila.load</cite> command file:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>$ pgloader sakila.load
+... LOG Starting pgloader, log system is ready.
+... LOG Parsing commands from file &quot;/Users/dim/dev/pgloader/test/sakila.load&quot;
+   &lt;WARNING: table &quot;xxx&quot; does not exists have been edited away&gt;
+
+            table name       read   imported     errors            time
+----------------------  ---------  ---------  ---------  --------------
+           before load          1          1          0          0.007s
+       fetch meta data         45         45          0          0.402s
+          create, drop          0         36          0          0.208s
+----------------------  ---------  ---------  ---------  --------------
+                 actor        200        200          0          0.071s
+               address        603        603          0          0.035s
+              category         16         16          0          0.018s
+                  city        600        600          0          0.037s
+               country        109        109          0          0.023s
+              customer        599        599          0          0.073s
+                  film       1000       1000          0          0.135s
+            film_actor       5462       5462          0          0.236s
+         film_category       1000       1000          0          0.070s
+             film_text       1000       1000          0          0.080s
+             inventory       4581       4581          0          0.136s
+              language          6          6          0          0.036s
+               payment      16049      16049          0          0.539s
+                rental      16044      16044          0          0.648s
+                 staff          2          2          0          0.041s
+                 store          2          2          0          0.036s
+             film_list        997        997          0          0.247s
+            staff_list          2          2          0          0.135s
+Index Build Completion          0          0          0          0.000s
+----------------------  ---------  ---------  ---------  --------------
+        Create Indexes         41         41          0          0.964s
+       Reset Sequences          0          1          0          0.035s
+          Foreign Keys         22         22          0          0.254s
+----------------------  ---------  ---------  ---------  --------------
+     Total import time      48272      48272          0          3.502s
+</pre></div>
+</div>
+<p>The <em>WARNING</em> messages we see here are expected as the PostgreSQL database
+is empty when running the command, and pgloader is using the SQL commands
+<cite>DROP TABLE IF EXISTS</cite> when the given command uses the <cite>include drop</cite>
+option.</p>
+<p>Note that the output of the command has been edited to facilitate its
+browsing online.</p>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="../index.html">Documentation overview</a><ul>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="../_sources/tutorial/mysql.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/tutorial/quickstart.html

@@ -0,0 +1,218 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>PgLoader Quick Start &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="../_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="index" title="Index" href="../genindex.html" />
+    <link rel="search" title="Search" href="../search.html" />
+   
+  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="pgloader-quick-start">
+<h1>PgLoader Quick Start<a class="headerlink" href="#pgloader-quick-start" title="Permalink to this headline">¶</a></h1>
+<p>In simple cases, pgloader is very easy to use.</p>
+<div class="section" id="csv">
+<h2>CSV<a class="headerlink" href="#csv" title="Permalink to this headline">¶</a></h2>
+<p>Load data from a CSV file into a pre-existing table in your database:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>pgloader --type csv                                   \
+         --field id --field field                     \
+         --with truncate                              \
+         --with &quot;fields terminated by &#39;,&#39;&quot;            \
+         ./test/data/matching-1.csv                   \
+         postgres:///pgloader?tablename=matching
+</pre></div>
+</div>
+<p>In that example the whole loading is driven from the command line, bypassing
+the need for writing a command in the pgloader command syntax entirely. As
+there’s no command though, the extra information needed must be provided on
+the command line using the <cite>–type</cite> and <cite>–field</cite> and <cite>–with</cite> switches.</p>
+<p>For documentation about the available syntaxes for the <cite>–field</cite> and
+<cite>–with</cite> switches, please refer to the CSV section later in the man page.</p>
+<p>Note also that the PostgreSQL URI includes the target <em>tablename</em>.</p>
+</div>
+<div class="section" id="reading-from-stdin">
+<h2>Reading from STDIN<a class="headerlink" href="#reading-from-stdin" title="Permalink to this headline">¶</a></h2>
+<p>File based pgloader sources can be loaded from the standard input, as in the
+following example:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>pgloader --type csv                                         \
+         --field &quot;usps,geoid,aland,awater,aland_sqmi,awater_sqmi,intptlat,intptlong&quot; \
+         --with &quot;skip header = 1&quot;                          \
+         --with &quot;fields terminated by &#39;\t&#39;&quot;                \
+         -                                                 \
+         postgresql:///pgloader?districts_longlat          \
+         &lt; test/data/2013_Gaz_113CDs_national.txt
+</pre></div>
+</div>
+<p>The dash (<cite>-</cite>) character as a source is used to mean <em>standard input</em>, as
+usual in Unix command lines. It’s possible to stream compressed content to
+pgloader with this technique, using the Unix pipe:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>gunzip -c source.gz | pgloader --type csv ... - pgsql:///target?foo
+</pre></div>
+</div>
+</div>
+<div class="section" id="loading-from-csv-available-through-http">
+<h2>Loading from CSV available through HTTP<a class="headerlink" href="#loading-from-csv-available-through-http" title="Permalink to this headline">¶</a></h2>
+<p>The same command as just above can also be run if the CSV file happens to be
+found on a remote HTTP location:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>pgloader --type csv                                                     \
+         --field &quot;usps,geoid,aland,awater,aland_sqmi,awater_sqmi,intptlat,intptlong&quot; \
+         --with &quot;skip header = 1&quot;                                       \
+         --with &quot;fields terminated by &#39;\t&#39;&quot;                             \
+         http://pgsql.tapoueh.org/temp/2013_Gaz_113CDs_national.txt     \
+         postgresql:///pgloader?districts_longlat
+</pre></div>
+</div>
+<p>Some more options have to be used in that case, as the file contains a
+one-line header (most commonly that’s column names, could be a copyright
+notice). Also, in that case, we specify all the fields right into a single
+<cite>–field</cite> option argument.</p>
+<p>Again, the PostgreSQL target connection string must contain the <em>tablename</em>
+option and you have to ensure that the target table exists and may fit the
+data. Here’s the SQL command used in that example in case you want to try it
+yourself:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">create</span> <span class="n">table</span> <span class="n">districts_longlat</span>
+<span class="p">(</span>
+         <span class="n">usps</span>        <span class="n">text</span><span class="p">,</span>
+         <span class="n">geoid</span>       <span class="n">text</span><span class="p">,</span>
+         <span class="n">aland</span>       <span class="n">bigint</span><span class="p">,</span>
+         <span class="n">awater</span>      <span class="n">bigint</span><span class="p">,</span>
+         <span class="n">aland_sqmi</span>  <span class="n">double</span> <span class="n">precision</span><span class="p">,</span>
+         <span class="n">awater_sqmi</span> <span class="n">double</span> <span class="n">precision</span><span class="p">,</span>
+         <span class="n">intptlat</span>    <span class="n">double</span> <span class="n">precision</span><span class="p">,</span>
+         <span class="n">intptlong</span>   <span class="n">double</span> <span class="n">precision</span>
+<span class="p">);</span>
+</pre></div>
+</div>
+<p>Also notice that the same command will work against an archived version of
+the same data.</p>
+</div>
+<div class="section" id="streaming-csv-data-from-an-http-compressed-file">
+<h2>Streaming CSV data from an HTTP compressed file<a class="headerlink" href="#streaming-csv-data-from-an-http-compressed-file" title="Permalink to this headline">¶</a></h2>
+<p>Finally, it’s important to note that pgloader first fetches the content from
+the HTTP URL it to a local file, then expand the archive when it’s
+recognized to be one, and only then processes the locally expanded file.</p>
+<p>In some cases, either because pgloader has no direct support for your
+archive format or maybe because expanding the archive is not feasible in
+your environment, you might want to <em>stream</em> the content straight from its
+remote location into PostgreSQL. Here’s how to do that, using the old battle
+tested Unix Pipes trick:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>curl http://pgsql.tapoueh.org/temp/2013_Gaz_113CDs_national.txt.gz \
+| gunzip -c                                                        \
+| pgloader --type csv                                              \
+           --field &quot;usps,geoid,aland,awater,aland_sqmi,awater_sqmi,intptlat,intptlong&quot;
+           --with &quot;skip header = 1&quot;                                \
+           --with &quot;fields terminated by &#39;\t&#39;&quot;                      \
+           -                                                       \
+           postgresql:///pgloader?districts_longlat
+</pre></div>
+</div>
+<p>Now the OS will take care of the streaming and buffering between the network
+and the commands and pgloader will take care of streaming the data down to
+PostgreSQL.</p>
+</div>
+<div class="section" id="migrating-from-sqlite">
+<h2>Migrating from SQLite<a class="headerlink" href="#migrating-from-sqlite" title="Permalink to this headline">¶</a></h2>
+<p>The following command will open the SQLite database, discover its tables
+definitions including indexes and foreign keys, migrate those definitions
+while <em>casting</em> the data type specifications to their PostgreSQL equivalent
+and then migrate the data over:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">createdb</span> <span class="n">newdb</span>
+<span class="n">pgloader</span> <span class="o">./</span><span class="n">test</span><span class="o">/</span><span class="n">sqlite</span><span class="o">/</span><span class="n">sqlite</span><span class="o">.</span><span class="n">db</span> <span class="n">postgresql</span><span class="p">:</span><span class="o">///</span><span class="n">newdb</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="migrating-from-mysql">
+<h2>Migrating from MySQL<a class="headerlink" href="#migrating-from-mysql" title="Permalink to this headline">¶</a></h2>
+<p>Just create a database where to host the MySQL data and definitions and have
+pgloader do the migration for you in a single command line:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">createdb</span> <span class="n">pagila</span>
+<span class="n">pgloader</span> <span class="n">mysql</span><span class="p">:</span><span class="o">//</span><span class="n">user</span><span class="nd">@localhost</span><span class="o">/</span><span class="n">sakila</span> <span class="n">postgresql</span><span class="p">:</span><span class="o">///</span><span class="n">pagila</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="fetching-an-archived-dbf-file-from-a-http-remote-location">
+<h2>Fetching an archived DBF file from a HTTP remote location<a class="headerlink" href="#fetching-an-archived-dbf-file-from-a-http-remote-location" title="Permalink to this headline">¶</a></h2>
+<p>It’s possible for pgloader to download a file from HTTP, unarchive it, and
+only then open it to discover the schema then load the data:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">createdb</span> <span class="n">foo</span>
+<span class="n">pgloader</span> <span class="o">--</span><span class="nb">type</span> <span class="n">dbf</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">www</span><span class="o">.</span><span class="n">insee</span><span class="o">.</span><span class="n">fr</span><span class="o">/</span><span class="n">fr</span><span class="o">/</span><span class="n">methodes</span><span class="o">/</span><span class="n">nomenclatures</span><span class="o">/</span><span class="n">cog</span><span class="o">/</span><span class="n">telechargement</span><span class="o">/</span><span class="mi">2013</span><span class="o">/</span><span class="n">dbf</span><span class="o">/</span><span class="n">historiq2013</span><span class="o">.</span><span class="n">zip</span> <span class="n">postgresql</span><span class="p">:</span><span class="o">///</span><span class="n">foo</span>
+</pre></div>
+</div>
+<p>Here it’s not possible for pgloader to guess the kind of data source it’s
+being given, so it’s necessary to use the <cite>–type</cite> command line switch.</p>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="../index.html">Documentation overview</a><ul>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="../_sources/tutorial/quickstart.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/tutorial/sqlite.html

@@ -0,0 +1,208 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>Loading SQLite files with pgloader &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="../_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="index" title="Index" href="../genindex.html" />
+    <link rel="search" title="Search" href="../search.html" />
+   
+  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="loading-sqlite-files-with-pgloader">
+<h1>Loading SQLite files with pgloader<a class="headerlink" href="#loading-sqlite-files-with-pgloader" title="Permalink to this headline">¶</a></h1>
+<p>The SQLite database is a respected solution to manage your data with. Its
+embeded nature makes it a source of migrations when a projects now needs to
+handle more concurrency, which [PostgreSQL](<a class="reference external" href="http://www.postgresql.org/">http://www.postgresql.org/</a>) is
+very good at. pgloader can help you there.</p>
+<div class="section" id="in-a-single-command-line">
+<h2>In a Single Command Line<a class="headerlink" href="#in-a-single-command-line" title="Permalink to this headline">¶</a></h2>
+<p>You can</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>$ createdb chinook
+$ pgloader https://github.com/lerocha/chinook-database/raw/master/ChinookDatabase/DataSources/Chinook_Sqlite_AutoIncrementPKs.sqlite pgsql:///chinook
+</pre></div>
+</div>
+<p>Done! All with the schema, data, constraints, primary keys and foreign keys,
+etc. We also see an error with the Chinook schema that contains several
+primary key definitions against the same table, which is not accepted by
+PostgreSQL:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="mi">2017</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">20</span><span class="n">T16</span><span class="p">:</span><span class="mi">18</span><span class="p">:</span><span class="mf">59.019000</span><span class="o">+</span><span class="mi">02</span><span class="p">:</span><span class="mi">00</span> <span class="n">LOG</span> <span class="n">Data</span> <span class="n">errors</span> <span class="ow">in</span> <span class="s1">&#39;/private/tmp/pgloader/&#39;</span>
+<span class="mi">2017</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">20</span><span class="n">T16</span><span class="p">:</span><span class="mi">18</span><span class="p">:</span><span class="mf">59.236000</span><span class="o">+</span><span class="mi">02</span><span class="p">:</span><span class="mi">00</span> <span class="n">LOG</span> <span class="n">Fetching</span> <span class="s1">&#39;https://github.com/lerocha/chinook-database/raw/master/ChinookDatabase/DataSources/Chinook_Sqlite_AutoIncrementPKs.sqlite&#39;</span>
+<span class="mi">2017</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">20</span><span class="n">T16</span><span class="p">:</span><span class="mi">19</span><span class="p">:</span><span class="mf">00.664000</span><span class="o">+</span><span class="mi">02</span><span class="p">:</span><span class="mi">00</span> <span class="n">ERROR</span> <span class="n">Database</span> <span class="n">error</span> <span class="mi">42</span><span class="n">P16</span><span class="p">:</span> <span class="n">multiple</span> <span class="n">primary</span> <span class="n">keys</span> <span class="k">for</span> <span class="n">table</span> <span class="s2">&quot;playlisttrack&quot;</span> <span class="n">are</span> <span class="ow">not</span> <span class="n">allowed</span>
+<span class="n">QUERY</span><span class="p">:</span> <span class="n">ALTER</span> <span class="n">TABLE</span> <span class="n">playlisttrack</span> <span class="n">ADD</span> <span class="n">PRIMARY</span> <span class="n">KEY</span> <span class="n">USING</span> <span class="n">INDEX</span> <span class="n">idx_66873_sqlite_autoindex_playlisttrack_1</span><span class="p">;</span>
+<span class="mi">2017</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">20</span><span class="n">T16</span><span class="p">:</span><span class="mi">19</span><span class="p">:</span><span class="mf">00.665000</span><span class="o">+</span><span class="mi">02</span><span class="p">:</span><span class="mi">00</span> <span class="n">LOG</span> <span class="n">report</span> <span class="n">summary</span> <span class="n">reset</span>
+             <span class="n">table</span> <span class="n">name</span>       <span class="n">read</span>   <span class="n">imported</span>     <span class="n">errors</span>      <span class="n">total</span> <span class="n">time</span>
+<span class="o">-----------------------</span>  <span class="o">---------</span>  <span class="o">---------</span>  <span class="o">---------</span>  <span class="o">--------------</span>
+                  <span class="n">fetch</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mf">0.877</span><span class="n">s</span>
+        <span class="n">fetch</span> <span class="n">meta</span> <span class="n">data</span>         <span class="mi">33</span>         <span class="mi">33</span>          <span class="mi">0</span>          <span class="mf">0.033</span><span class="n">s</span>
+         <span class="n">Create</span> <span class="n">Schemas</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mf">0.003</span><span class="n">s</span>
+       <span class="n">Create</span> <span class="n">SQL</span> <span class="n">Types</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mf">0.006</span><span class="n">s</span>
+          <span class="n">Create</span> <span class="n">tables</span>         <span class="mi">22</span>         <span class="mi">22</span>          <span class="mi">0</span>          <span class="mf">0.043</span><span class="n">s</span>
+         <span class="n">Set</span> <span class="n">Table</span> <span class="n">OIDs</span>         <span class="mi">11</span>         <span class="mi">11</span>          <span class="mi">0</span>          <span class="mf">0.012</span><span class="n">s</span>
+<span class="o">-----------------------</span>  <span class="o">---------</span>  <span class="o">---------</span>  <span class="o">---------</span>  <span class="o">--------------</span>
+                  <span class="n">album</span>        <span class="mi">347</span>        <span class="mi">347</span>          <span class="mi">0</span>          <span class="mf">0.023</span><span class="n">s</span>
+                 <span class="n">artist</span>        <span class="mi">275</span>        <span class="mi">275</span>          <span class="mi">0</span>          <span class="mf">0.023</span><span class="n">s</span>
+               <span class="n">customer</span>         <span class="mi">59</span>         <span class="mi">59</span>          <span class="mi">0</span>          <span class="mf">0.021</span><span class="n">s</span>
+               <span class="n">employee</span>          <span class="mi">8</span>          <span class="mi">8</span>          <span class="mi">0</span>          <span class="mf">0.018</span><span class="n">s</span>
+                <span class="n">invoice</span>        <span class="mi">412</span>        <span class="mi">412</span>          <span class="mi">0</span>          <span class="mf">0.031</span><span class="n">s</span>
+                  <span class="n">genre</span>         <span class="mi">25</span>         <span class="mi">25</span>          <span class="mi">0</span>          <span class="mf">0.021</span><span class="n">s</span>
+            <span class="n">invoiceline</span>       <span class="mi">2240</span>       <span class="mi">2240</span>          <span class="mi">0</span>          <span class="mf">0.034</span><span class="n">s</span>
+              <span class="n">mediatype</span>          <span class="mi">5</span>          <span class="mi">5</span>          <span class="mi">0</span>          <span class="mf">0.025</span><span class="n">s</span>
+          <span class="n">playlisttrack</span>       <span class="mi">8715</span>       <span class="mi">8715</span>          <span class="mi">0</span>          <span class="mf">0.040</span><span class="n">s</span>
+               <span class="n">playlist</span>         <span class="mi">18</span>         <span class="mi">18</span>          <span class="mi">0</span>          <span class="mf">0.016</span><span class="n">s</span>
+                  <span class="n">track</span>       <span class="mi">3503</span>       <span class="mi">3503</span>          <span class="mi">0</span>          <span class="mf">0.111</span><span class="n">s</span>
+<span class="o">-----------------------</span>  <span class="o">---------</span>  <span class="o">---------</span>  <span class="o">---------</span>  <span class="o">--------------</span>
+<span class="n">COPY</span> <span class="n">Threads</span> <span class="n">Completion</span>         <span class="mi">33</span>         <span class="mi">33</span>          <span class="mi">0</span>          <span class="mf">0.313</span><span class="n">s</span>
+         <span class="n">Create</span> <span class="n">Indexes</span>         <span class="mi">22</span>         <span class="mi">22</span>          <span class="mi">0</span>          <span class="mf">0.160</span><span class="n">s</span>
+ <span class="n">Index</span> <span class="n">Build</span> <span class="n">Completion</span>         <span class="mi">22</span>         <span class="mi">22</span>          <span class="mi">0</span>          <span class="mf">0.027</span><span class="n">s</span>
+        <span class="n">Reset</span> <span class="n">Sequences</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mf">0.017</span><span class="n">s</span>
+           <span class="n">Primary</span> <span class="n">Keys</span>         <span class="mi">12</span>          <span class="mi">0</span>          <span class="mi">1</span>          <span class="mf">0.013</span><span class="n">s</span>
+    <span class="n">Create</span> <span class="n">Foreign</span> <span class="n">Keys</span>         <span class="mi">11</span>         <span class="mi">11</span>          <span class="mi">0</span>          <span class="mf">0.040</span><span class="n">s</span>
+        <span class="n">Create</span> <span class="n">Triggers</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mf">0.000</span><span class="n">s</span>
+       <span class="n">Install</span> <span class="n">Comments</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mf">0.000</span><span class="n">s</span>
+<span class="o">-----------------------</span>  <span class="o">---------</span>  <span class="o">---------</span>  <span class="o">---------</span>  <span class="o">--------------</span>
+      <span class="n">Total</span> <span class="kn">import</span> <span class="nn">time</span>      <span class="mi">15607</span>      <span class="mi">15607</span>          <span class="mi">0</span>          <span class="mf">1.669</span><span class="n">s</span>
+</pre></div>
+</div>
+<p>You may need to have special cases to take care of tho. In advanced case you
+can use the pgloader command.</p>
+</div>
+<div class="section" id="the-command">
+<h2>The Command<a class="headerlink" href="#the-command" title="Permalink to this headline">¶</a></h2>
+<p>To load data with [pgloader](<a class="reference external" href="http://pgloader.io/">http://pgloader.io/</a>) you need to define in a
+<em>command</em> the operations in some details. Here’s our command:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">load</span> <span class="n">database</span>
+     <span class="kn">from</span> <span class="s1">&#39;sqlite/Chinook_Sqlite_AutoIncrementPKs.sqlite&#39;</span>
+     <span class="n">into</span> <span class="n">postgresql</span><span class="p">:</span><span class="o">///</span><span class="n">pgloader</span>
+
+ <span class="k">with</span> <span class="n">include</span> <span class="n">drop</span><span class="p">,</span> <span class="n">create</span> <span class="n">tables</span><span class="p">,</span> <span class="n">create</span> <span class="n">indexes</span><span class="p">,</span> <span class="n">reset</span> <span class="n">sequences</span>
+
+  <span class="nb">set</span> <span class="n">work_mem</span> <span class="n">to</span> <span class="s1">&#39;16MB&#39;</span><span class="p">,</span> <span class="n">maintenance_work_mem</span> <span class="n">to</span> <span class="s1">&#39;512 MB&#39;</span><span class="p">;</span>
+</pre></div>
+</div>
+<p>Note that here pgloader will benefit from the meta-data information found in
+the SQLite file to create a PostgreSQL database capable of hosting the data
+as described, then load the data.</p>
+</div>
+<div class="section" id="loading-the-data">
+<h2>Loading the data<a class="headerlink" href="#loading-the-data" title="Permalink to this headline">¶</a></h2>
+<p>Let’s start the <cite>pgloader</cite> command with our <cite>sqlite.load</cite> command file:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>$ pgloader sqlite.load
+... LOG Starting pgloader, log system is ready.
+... LOG Parsing commands from file &quot;/Users/dim/dev/pgloader/test/sqlite.load&quot;
+... WARNING Postgres warning: table &quot;album&quot; does not exist, skipping
+... WARNING Postgres warning: table &quot;artist&quot; does not exist, skipping
+... WARNING Postgres warning: table &quot;customer&quot; does not exist, skipping
+... WARNING Postgres warning: table &quot;employee&quot; does not exist, skipping
+... WARNING Postgres warning: table &quot;genre&quot; does not exist, skipping
+... WARNING Postgres warning: table &quot;invoice&quot; does not exist, skipping
+... WARNING Postgres warning: table &quot;invoiceline&quot; does not exist, skipping
+... WARNING Postgres warning: table &quot;mediatype&quot; does not exist, skipping
+... WARNING Postgres warning: table &quot;playlist&quot; does not exist, skipping
+... WARNING Postgres warning: table &quot;playlisttrack&quot; does not exist, skipping
+... WARNING Postgres warning: table &quot;track&quot; does not exist, skipping
+            table name       read   imported     errors            time
+----------------------  ---------  ---------  ---------  --------------
+      create, truncate          0          0          0          0.052s
+                 Album        347        347          0          0.070s
+                Artist        275        275          0          0.014s
+              Customer         59         59          0          0.014s
+              Employee          8          8          0          0.012s
+                 Genre         25         25          0          0.018s
+               Invoice        412        412          0          0.032s
+           InvoiceLine       2240       2240          0          0.077s
+             MediaType          5          5          0          0.012s
+              Playlist         18         18          0          0.008s
+         PlaylistTrack       8715       8715          0          0.071s
+                 Track       3503       3503          0          0.105s
+index build completion          0          0          0          0.000s
+----------------------  ---------  ---------  ---------  --------------
+        Create Indexes         20         20          0          0.279s
+       reset sequences          0          0          0          0.043s
+----------------------  ---------  ---------  ---------  --------------
+  Total streaming time      15607      15607          0          0.476s
+</pre></div>
+</div>
+<p>We can see that <a class="reference external" href="http://pgloader.io">pgloader</a> did download the file from
+its HTTP URL location then <em>unziped</em> it before loading it.</p>
+<p>Also, the <em>WARNING</em> messages we see here are expected as the PostgreSQL
+database is empty when running the command, and pgloader is using the SQL
+commands <cite>DROP TABLE IF EXISTS</cite> when the given command uses the <cite>include
+drop</cite> option.</p>
+<p>Note that the output of the command has been edited to facilitate its
+browsing online.</p>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="../index.html">Documentation overview</a><ul>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="../_sources/tutorial/sqlite.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/_build/html/tutorial/tutorial.html

@@ -0,0 +1,893 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>PgLoader Tutorial &#8212; pgloader 3.4.1 documentation</title>
+    <link rel="stylesheet" href="../_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     '3.4.1',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="index" title="Index" href="../genindex.html" />
+    <link rel="search" title="Search" href="../search.html" />
+    <link rel="next" title="PgLoader Reference Manual" href="../pgloader.html" />
+    <link rel="prev" title="Introduction" href="../intro.html" />
+   
+  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
+  
+  
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+
+  </head>
+  <body>
+  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="pgloader-tutorial">
+<h1>PgLoader Tutorial<a class="headerlink" href="#pgloader-tutorial" title="Permalink to this headline">¶</a></h1>
+<div class="section" id="pgloader-quick-start">
+<h2>PgLoader Quick Start<a class="headerlink" href="#pgloader-quick-start" title="Permalink to this headline">¶</a></h2>
+<p>In simple cases, pgloader is very easy to use.</p>
+<div class="section" id="csv">
+<h3>CSV<a class="headerlink" href="#csv" title="Permalink to this headline">¶</a></h3>
+<p>Load data from a CSV file into a pre-existing table in your database:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>pgloader --type csv                                   \
+         --field id --field field                     \
+         --with truncate                              \
+         --with &quot;fields terminated by &#39;,&#39;&quot;            \
+         ./test/data/matching-1.csv                   \
+         postgres:///pgloader?tablename=matching
+</pre></div>
+</div>
+<p>In that example the whole loading is driven from the command line, bypassing
+the need for writing a command in the pgloader command syntax entirely. As
+there’s no command though, the extra information needed must be provided on
+the command line using the <cite>–type</cite> and <cite>–field</cite> and <cite>–with</cite> switches.</p>
+<p>For documentation about the available syntaxes for the <cite>–field</cite> and
+<cite>–with</cite> switches, please refer to the CSV section later in the man page.</p>
+<p>Note also that the PostgreSQL URI includes the target <em>tablename</em>.</p>
+</div>
+<div class="section" id="reading-from-stdin">
+<h3>Reading from STDIN<a class="headerlink" href="#reading-from-stdin" title="Permalink to this headline">¶</a></h3>
+<p>File based pgloader sources can be loaded from the standard input, as in the
+following example:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>pgloader --type csv                                         \
+         --field &quot;usps,geoid,aland,awater,aland_sqmi,awater_sqmi,intptlat,intptlong&quot; \
+         --with &quot;skip header = 1&quot;                          \
+         --with &quot;fields terminated by &#39;\t&#39;&quot;                \
+         -                                                 \
+         postgresql:///pgloader?districts_longlat          \
+         &lt; test/data/2013_Gaz_113CDs_national.txt
+</pre></div>
+</div>
+<p>The dash (<cite>-</cite>) character as a source is used to mean <em>standard input</em>, as
+usual in Unix command lines. It’s possible to stream compressed content to
+pgloader with this technique, using the Unix pipe:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>gunzip -c source.gz | pgloader --type csv ... - pgsql:///target?foo
+</pre></div>
+</div>
+</div>
+<div class="section" id="loading-from-csv-available-through-http">
+<h3>Loading from CSV available through HTTP<a class="headerlink" href="#loading-from-csv-available-through-http" title="Permalink to this headline">¶</a></h3>
+<p>The same command as just above can also be run if the CSV file happens to be
+found on a remote HTTP location:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>pgloader --type csv                                                     \
+         --field &quot;usps,geoid,aland,awater,aland_sqmi,awater_sqmi,intptlat,intptlong&quot; \
+         --with &quot;skip header = 1&quot;                                       \
+         --with &quot;fields terminated by &#39;\t&#39;&quot;                             \
+         http://pgsql.tapoueh.org/temp/2013_Gaz_113CDs_national.txt     \
+         postgresql:///pgloader?districts_longlat
+</pre></div>
+</div>
+<p>Some more options have to be used in that case, as the file contains a
+one-line header (most commonly that’s column names, could be a copyright
+notice). Also, in that case, we specify all the fields right into a single
+<cite>–field</cite> option argument.</p>
+<p>Again, the PostgreSQL target connection string must contain the <em>tablename</em>
+option and you have to ensure that the target table exists and may fit the
+data. Here’s the SQL command used in that example in case you want to try it
+yourself:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">create</span> <span class="n">table</span> <span class="n">districts_longlat</span>
+<span class="p">(</span>
+         <span class="n">usps</span>        <span class="n">text</span><span class="p">,</span>
+         <span class="n">geoid</span>       <span class="n">text</span><span class="p">,</span>
+         <span class="n">aland</span>       <span class="n">bigint</span><span class="p">,</span>
+         <span class="n">awater</span>      <span class="n">bigint</span><span class="p">,</span>
+         <span class="n">aland_sqmi</span>  <span class="n">double</span> <span class="n">precision</span><span class="p">,</span>
+         <span class="n">awater_sqmi</span> <span class="n">double</span> <span class="n">precision</span><span class="p">,</span>
+         <span class="n">intptlat</span>    <span class="n">double</span> <span class="n">precision</span><span class="p">,</span>
+         <span class="n">intptlong</span>   <span class="n">double</span> <span class="n">precision</span>
+<span class="p">);</span>
+</pre></div>
+</div>
+<p>Also notice that the same command will work against an archived version of
+the same data.</p>
+</div>
+<div class="section" id="streaming-csv-data-from-an-http-compressed-file">
+<h3>Streaming CSV data from an HTTP compressed file<a class="headerlink" href="#streaming-csv-data-from-an-http-compressed-file" title="Permalink to this headline">¶</a></h3>
+<p>Finally, it’s important to note that pgloader first fetches the content from
+the HTTP URL it to a local file, then expand the archive when it’s
+recognized to be one, and only then processes the locally expanded file.</p>
+<p>In some cases, either because pgloader has no direct support for your
+archive format or maybe because expanding the archive is not feasible in
+your environment, you might want to <em>stream</em> the content straight from its
+remote location into PostgreSQL. Here’s how to do that, using the old battle
+tested Unix Pipes trick:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>curl http://pgsql.tapoueh.org/temp/2013_Gaz_113CDs_national.txt.gz \
+| gunzip -c                                                        \
+| pgloader --type csv                                              \
+           --field &quot;usps,geoid,aland,awater,aland_sqmi,awater_sqmi,intptlat,intptlong&quot;
+           --with &quot;skip header = 1&quot;                                \
+           --with &quot;fields terminated by &#39;\t&#39;&quot;                      \
+           -                                                       \
+           postgresql:///pgloader?districts_longlat
+</pre></div>
+</div>
+<p>Now the OS will take care of the streaming and buffering between the network
+and the commands and pgloader will take care of streaming the data down to
+PostgreSQL.</p>
+</div>
+<div class="section" id="migrating-from-sqlite">
+<h3>Migrating from SQLite<a class="headerlink" href="#migrating-from-sqlite" title="Permalink to this headline">¶</a></h3>
+<p>The following command will open the SQLite database, discover its tables
+definitions including indexes and foreign keys, migrate those definitions
+while <em>casting</em> the data type specifications to their PostgreSQL equivalent
+and then migrate the data over:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">createdb</span> <span class="n">newdb</span>
+<span class="n">pgloader</span> <span class="o">./</span><span class="n">test</span><span class="o">/</span><span class="n">sqlite</span><span class="o">/</span><span class="n">sqlite</span><span class="o">.</span><span class="n">db</span> <span class="n">postgresql</span><span class="p">:</span><span class="o">///</span><span class="n">newdb</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="migrating-from-mysql">
+<h3>Migrating from MySQL<a class="headerlink" href="#migrating-from-mysql" title="Permalink to this headline">¶</a></h3>
+<p>Just create a database where to host the MySQL data and definitions and have
+pgloader do the migration for you in a single command line:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">createdb</span> <span class="n">pagila</span>
+<span class="n">pgloader</span> <span class="n">mysql</span><span class="p">:</span><span class="o">//</span><span class="n">user</span><span class="nd">@localhost</span><span class="o">/</span><span class="n">sakila</span> <span class="n">postgresql</span><span class="p">:</span><span class="o">///</span><span class="n">pagila</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="fetching-an-archived-dbf-file-from-a-http-remote-location">
+<h3>Fetching an archived DBF file from a HTTP remote location<a class="headerlink" href="#fetching-an-archived-dbf-file-from-a-http-remote-location" title="Permalink to this headline">¶</a></h3>
+<p>It’s possible for pgloader to download a file from HTTP, unarchive it, and
+only then open it to discover the schema then load the data:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">createdb</span> <span class="n">foo</span>
+<span class="n">pgloader</span> <span class="o">--</span><span class="nb">type</span> <span class="n">dbf</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">www</span><span class="o">.</span><span class="n">insee</span><span class="o">.</span><span class="n">fr</span><span class="o">/</span><span class="n">fr</span><span class="o">/</span><span class="n">methodes</span><span class="o">/</span><span class="n">nomenclatures</span><span class="o">/</span><span class="n">cog</span><span class="o">/</span><span class="n">telechargement</span><span class="o">/</span><span class="mi">2013</span><span class="o">/</span><span class="n">dbf</span><span class="o">/</span><span class="n">historiq2013</span><span class="o">.</span><span class="n">zip</span> <span class="n">postgresql</span><span class="p">:</span><span class="o">///</span><span class="n">foo</span>
+</pre></div>
+</div>
+<p>Here it’s not possible for pgloader to guess the kind of data source it’s
+being given, so it’s necessary to use the <cite>–type</cite> command line switch.</p>
+</div>
+</div>
+<div class="section" id="loading-csv-data-with-pgloader">
+<h2>Loading CSV Data with pgloader<a class="headerlink" href="#loading-csv-data-with-pgloader" title="Permalink to this headline">¶</a></h2>
+<p>CSV means <em>comma separated values</em> and is often found with quite varying
+specifications. pgloader allows you to describe those specs in its command.</p>
+<div class="section" id="the-command">
+<h3>The Command<a class="headerlink" href="#the-command" title="Permalink to this headline">¶</a></h3>
+<p>To load data with [pgloader](<a class="reference external" href="http://pgloader.io/">http://pgloader.io/</a>) you need to define in a
+<em>command</em> the operations in some details. Here’s our example for loading CSV
+data:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>LOAD CSV
+     FROM &#39;path/to/file.csv&#39; (x, y, a, b, c, d)
+     INTO postgresql:///pgloader?csv (a, b, d, c)
+
+     WITH truncate,
+          skip header = 1,
+          fields optionally enclosed by &#39;&quot;&#39;,
+          fields escaped by double-quote,
+          fields terminated by &#39;,&#39;
+
+      SET client_encoding to &#39;latin1&#39;,
+          work_mem to &#39;12MB&#39;,
+          standard_conforming_strings to &#39;on&#39;
+
+   BEFORE LOAD DO
+    $$ drop table if exists csv; $$,
+    $$ create table csv (
+        a bigint,
+        b bigint,
+        c char(2),
+        d text
+       );
+  $$;
+</pre></div>
+</div>
+</div>
+<div class="section" id="the-data">
+<h3>The Data<a class="headerlink" href="#the-data" title="Permalink to this headline">¶</a></h3>
+<p>This command allows loading the following CSV file content:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>Header, with a © sign
+&quot;2.6.190.56&quot;,&quot;2.6.190.63&quot;,&quot;33996344&quot;,&quot;33996351&quot;,&quot;GB&quot;,&quot;United Kingdom&quot;
+&quot;3.0.0.0&quot;,&quot;4.17.135.31&quot;,&quot;50331648&quot;,&quot;68257567&quot;,&quot;US&quot;,&quot;United States&quot;
+&quot;4.17.135.32&quot;,&quot;4.17.135.63&quot;,&quot;68257568&quot;,&quot;68257599&quot;,&quot;CA&quot;,&quot;Canada&quot;
+&quot;4.17.135.64&quot;,&quot;4.17.142.255&quot;,&quot;68257600&quot;,&quot;68259583&quot;,&quot;US&quot;,&quot;United States&quot;
+&quot;4.17.143.0&quot;,&quot;4.17.143.15&quot;,&quot;68259584&quot;,&quot;68259599&quot;,&quot;CA&quot;,&quot;Canada&quot;
+&quot;4.17.143.16&quot;,&quot;4.18.32.71&quot;,&quot;68259600&quot;,&quot;68296775&quot;,&quot;US&quot;,&quot;United States&quot;
+</pre></div>
+</div>
+</div>
+<div class="section" id="loading-the-data">
+<h3>Loading the data<a class="headerlink" href="#loading-the-data" title="Permalink to this headline">¶</a></h3>
+<p>Here’s how to start loading the data. Note that the ouput here has been
+edited so as to facilitate its browsing online:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>$ pgloader csv.load
+... LOG Starting pgloader, log system is ready.
+... LOG Parsing commands from file &quot;/Users/dim/dev/pgloader/test/csv.load&quot;
+
+       table name       read   imported     errors            time
+-----------------  ---------  ---------  ---------  --------------
+      before load          2          2          0          0.039s
+-----------------  ---------  ---------  ---------  --------------
+              csv          6          6          0          0.019s
+-----------------  ---------  ---------  ---------  --------------
+Total import time          6          6          0          0.058s
+</pre></div>
+</div>
+</div>
+<div class="section" id="the-result">
+<h3>The result<a class="headerlink" href="#the-result" title="Permalink to this headline">¶</a></h3>
+<p>As you can see, the command described above is filtering the input and only
+importing some of the columns from the example data file. Here’s what gets
+loaded in the PostgreSQL database:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">pgloader</span><span class="c1"># table csv;</span>
+    <span class="n">a</span>     <span class="o">|</span>    <span class="n">b</span>     <span class="o">|</span> <span class="n">c</span>  <span class="o">|</span>       <span class="n">d</span>
+<span class="o">----------+----------+----+----------------</span>
+ <span class="mi">33996344</span> <span class="o">|</span> <span class="mi">33996351</span> <span class="o">|</span> <span class="n">GB</span> <span class="o">|</span> <span class="n">United</span> <span class="n">Kingdom</span>
+ <span class="mi">50331648</span> <span class="o">|</span> <span class="mi">68257567</span> <span class="o">|</span> <span class="n">US</span> <span class="o">|</span> <span class="n">United</span> <span class="n">States</span>
+ <span class="mi">68257568</span> <span class="o">|</span> <span class="mi">68257599</span> <span class="o">|</span> <span class="n">CA</span> <span class="o">|</span> <span class="n">Canada</span>
+ <span class="mi">68257600</span> <span class="o">|</span> <span class="mi">68259583</span> <span class="o">|</span> <span class="n">US</span> <span class="o">|</span> <span class="n">United</span> <span class="n">States</span>
+ <span class="mi">68259584</span> <span class="o">|</span> <span class="mi">68259599</span> <span class="o">|</span> <span class="n">CA</span> <span class="o">|</span> <span class="n">Canada</span>
+ <span class="mi">68259600</span> <span class="o">|</span> <span class="mi">68296775</span> <span class="o">|</span> <span class="n">US</span> <span class="o">|</span> <span class="n">United</span> <span class="n">States</span>
+<span class="p">(</span><span class="mi">6</span> <span class="n">rows</span><span class="p">)</span>
+</pre></div>
+</div>
+</div>
+</div>
+<div class="section" id="loading-fixed-width-data-file-with-pgloader">
+<h2>Loading Fixed Width Data File with pgloader<a class="headerlink" href="#loading-fixed-width-data-file-with-pgloader" title="Permalink to this headline">¶</a></h2>
+<p>Some data providers still use a format where each column is specified with a
+starting index position and a given length. Usually the columns are
+blank-padded when the data is shorter than the full reserved range.</p>
+<div class="section" id="id1">
+<h3>The Command<a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h3>
+<p>To load data with [pgloader](<a class="reference external" href="http://pgloader.io/">http://pgloader.io/</a>) you need to define in a
+<em>command</em> the operations in some details. Here’s our example for loading
+Fixed Width Data, using a file provided by the US census.</p>
+<p>You can find more files from them at the
+[Census 2000 Gazetteer Files](<a class="reference external" href="http://www.census.gov/geo/maps-data/data/gazetteer2000.html">http://www.census.gov/geo/maps-data/data/gazetteer2000.html</a>).</p>
+<p>Here’s our command:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>LOAD ARCHIVE
+   FROM http://www.census.gov/geo/maps-data/data/docs/gazetteer/places2k.zip
+   INTO postgresql:///pgloader
+
+   BEFORE LOAD DO
+     $$ drop table if exists places; $$,
+     $$ create table places
+       (
+          usps      char(2)  not null,
+          fips      char(2)  not null,
+          fips_code char(5),
+          loc_name  varchar(64)
+       );
+     $$
+
+   LOAD FIXED
+        FROM FILENAME MATCHING ~/places2k.txt/
+             WITH ENCODING latin1
+             (
+                usps           from   0 for  2,
+                fips           from   2 for  2,
+                fips_code      from   4 for  5,
+                &quot;LocationName&quot; from   9 for 64 [trim right whitespace],
+                p              from  73 for  9,
+                h              from  82 for  9,
+                land           from  91 for 14,
+                water          from 105 for 14,
+                ldm            from 119 for 14,
+                wtm            from 131 for 14,
+                lat            from 143 for 10,
+                long           from 153 for 11
+             )
+        INTO postgresql:///pgloader?places
+             (
+            usps, fips, fips_code, &quot;LocationName&quot;
+             );
+</pre></div>
+</div>
+</div>
+<div class="section" id="id2">
+<h3>The Data<a class="headerlink" href="#id2" title="Permalink to this headline">¶</a></h3>
+<p>This command allows loading the following file content, where we are only
+showing the first couple of lines:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">AL0100124Abbeville</span> <span class="n">city</span>                                                       <span class="mi">2987</span>     <span class="mi">1353</span>      <span class="mi">40301945</span>        <span class="mi">120383</span>   <span class="mf">15.560669</span>    <span class="mf">0.046480</span> <span class="mf">31.566367</span> <span class="o">-</span><span class="mf">85.251300</span>
+<span class="n">AL0100460Adamsville</span> <span class="n">city</span>                                                      <span class="mi">4965</span>     <span class="mi">2042</span>      <span class="mi">50779330</span>         <span class="mi">14126</span>   <span class="mf">19.606010</span>    <span class="mf">0.005454</span> <span class="mf">33.590411</span> <span class="o">-</span><span class="mf">86.949166</span>
+<span class="n">AL0100484Addison</span> <span class="n">town</span>                                                          <span class="mi">723</span>      <span class="mi">339</span>       <span class="mi">9101325</span>             <span class="mi">0</span>    <span class="mf">3.514041</span>    <span class="mf">0.000000</span> <span class="mf">34.200042</span> <span class="o">-</span><span class="mf">87.177851</span>
+<span class="n">AL0100676Akron</span> <span class="n">town</span>                                                            <span class="mi">521</span>      <span class="mi">239</span>       <span class="mi">1436797</span>             <span class="mi">0</span>    <span class="mf">0.554750</span>    <span class="mf">0.000000</span> <span class="mf">32.876425</span> <span class="o">-</span><span class="mf">87.740978</span>
+<span class="n">AL0100820Alabaster</span> <span class="n">city</span>                                                      <span class="mi">22619</span>     <span class="mi">8594</span>      <span class="mi">53023800</span>        <span class="mi">141711</span>   <span class="mf">20.472605</span>    <span class="mf">0.054715</span> <span class="mf">33.231162</span> <span class="o">-</span><span class="mf">86.823829</span>
+<span class="n">AL0100988Albertville</span> <span class="n">city</span>                                                    <span class="mi">17247</span>     <span class="mi">7090</span>      <span class="mi">67212867</span>        <span class="mi">258738</span>   <span class="mf">25.951034</span>    <span class="mf">0.099899</span> <span class="mf">34.265362</span> <span class="o">-</span><span class="mf">86.211261</span>
+<span class="n">AL0101132Alexander</span> <span class="n">City</span> <span class="n">city</span>                                                 <span class="mi">15008</span>     <span class="mi">6855</span>     <span class="mi">100534344</span>        <span class="mi">433413</span>   <span class="mf">38.816529</span>    <span class="mf">0.167342</span> <span class="mf">32.933157</span> <span class="o">-</span><span class="mf">85.936008</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="id3">
+<h3>Loading the data<a class="headerlink" href="#id3" title="Permalink to this headline">¶</a></h3>
+<p>Let’s start the <cite>pgloader</cite> command with our <cite>census-places.load</cite> command file:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>$ pgloader census-places.load
+... LOG Starting pgloader, log system is ready.
+... LOG Parsing commands from file &quot;/Users/dim/dev/pgloader/test/census-places.load&quot;
+... LOG Fetching &#39;http://www.census.gov/geo/maps-data/data/docs/gazetteer/places2k.zip&#39;
+... LOG Extracting files from archive &#39;//private/var/folders/w7/9n8v8pw54t1gngfff0lj16040000gn/T/pgloader//places2k.zip&#39;
+
+       table name       read   imported     errors            time
+-----------------  ---------  ---------  ---------  --------------
+         download          0          0          0          1.494s
+          extract          0          0          0          1.013s
+      before load          2          2          0          0.013s
+-----------------  ---------  ---------  ---------  --------------
+           places      25375      25375          0          0.499s
+-----------------  ---------  ---------  ---------  --------------
+Total import time      25375      25375          0          3.019s
+</pre></div>
+</div>
+<p>We can see that pgloader did download the file from its HTTP URL location
+then <em>unziped</em> it before the loading itself.</p>
+<p>Note that the output of the command has been edited to facilitate its
+browsing online.</p>
+</div>
+</div>
+<div class="section" id="loading-maxmind-geolite-data-with-pgloader">
+<h2>Loading MaxMind Geolite Data with pgloader<a class="headerlink" href="#loading-maxmind-geolite-data-with-pgloader" title="Permalink to this headline">¶</a></h2>
+<p><a class="reference external" href="http://www.maxmind.com/">MaxMind</a> provides a free dataset for
+geolocation, which is quite popular. Using pgloader you can download the
+lastest version of it, extract the CSV files from the archive and load their
+content into your database directly.</p>
+<div class="section" id="id4">
+<h3>The Command<a class="headerlink" href="#id4" title="Permalink to this headline">¶</a></h3>
+<p>To load data with pgloader you need to define in a <em>command</em> the operations
+in some details. Here’s our example for loading the Geolite data:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>/*
+ * Loading from a ZIP archive containing CSV files. The full test can be
+ * done with using the archive found at
+ * http://geolite.maxmind.com/download/geoip/database/GeoLiteCity_CSV/GeoLiteCity-latest.zip
+ *
+ * And a very light version of this data set is found at
+ * http://pgsql.tapoueh.org/temp/foo.zip for quick testing.
+ */
+
+LOAD ARCHIVE
+   FROM http://geolite.maxmind.com/download/geoip/database/GeoLiteCity_CSV/GeoLiteCity-latest.zip
+   INTO postgresql:///ip4r
+
+   BEFORE LOAD DO
+     $$ create extension if not exists ip4r; $$,
+     $$ create schema if not exists geolite; $$,
+     $$ create table if not exists geolite.location
+       (
+          locid      integer primary key,
+          country    text,
+          region     text,
+          city       text,
+          postalcode text,
+          location   point,
+          metrocode  text,
+          areacode   text
+       );
+     $$,
+     $$ create table if not exists geolite.blocks
+       (
+          iprange    ip4r,
+          locid      integer
+       );
+     $$,
+     $$ drop index if exists geolite.blocks_ip4r_idx; $$,
+     $$ truncate table geolite.blocks, geolite.location cascade; $$
+
+   LOAD CSV
+        FROM FILENAME MATCHING ~/GeoLiteCity-Location.csv/
+             WITH ENCODING iso-8859-1
+             (
+                locId,
+                country,
+                region     null if blanks,
+                city       null if blanks,
+                postalCode null if blanks,
+                latitude,
+                longitude,
+                metroCode  null if blanks,
+                areaCode   null if blanks
+             )
+        INTO postgresql:///ip4r?geolite.location
+             (
+                locid,country,region,city,postalCode,
+                location point using (format nil &quot;(~a,~a)&quot; longitude latitude),
+                metroCode,areaCode
+             )
+        WITH skip header = 2,
+             fields optionally enclosed by &#39;&quot;&#39;,
+             fields escaped by double-quote,
+             fields terminated by &#39;,&#39;
+
+  AND LOAD CSV
+        FROM FILENAME MATCHING ~/GeoLiteCity-Blocks.csv/
+             WITH ENCODING iso-8859-1
+             (
+                startIpNum, endIpNum, locId
+             )
+        INTO postgresql:///ip4r?geolite.blocks
+             (
+                iprange ip4r using (ip-range startIpNum endIpNum),
+                locId
+             )
+        WITH skip header = 2,
+             fields optionally enclosed by &#39;&quot;&#39;,
+             fields escaped by double-quote,
+             fields terminated by &#39;,&#39;
+
+   FINALLY DO
+     $$ create index blocks_ip4r_idx on geolite.blocks using gist(iprange); $$;
+</pre></div>
+</div>
+<p>Note that while the <em>Geolite</em> data is using a pair of integers (<em>start</em>,
+<em>end</em>) to represent <em>ipv4</em> data, we use the very poweful <a class="reference external" href="https://github.com/RhodiumToad/ip4r">ip4r</a> PostgreSQL Extension instead.</p>
+<p>The transformation from a pair of integers into an IP is done dynamically by
+the pgloader process.</p>
+<p>Also, the location is given as a pair of <em>float</em> columns for the <em>longitude</em>
+and the <em>latitude</em> where PostgreSQL offers the
+<a class="reference external" href="http://www.postgresql.org/docs/9.3/interactive/functions-geometry.html">point</a>
+datatype, so the pgloader command here will actually transform the data on
+the fly to use the appropriate data type and its input representation.</p>
+</div>
+<div class="section" id="id5">
+<h3>Loading the data<a class="headerlink" href="#id5" title="Permalink to this headline">¶</a></h3>
+<p>Here’s how to start loading the data. Note that the ouput here has been
+edited so as to facilitate its browsing online:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>$ pgloader archive.load
+... LOG Starting pgloader, log system is ready.
+... LOG Parsing commands from file &quot;/Users/dim/dev/pgloader/test/archive.load&quot;
+... LOG Fetching &#39;http://geolite.maxmind.com/download/geoip/database/GeoLiteCity_CSV/GeoLiteCity-latest.zip&#39;
+... LOG Extracting files from archive &#39;//private/var/folders/w7/9n8v8pw54t1gngfff0lj16040000gn/T/pgloader//GeoLiteCity-latest.zip&#39;
+
+       table name       read   imported     errors            time
+-----------------  ---------  ---------  ---------  --------------
+         download          0          0          0         11.592s
+          extract          0          0          0          1.012s
+      before load          6          6          0          0.019s
+-----------------  ---------  ---------  ---------  --------------
+ geolite.location     470387     470387          0          7.743s
+   geolite.blocks    1903155    1903155          0         16.332s
+-----------------  ---------  ---------  ---------  --------------
+          finally          1          1          0         31.692s
+-----------------  ---------  ---------  ---------  --------------
+Total import time    2373542    2373542          0        1m8.390s
+</pre></div>
+</div>
+<p>The timing of course includes the transformation of the <em>1.9 million</em> pairs
+of integer into a single <em>ipv4 range</em> each. The <em>finally</em> step consists of
+creating the <em>GiST</em> specialized index as given in the main command:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">CREATE</span> <span class="n">INDEX</span> <span class="n">blocks_ip4r_idx</span> <span class="n">ON</span> <span class="n">geolite</span><span class="o">.</span><span class="n">blocks</span> <span class="n">USING</span> <span class="n">gist</span><span class="p">(</span><span class="n">iprange</span><span class="p">);</span>
+</pre></div>
+</div>
+<p>That index will then be used to speed up queries wanting to find which
+recorded geolocation contains a specific IP address:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">ip4r</span><span class="o">&gt;</span> <span class="n">select</span> <span class="o">*</span>
+        <span class="kn">from</span>      <span class="nn">geolite.location</span> <span class="n">l</span>
+             <span class="n">join</span> <span class="n">geolite</span><span class="o">.</span><span class="n">blocks</span> <span class="n">b</span> <span class="n">using</span><span class="p">(</span><span class="n">locid</span><span class="p">)</span>
+       <span class="n">where</span> <span class="n">iprange</span> <span class="o">&gt;&gt;=</span> <span class="s1">&#39;8.8.8.8&#39;</span><span class="p">;</span>
+
+<span class="o">-</span><span class="p">[</span> <span class="n">RECORD</span> <span class="mi">1</span> <span class="p">]</span><span class="o">------------------</span>
+<span class="n">locid</span>      <span class="o">|</span> <span class="mi">223</span>
+<span class="n">country</span>    <span class="o">|</span> <span class="n">US</span>
+<span class="n">region</span>     <span class="o">|</span>
+<span class="n">city</span>       <span class="o">|</span>
+<span class="n">postalcode</span> <span class="o">|</span>
+<span class="n">location</span>   <span class="o">|</span> <span class="p">(</span><span class="o">-</span><span class="mi">97</span><span class="p">,</span><span class="mi">38</span><span class="p">)</span>
+<span class="n">metrocode</span>  <span class="o">|</span>
+<span class="n">areacode</span>   <span class="o">|</span>
+<span class="n">iprange</span>    <span class="o">|</span> <span class="mf">8.8</span><span class="o">.</span><span class="mf">8.8</span><span class="o">-</span><span class="mf">8.8</span><span class="o">.</span><span class="mf">37.255</span>
+
+<span class="n">Time</span><span class="p">:</span> <span class="mf">0.747</span> <span class="n">ms</span>
+</pre></div>
+</div>
+</div>
+</div>
+<div class="section" id="loading-dbase-files-with-pgloader">
+<h2>Loading dBase files with pgloader<a class="headerlink" href="#loading-dbase-files-with-pgloader" title="Permalink to this headline">¶</a></h2>
+<p>The dBase format is still in use in some places as modern tools such as
+<em>Filemaker</em> and <em>Excel</em> offer some level of support for it. Speaking of
+support in modern tools, pgloader is right there on the list too!</p>
+<div class="section" id="id6">
+<h3>The Command<a class="headerlink" href="#id6" title="Permalink to this headline">¶</a></h3>
+<p>To load data with [pgloader](<a class="reference external" href="http://pgloader.io/">http://pgloader.io/</a>) you need to define in a
+<em>command</em> the operations in some details. Here’s our example for loading a
+dBase file, using a file provided by the french administration.</p>
+<p>You can find more files from them at the <a class="reference external" href="http://www.insee.fr/fr/methodes/nomenclatures/cog/telechargement.asp">Insee</a>
+website.</p>
+<p>Here’s our command:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">LOAD</span> <span class="n">DBF</span>
+    <span class="n">FROM</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">www</span><span class="o">.</span><span class="n">insee</span><span class="o">.</span><span class="n">fr</span><span class="o">/</span><span class="n">fr</span><span class="o">/</span><span class="n">methodes</span><span class="o">/</span><span class="n">nomenclatures</span><span class="o">/</span><span class="n">cog</span><span class="o">/</span><span class="n">telechargement</span><span class="o">/</span><span class="mi">2013</span><span class="o">/</span><span class="n">dbf</span><span class="o">/</span><span class="n">historiq2013</span><span class="o">.</span><span class="n">zip</span>
+    <span class="n">INTO</span> <span class="n">postgresql</span><span class="p">:</span><span class="o">///</span><span class="n">pgloader</span>
+    <span class="n">WITH</span> <span class="n">truncate</span><span class="p">,</span> <span class="n">create</span> <span class="n">table</span>
+     <span class="n">SET</span> <span class="n">client_encoding</span> <span class="n">TO</span> <span class="s1">&#39;latin1&#39;</span><span class="p">;</span>
+</pre></div>
+</div>
+<p>Note that here pgloader will benefit from the meta-data information found in
+the dBase file to create a PostgreSQL table capable of hosting the data as
+described, then load the data.</p>
+</div>
+<div class="section" id="id7">
+<h3>Loading the data<a class="headerlink" href="#id7" title="Permalink to this headline">¶</a></h3>
+<p>Let’s start the <cite>pgloader</cite> command with our <cite>dbf-zip.load</cite> command file:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>$ pgloader dbf-zip.load
+... LOG Starting pgloader, log system is ready.
+... LOG Parsing commands from file &quot;/Users/dim/dev/pgloader/test/dbf-zip.load&quot;
+... LOG Fetching &#39;http://www.insee.fr/fr/methodes/nomenclatures/cog/telechargement/2013/dbf/historiq2013.zip&#39;
+... LOG Extracting files from archive &#39;//private/var/folders/w7/9n8v8pw54t1gngfff0lj16040000gn/T/pgloader//historiq2013.zip&#39;
+
+       table name       read   imported     errors            time
+-----------------  ---------  ---------  ---------  --------------
+         download          0          0          0          0.167s
+          extract          0          0          0          1.010s
+ create, truncate          0          0          0          0.071s
+-----------------  ---------  ---------  ---------  --------------
+     historiq2013       9181       9181          0          0.658s
+-----------------  ---------  ---------  ---------  --------------
+Total import time       9181       9181          0          1.906s
+</pre></div>
+</div>
+<p>We can see that <a class="reference external" href="http://pgloader.io">pgloader</a> did download the file from
+its HTTP URL location then <em>unziped</em> it before the loading itself.</p>
+<p>Note that the output of the command has been edited to facilitate its
+browsing online.</p>
+</div>
+</div>
+<div class="section" id="loading-sqlite-files-with-pgloader">
+<h2>Loading SQLite files with pgloader<a class="headerlink" href="#loading-sqlite-files-with-pgloader" title="Permalink to this headline">¶</a></h2>
+<p>The SQLite database is a respected solution to manage your data with. Its
+embeded nature makes it a source of migrations when a projects now needs to
+handle more concurrency, which [PostgreSQL](<a class="reference external" href="http://www.postgresql.org/">http://www.postgresql.org/</a>) is
+very good at. pgloader can help you there.</p>
+<div class="section" id="in-a-single-command-line">
+<h3>In a Single Command Line<a class="headerlink" href="#in-a-single-command-line" title="Permalink to this headline">¶</a></h3>
+<p>You can</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>$ createdb chinook
+$ pgloader https://github.com/lerocha/chinook-database/raw/master/ChinookDatabase/DataSources/Chinook_Sqlite_AutoIncrementPKs.sqlite pgsql:///chinook
+</pre></div>
+</div>
+<p>Done! All with the schema, data, constraints, primary keys and foreign keys,
+etc. We also see an error with the Chinook schema that contains several
+primary key definitions against the same table, which is not accepted by
+PostgreSQL:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="mi">2017</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">20</span><span class="n">T16</span><span class="p">:</span><span class="mi">18</span><span class="p">:</span><span class="mf">59.019000</span><span class="o">+</span><span class="mi">02</span><span class="p">:</span><span class="mi">00</span> <span class="n">LOG</span> <span class="n">Data</span> <span class="n">errors</span> <span class="ow">in</span> <span class="s1">&#39;/private/tmp/pgloader/&#39;</span>
+<span class="mi">2017</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">20</span><span class="n">T16</span><span class="p">:</span><span class="mi">18</span><span class="p">:</span><span class="mf">59.236000</span><span class="o">+</span><span class="mi">02</span><span class="p">:</span><span class="mi">00</span> <span class="n">LOG</span> <span class="n">Fetching</span> <span class="s1">&#39;https://github.com/lerocha/chinook-database/raw/master/ChinookDatabase/DataSources/Chinook_Sqlite_AutoIncrementPKs.sqlite&#39;</span>
+<span class="mi">2017</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">20</span><span class="n">T16</span><span class="p">:</span><span class="mi">19</span><span class="p">:</span><span class="mf">00.664000</span><span class="o">+</span><span class="mi">02</span><span class="p">:</span><span class="mi">00</span> <span class="n">ERROR</span> <span class="n">Database</span> <span class="n">error</span> <span class="mi">42</span><span class="n">P16</span><span class="p">:</span> <span class="n">multiple</span> <span class="n">primary</span> <span class="n">keys</span> <span class="k">for</span> <span class="n">table</span> <span class="s2">&quot;playlisttrack&quot;</span> <span class="n">are</span> <span class="ow">not</span> <span class="n">allowed</span>
+<span class="n">QUERY</span><span class="p">:</span> <span class="n">ALTER</span> <span class="n">TABLE</span> <span class="n">playlisttrack</span> <span class="n">ADD</span> <span class="n">PRIMARY</span> <span class="n">KEY</span> <span class="n">USING</span> <span class="n">INDEX</span> <span class="n">idx_66873_sqlite_autoindex_playlisttrack_1</span><span class="p">;</span>
+<span class="mi">2017</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">20</span><span class="n">T16</span><span class="p">:</span><span class="mi">19</span><span class="p">:</span><span class="mf">00.665000</span><span class="o">+</span><span class="mi">02</span><span class="p">:</span><span class="mi">00</span> <span class="n">LOG</span> <span class="n">report</span> <span class="n">summary</span> <span class="n">reset</span>
+             <span class="n">table</span> <span class="n">name</span>       <span class="n">read</span>   <span class="n">imported</span>     <span class="n">errors</span>      <span class="n">total</span> <span class="n">time</span>
+<span class="o">-----------------------</span>  <span class="o">---------</span>  <span class="o">---------</span>  <span class="o">---------</span>  <span class="o">--------------</span>
+                  <span class="n">fetch</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mf">0.877</span><span class="n">s</span>
+        <span class="n">fetch</span> <span class="n">meta</span> <span class="n">data</span>         <span class="mi">33</span>         <span class="mi">33</span>          <span class="mi">0</span>          <span class="mf">0.033</span><span class="n">s</span>
+         <span class="n">Create</span> <span class="n">Schemas</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mf">0.003</span><span class="n">s</span>
+       <span class="n">Create</span> <span class="n">SQL</span> <span class="n">Types</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mf">0.006</span><span class="n">s</span>
+          <span class="n">Create</span> <span class="n">tables</span>         <span class="mi">22</span>         <span class="mi">22</span>          <span class="mi">0</span>          <span class="mf">0.043</span><span class="n">s</span>
+         <span class="n">Set</span> <span class="n">Table</span> <span class="n">OIDs</span>         <span class="mi">11</span>         <span class="mi">11</span>          <span class="mi">0</span>          <span class="mf">0.012</span><span class="n">s</span>
+<span class="o">-----------------------</span>  <span class="o">---------</span>  <span class="o">---------</span>  <span class="o">---------</span>  <span class="o">--------------</span>
+                  <span class="n">album</span>        <span class="mi">347</span>        <span class="mi">347</span>          <span class="mi">0</span>          <span class="mf">0.023</span><span class="n">s</span>
+                 <span class="n">artist</span>        <span class="mi">275</span>        <span class="mi">275</span>          <span class="mi">0</span>          <span class="mf">0.023</span><span class="n">s</span>
+               <span class="n">customer</span>         <span class="mi">59</span>         <span class="mi">59</span>          <span class="mi">0</span>          <span class="mf">0.021</span><span class="n">s</span>
+               <span class="n">employee</span>          <span class="mi">8</span>          <span class="mi">8</span>          <span class="mi">0</span>          <span class="mf">0.018</span><span class="n">s</span>
+                <span class="n">invoice</span>        <span class="mi">412</span>        <span class="mi">412</span>          <span class="mi">0</span>          <span class="mf">0.031</span><span class="n">s</span>
+                  <span class="n">genre</span>         <span class="mi">25</span>         <span class="mi">25</span>          <span class="mi">0</span>          <span class="mf">0.021</span><span class="n">s</span>
+            <span class="n">invoiceline</span>       <span class="mi">2240</span>       <span class="mi">2240</span>          <span class="mi">0</span>          <span class="mf">0.034</span><span class="n">s</span>
+              <span class="n">mediatype</span>          <span class="mi">5</span>          <span class="mi">5</span>          <span class="mi">0</span>          <span class="mf">0.025</span><span class="n">s</span>
+          <span class="n">playlisttrack</span>       <span class="mi">8715</span>       <span class="mi">8715</span>          <span class="mi">0</span>          <span class="mf">0.040</span><span class="n">s</span>
+               <span class="n">playlist</span>         <span class="mi">18</span>         <span class="mi">18</span>          <span class="mi">0</span>          <span class="mf">0.016</span><span class="n">s</span>
+                  <span class="n">track</span>       <span class="mi">3503</span>       <span class="mi">3503</span>          <span class="mi">0</span>          <span class="mf">0.111</span><span class="n">s</span>
+<span class="o">-----------------------</span>  <span class="o">---------</span>  <span class="o">---------</span>  <span class="o">---------</span>  <span class="o">--------------</span>
+<span class="n">COPY</span> <span class="n">Threads</span> <span class="n">Completion</span>         <span class="mi">33</span>         <span class="mi">33</span>          <span class="mi">0</span>          <span class="mf">0.313</span><span class="n">s</span>
+         <span class="n">Create</span> <span class="n">Indexes</span>         <span class="mi">22</span>         <span class="mi">22</span>          <span class="mi">0</span>          <span class="mf">0.160</span><span class="n">s</span>
+ <span class="n">Index</span> <span class="n">Build</span> <span class="n">Completion</span>         <span class="mi">22</span>         <span class="mi">22</span>          <span class="mi">0</span>          <span class="mf">0.027</span><span class="n">s</span>
+        <span class="n">Reset</span> <span class="n">Sequences</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mf">0.017</span><span class="n">s</span>
+           <span class="n">Primary</span> <span class="n">Keys</span>         <span class="mi">12</span>          <span class="mi">0</span>          <span class="mi">1</span>          <span class="mf">0.013</span><span class="n">s</span>
+    <span class="n">Create</span> <span class="n">Foreign</span> <span class="n">Keys</span>         <span class="mi">11</span>         <span class="mi">11</span>          <span class="mi">0</span>          <span class="mf">0.040</span><span class="n">s</span>
+        <span class="n">Create</span> <span class="n">Triggers</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mf">0.000</span><span class="n">s</span>
+       <span class="n">Install</span> <span class="n">Comments</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mi">0</span>          <span class="mf">0.000</span><span class="n">s</span>
+<span class="o">-----------------------</span>  <span class="o">---------</span>  <span class="o">---------</span>  <span class="o">---------</span>  <span class="o">--------------</span>
+      <span class="n">Total</span> <span class="kn">import</span> <span class="nn">time</span>      <span class="mi">15607</span>      <span class="mi">15607</span>          <span class="mi">0</span>          <span class="mf">1.669</span><span class="n">s</span>
+</pre></div>
+</div>
+<p>You may need to have special cases to take care of tho. In advanced case you
+can use the pgloader command.</p>
+</div>
+<div class="section" id="id8">
+<h3>The Command<a class="headerlink" href="#id8" title="Permalink to this headline">¶</a></h3>
+<p>To load data with [pgloader](<a class="reference external" href="http://pgloader.io/">http://pgloader.io/</a>) you need to define in a
+<em>command</em> the operations in some details. Here’s our command:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">load</span> <span class="n">database</span>
+     <span class="kn">from</span> <span class="s1">&#39;sqlite/Chinook_Sqlite_AutoIncrementPKs.sqlite&#39;</span>
+     <span class="n">into</span> <span class="n">postgresql</span><span class="p">:</span><span class="o">///</span><span class="n">pgloader</span>
+
+ <span class="k">with</span> <span class="n">include</span> <span class="n">drop</span><span class="p">,</span> <span class="n">create</span> <span class="n">tables</span><span class="p">,</span> <span class="n">create</span> <span class="n">indexes</span><span class="p">,</span> <span class="n">reset</span> <span class="n">sequences</span>
+
+  <span class="nb">set</span> <span class="n">work_mem</span> <span class="n">to</span> <span class="s1">&#39;16MB&#39;</span><span class="p">,</span> <span class="n">maintenance_work_mem</span> <span class="n">to</span> <span class="s1">&#39;512 MB&#39;</span><span class="p">;</span>
+</pre></div>
+</div>
+<p>Note that here pgloader will benefit from the meta-data information found in
+the SQLite file to create a PostgreSQL database capable of hosting the data
+as described, then load the data.</p>
+</div>
+<div class="section" id="id9">
+<h3>Loading the data<a class="headerlink" href="#id9" title="Permalink to this headline">¶</a></h3>
+<p>Let’s start the <cite>pgloader</cite> command with our <cite>sqlite.load</cite> command file:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>$ pgloader sqlite.load
+... LOG Starting pgloader, log system is ready.
+... LOG Parsing commands from file &quot;/Users/dim/dev/pgloader/test/sqlite.load&quot;
+... WARNING Postgres warning: table &quot;album&quot; does not exist, skipping
+... WARNING Postgres warning: table &quot;artist&quot; does not exist, skipping
+... WARNING Postgres warning: table &quot;customer&quot; does not exist, skipping
+... WARNING Postgres warning: table &quot;employee&quot; does not exist, skipping
+... WARNING Postgres warning: table &quot;genre&quot; does not exist, skipping
+... WARNING Postgres warning: table &quot;invoice&quot; does not exist, skipping
+... WARNING Postgres warning: table &quot;invoiceline&quot; does not exist, skipping
+... WARNING Postgres warning: table &quot;mediatype&quot; does not exist, skipping
+... WARNING Postgres warning: table &quot;playlist&quot; does not exist, skipping
+... WARNING Postgres warning: table &quot;playlisttrack&quot; does not exist, skipping
+... WARNING Postgres warning: table &quot;track&quot; does not exist, skipping
+            table name       read   imported     errors            time
+----------------------  ---------  ---------  ---------  --------------
+      create, truncate          0          0          0          0.052s
+                 Album        347        347          0          0.070s
+                Artist        275        275          0          0.014s
+              Customer         59         59          0          0.014s
+              Employee          8          8          0          0.012s
+                 Genre         25         25          0          0.018s
+               Invoice        412        412          0          0.032s
+           InvoiceLine       2240       2240          0          0.077s
+             MediaType          5          5          0          0.012s
+              Playlist         18         18          0          0.008s
+         PlaylistTrack       8715       8715          0          0.071s
+                 Track       3503       3503          0          0.105s
+index build completion          0          0          0          0.000s
+----------------------  ---------  ---------  ---------  --------------
+        Create Indexes         20         20          0          0.279s
+       reset sequences          0          0          0          0.043s
+----------------------  ---------  ---------  ---------  --------------
+  Total streaming time      15607      15607          0          0.476s
+</pre></div>
+</div>
+<p>We can see that <a class="reference external" href="http://pgloader.io">pgloader</a> did download the file from
+its HTTP URL location then <em>unziped</em> it before loading it.</p>
+<p>Also, the <em>WARNING</em> messages we see here are expected as the PostgreSQL
+database is empty when running the command, and pgloader is using the SQL
+commands <cite>DROP TABLE IF EXISTS</cite> when the given command uses the <cite>include
+drop</cite> option.</p>
+<p>Note that the output of the command has been edited to facilitate its
+browsing online.</p>
+</div>
+</div>
+<div class="section" id="migrating-from-mysql-to-postgresql">
+<h2>Migrating from MySQL to PostgreSQL<a class="headerlink" href="#migrating-from-mysql-to-postgresql" title="Permalink to this headline">¶</a></h2>
+<p>If you want to migrate your data over to <a class="reference external" href="http://www.postgresql.org">PostgreSQL</a> from MySQL then pgloader is the tool of
+choice!</p>
+<p>Most tools around are skipping the main problem with migrating from MySQL,
+which is to do with the type casting and data sanitizing that needs to be
+done. pgloader will not leave you alone on those topics.</p>
+<div class="section" id="id11">
+<h3>In a Single Command Line<a class="headerlink" href="#id11" title="Permalink to this headline">¶</a></h3>
+<p>As an example, we will use the f1db database from &lt;<a class="reference external" href="http://ergast.com/mrd/">http://ergast.com/mrd/</a>&gt;
+which which provides a historical record of motor racing data for
+non-commercial purposes. You can either use their API or download the whole
+database at <a class="reference external" href="http://ergast.com/downloads/f1db.sql.gz">http://ergast.com/downloads/f1db.sql.gz</a>. Once you’ve done that load the
+database in MySQL:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>$ mysql -u root
+&gt; create database f1db;
+&gt; source f1db.sql
+</pre></div>
+</div>
+<p>Now let’s migrate this database into PostgreSQL in a single command line:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>$ createdb f1db
+$ pgloader mysql://root@localhost/f1db pgsql:///f1db
+</pre></div>
+</div>
+<p>Done! All with schema, table definitions, constraints, indexes, primary
+keys, <em>auto_increment</em> columns turned into <em>bigserial</em> , foreign keys,
+comments, and if you had some MySQL default values such as <em>ON UPDATE
+CURRENT_TIMESTAMP</em> they would have been translated to a <a class="reference external" href="https://www.postgresql.org/docs/current/static/plpgsql-trigger.html">PostgreSQL before
+update trigger</a>
+automatically.</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>$ pgloader mysql://root@localhost/f1db pgsql:///f1db
+2017-06-16T08:56:14.064000+02:00 LOG Main logs in &#39;/private/tmp/pgloader/pgloader.log&#39;
+2017-06-16T08:56:14.068000+02:00 LOG Data errors in &#39;/private/tmp/pgloader/&#39;
+2017-06-16T08:56:19.542000+02:00 LOG report summary reset
+               table name       read   imported     errors      total time
+-------------------------  ---------  ---------  ---------  --------------
+          fetch meta data         33         33          0          0.365s
+           Create Schemas          0          0          0          0.007s
+         Create SQL Types          0          0          0          0.006s
+            Create tables         26         26          0          0.068s
+           Set Table OIDs         13         13          0          0.012s
+-------------------------  ---------  ---------  ---------  --------------
+  f1db.constructorresults      11011      11011          0          0.205s
+            f1db.circuits         73         73          0          0.150s
+        f1db.constructors        208        208          0          0.059s
+f1db.constructorstandings      11766      11766          0          0.365s
+             f1db.drivers        841        841          0          0.268s
+            f1db.laptimes     413578     413578          0          2.892s
+     f1db.driverstandings      31420      31420          0          0.583s
+            f1db.pitstops       5796       5796          0          2.154s
+               f1db.races        976        976          0          0.227s
+          f1db.qualifying       7257       7257          0          0.228s
+             f1db.seasons         68         68          0          0.527s
+             f1db.results      23514      23514          0          0.658s
+              f1db.status        133        133          0          0.130s
+-------------------------  ---------  ---------  ---------  --------------
+  COPY Threads Completion         39         39          0          4.303s
+           Create Indexes         20         20          0          1.497s
+   Index Build Completion         20         20          0          0.214s
+          Reset Sequences          0         10          0          0.058s
+             Primary Keys         13         13          0          0.012s
+      Create Foreign Keys          0          0          0          0.000s
+          Create Triggers          0          0          0          0.001s
+         Install Comments          0          0          0          0.000s
+-------------------------  ---------  ---------  ---------  --------------
+        Total import time     506641     506641          0          5.547s
+</pre></div>
+</div>
+<p>You may need to have special cases to take care of tho, or views that you
+want to materialize while doing the migration. In advanced case you can use
+the pgloader command.</p>
+</div>
+<div class="section" id="id12">
+<h3>The Command<a class="headerlink" href="#id12" title="Permalink to this headline">¶</a></h3>
+<p>To load data with pgloader you need to define in a <em>command</em> the operations
+in some details. Here’s our example for loading the <a class="reference external" href="http://dev.mysql.com/doc/sakila/en/">MySQL Sakila Sample
+Database</a>.</p>
+<p>Here’s our command:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>load database
+     from      mysql://root@localhost/sakila
+     into postgresql:///sakila
+
+ WITH include drop, create tables, no truncate,
+      create indexes, reset sequences, foreign keys
+
+  SET maintenance_work_mem to &#39;128MB&#39;, work_mem to &#39;12MB&#39;, search_path to &#39;sakila&#39;
+
+ CAST type datetime to timestamptz
+                drop default drop not null using zero-dates-to-null,
+      type date drop not null drop default using zero-dates-to-null
+
+ MATERIALIZE VIEWS film_list, staff_list
+
+ -- INCLUDING ONLY TABLE NAMES MATCHING ~/film/, &#39;actor&#39;
+ -- EXCLUDING TABLE NAMES MATCHING ~&lt;ory&gt;
+
+ BEFORE LOAD DO
+ $$ create schema if not exists sakila; $$;
+</pre></div>
+</div>
+<p>Note that here pgloader will benefit from the meta-data information found in
+the MySQL database to create a PostgreSQL database capable of hosting the
+data as described, then load the data.</p>
+<p>In particular, some specific <em>casting rules</em> are given here, to cope with
+date values such as <cite>0000-00-00</cite> that MySQL allows and PostgreSQL rejects
+for not existing in our calendar. It’s possible to add per-column casting
+rules too, which is useful is some of your <cite>tinyint</cite> are in fact <cite>smallint</cite>
+while some others are in fact <cite>boolean</cite> values.</p>
+<p>Finaly note that we are using the <em>MATERIALIZE VIEWS</em> clause of pgloader:
+the selected views here will be migrated over to PostgreSQL <em>with their
+contents</em>.</p>
+<p>It’s possible to use the <em>MATERIALIZE VIEWS</em> clause and give both the name
+and the SQL (in MySQL dialect) definition of view, then pgloader creates the
+view before loading the data, then drops it again at the end.</p>
+<p>## Loading the data</p>
+<p>Let’s start the <cite>pgloader</cite> command with our <cite>sakila.load</cite> command file:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span>$ pgloader sakila.load
+... LOG Starting pgloader, log system is ready.
+... LOG Parsing commands from file &quot;/Users/dim/dev/pgloader/test/sakila.load&quot;
+   &lt;WARNING: table &quot;xxx&quot; does not exists have been edited away&gt;
+
+            table name       read   imported     errors            time
+----------------------  ---------  ---------  ---------  --------------
+           before load          1          1          0          0.007s
+       fetch meta data         45         45          0          0.402s
+          create, drop          0         36          0          0.208s
+----------------------  ---------  ---------  ---------  --------------
+                 actor        200        200          0          0.071s
+               address        603        603          0          0.035s
+              category         16         16          0          0.018s
+                  city        600        600          0          0.037s
+               country        109        109          0          0.023s
+              customer        599        599          0          0.073s
+                  film       1000       1000          0          0.135s
+            film_actor       5462       5462          0          0.236s
+         film_category       1000       1000          0          0.070s
+             film_text       1000       1000          0          0.080s
+             inventory       4581       4581          0          0.136s
+              language          6          6          0          0.036s
+               payment      16049      16049          0          0.539s
+                rental      16044      16044          0          0.648s
+                 staff          2          2          0          0.041s
+                 store          2          2          0          0.036s
+             film_list        997        997          0          0.247s
+            staff_list          2          2          0          0.135s
+Index Build Completion          0          0          0          0.000s
+----------------------  ---------  ---------  ---------  --------------
+        Create Indexes         41         41          0          0.964s
+       Reset Sequences          0          1          0          0.035s
+          Foreign Keys         22         22          0          0.254s
+----------------------  ---------  ---------  ---------  --------------
+     Total import time      48272      48272          0          3.502s
+</pre></div>
+</div>
+<p>The <em>WARNING</em> messages we see here are expected as the PostgreSQL database
+is empty when running the command, and pgloader is using the SQL commands
+<cite>DROP TABLE IF EXISTS</cite> when the given command uses the <cite>include drop</cite>
+option.</p>
+<p>Note that the output of the command has been edited to facilitate its
+browsing online.</p>
+</div>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper"><div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="../index.html">Documentation overview</a><ul>
+      <li>Previous: <a href="../intro.html" title="previous chapter">Introduction</a></li>
+      <li>Next: <a href="../pgloader.html" title="next chapter">PgLoader Reference Manual</a></li>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="footer">
+      &copy;2017, Dimitri Fontaine.
+      
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.5</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+      
+      |
+      <a href="../_sources/tutorial/tutorial.rst.txt"
+          rel="nofollow">Page source</a>
+    </div>
+
+    
+
+    
+  </body>
+</html>

docs/bugreport.rst

@@ -0,0 +1,49 @@
+Reporting Bugs
+==============
+
+pgloader is a software and as such contains bugs. Most bugs are easy to
+solve and taken care of in a short delay. For this to be possible though,
+bug reports need to follow those recommandations:
+
+  - include pgloader version,
+  - include problematic input and output,
+  - include a description of the output you expected,
+  - explain the difference between the ouput you have and the one you expected,
+  - include a self-reproducing test-case
+
+Test Cases to Reproduce Bugs
+----------------------------
+
+Use the *inline* source type to help reproduce a bug, as in the pgloader tests::
+
+  LOAD CSV
+     FROM INLINE
+     INTO postgresql://dim@localhost/pgloader?public."HS"
+
+     WITH truncate,
+          fields terminated by '\t',
+          fields not enclosed,
+          fields escaped by backslash-quote,
+          quote identifiers
+
+      SET work_mem to '128MB',
+          standard_conforming_strings to 'on',
+          application_name to 'my app name'
+
+   BEFORE LOAD DO
+    $$ create extension if not exists hstore; $$,
+    $$ drop table if exists "HS"; $$,
+    $$ CREATE TABLE "HS"
+       (
+          id     serial primary key,
+          kv     hstore
+       )
+    $$;
+  
+  
+  1	email=>foo@example.com,a=>b
+  2	test=>value
+  3	a=>b,c=>"quoted hstore value",d=>other
+  4	baddata
+
+

docs/conf.py

@@ -0,0 +1,177 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+#
+# pgloader documentation build configuration file, created by
+# sphinx-quickstart on Tue Dec  5 19:23:32 2017.
+#
+# This file is execfile()d with the current directory set to its
+# containing dir.
+#
+# Note that not all possible configuration values are present in this
+# autogenerated file.
+#
+# All configuration values have a default; values that are commented out
+# serve to show the default.
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+# import os
+# import sys
+# sys.path.insert(0, os.path.abspath('.'))
+
+
+# -- General configuration ------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+#
+# needs_sphinx = '1.0'
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = ['sphinx.ext.githubpages']
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+#
+# source_suffix = '.rst'
+source_suffix = ['.rst', '.md']
+
+# The master toctree document.
+master_doc = 'index'
+
+# General information about the project.
+project = 'pgloader'
+copyright = '2017, Dimitri Fontaine'
+author = 'Dimitri Fontaine'
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the
+# built documents.
+#
+# The short X.Y version.
+version = '3.4'
+# The full version, including alpha/beta/rc tags.
+release = '3.4.1'
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = None
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This patterns also effect to html_static_path and html_extra_path
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = 'sphinx'
+
+# If true, `todo` and `todoList` produce output, else they produce nothing.
+todo_include_todos = False
+
+
+# -- Options for HTML output ----------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'alabaster'
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+#
+# html_theme_options = {}
+html_theme_options = {
+    'github_user': 'dimitri',
+    'github_repo': 'pgloader',
+    'description': 'your migration companion',
+    'travis_button': True,
+    'show_related': True,
+    'sidebar_collapse': False,
+}
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+
+# Custom sidebar templates, must be a dictionary that maps document names
+# to template names.
+#
+# This is required for the alabaster theme
+# refs: http://alabaster.readthedocs.io/en/latest/installation.html#sidebars
+html_sidebars = {
+    '**': [
+        'relations.html',  # needs 'show_related': True theme option to display
+        'searchbox.html',
+    ]
+}
+
+
+# -- Options for HTMLHelp output ------------------------------------------
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = 'pgloaderdoc'
+
+
+# -- Options for LaTeX output ---------------------------------------------
+
+latex_elements = {
+    # The paper size ('letterpaper' or 'a4paper').
+    #
+    # 'papersize': 'letterpaper',
+
+    # The font size ('10pt', '11pt' or '12pt').
+    #
+    # 'pointsize': '10pt',
+
+    # Additional stuff for the LaTeX preamble.
+    #
+    # 'preamble': '',
+
+    # Latex figure (float) alignment
+    #
+    # 'figure_align': 'htbp',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+#  author, documentclass [howto, manual, or own class]).
+latex_documents = [
+    (master_doc, 'pgloader.tex', 'pgloader Documentation',
+     'Dimitri Fontaine', 'manual'),
+]
+
+
+# -- Options for manual page output ---------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+    (master_doc, 'pgloader', 'pgloader Documentation',
+     [author], 1)
+]
+
+
+# -- Options for Texinfo output -------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+#  dir menu entry, description, category)
+texinfo_documents = [
+    (master_doc, 'pgloader', 'pgloader Documentation',
+     author, 'pgloader', 'One line description of project.',
+     'Miscellaneous'),
+]
+
+
+