mirror of
https://github.com/dimitri/pgloader.git
synced 2026-05-04 18:36:12 +02:00
Improve pgloader usage from the command line.
Make it so that the following command line usages are accepted when
using pgloader without a command file:
./build/bin/pgloader ./test/sqlite/sqlite.db postgresql:///pgloader
./build/bin/pgloader --set "search_path='sakila'" \
mysql://root@localhost/sakila \
postgresql:///sakila
./build/bin/pgloader --type csv \
--field id --field field \
--with truncate \
--with "fields terminated by ','" \
./test/data/matching-1.csv \
postgres:///pgloader?matching
It's now possible in most cases to just use command-line options, which
should make the entry bar to pgloader much lower.
This commit is contained in:
Dimitri Fontaine
parent
34b9a908b0
commit
65c2043694
84
pgloader.1
84
pgloader.1
@ -11,6 +11,7 @@
|
||||
.nf
|
||||
|
||||
pgloader [<options>] [<command\-file>]\.\.\.
|
||||
pgloader [<options>] SOURCE TARGET
|
||||
.
|
||||
.fi
|
||||
.
|
||||
@ -18,7 +19,7 @@ pgloader [<options>] [<command\-file>]\.\.\.
|
||||
pgloader loads data from various sources into PostgreSQL\. It can transform the data it reads on the fly and submit raw SQL before and after the loading\. It uses the \fBCOPY\fR PostgreSQL protocol to stream the data into the server, and manages errors by filling a pair of \fIreject\.dat\fR and \fIreject\.log\fR files\.
|
||||
.
|
||||
.P
|
||||
pgloader operates using commands which are read from files:
|
||||
pgloader operates either using commands which are read from files:
|
||||
.
|
||||
.IP "" 4
|
||||
.
|
||||
@ -30,8 +31,24 @@ pgloader commands\.load
|
||||
.
|
||||
.IP "" 0
|
||||
.
|
||||
.P
|
||||
or by using arguments and options all provided on the command line:
|
||||
.
|
||||
.IP "" 4
|
||||
.
|
||||
.nf
|
||||
|
||||
pgloader SOURCE TARGET
|
||||
.
|
||||
.fi
|
||||
.
|
||||
.IP "" 0
|
||||
.
|
||||
.SH "OPTIONS"
|
||||
.
|
||||
.SS "INQUIRY OPTIONS"
|
||||
Use these options when you want to know more about how to use \fBpgloader\fR, as those options will cause \fBpgloader\fR not to load any data\.
|
||||
.
|
||||
.TP
|
||||
\fB\-h\fR, \fB\-\-help\fR
|
||||
Show command usage summary and exit\.
|
||||
@ -41,6 +58,17 @@ Show command usage summary and exit\.
|
||||
Show pgloader version string and exit\.
|
||||
.
|
||||
.TP
|
||||
\fB\-E\fR, \fB\-\-list\-encodings\fR
|
||||
List known encodings in this version of pgloader\.
|
||||
.
|
||||
.TP
|
||||
\fB\-U\fR, \fB\-\-upgrade\-config\fR
|
||||
Parse given files in the command line as \fBpgloader\.conf\fR files with the \fBINI\fR syntax that was in use in pgloader versions 2\.x, and output the new command syntax for pgloader on standard output\.
|
||||
.
|
||||
.SS "GENERAL OPTIONS"
|
||||
Those options are meant to tweak \fBpgloader\fR behavior when loading data\.
|
||||
.
|
||||
.TP
|
||||
\fB\-v\fR, \fB\-\-verbose\fR
|
||||
Be verbose\.
|
||||
.
|
||||
@ -73,14 +101,6 @@ Minimum level of verbosity needed for log message to make it to the console\. On
|
||||
A filename where to copy the summary output\. When relative, the filename is expanded into \fB*root\-dir\fR\.
|
||||
.
|
||||
.TP
|
||||
\fB\-E\fR, \fB\-\-list\-encodings\fR
|
||||
List known encodings in this version of pgloader\.
|
||||
.
|
||||
.TP
|
||||
\fB\-U\fR, \fB\-\-upgrade\-config\fR
|
||||
Parse given files in the command line as \fBpgloader\.conf\fR files with the \fBINI\fR syntax that was in use in pgloader versions 2\.x, and output the new command syntax for pgloader on standard output\.
|
||||
.
|
||||
.TP
|
||||
\fB\-l <file>\fR, \fB\-\-load\-lisp\-file <file>\fR
|
||||
Specify a lisp \fIfile\fR to compile and load into the pgloader image before reading the commands, allowing to define extra transformation function\. Those functions should be defined in the \fBpgloader\.transforms\fR package\. This option can appear more than once in the command line\.
|
||||
.
|
||||
@ -90,7 +110,48 @@ Specify a lisp \fIfile\fR to compile and load into the pgloader image before rea
|
||||
.IP
|
||||
Specify a \fIdirectory\fR where to find pgloader sources so that one of the very first things it does is dynamically loading\-in (and compiling to machine code) another version of itself, usually a newer one like a very recent git checkout\.
|
||||
.
|
||||
.P
|
||||
.SS "NO COMMAND FILE OPTIONS"
|
||||
Those options are meant to be used when using \fBpgloader\fR from the command line only, rather than using a command file and the rich command clauses and parser\. In simple cases, it can be much easier to use the \fISOURCE\fR and \fITARGET\fR directly on the command line, then tweak the loading with those options:
|
||||
.
|
||||
.IP "\(bu" 4
|
||||
\fB\-\-with "option"\fR:
|
||||
.
|
||||
.IP
|
||||
Allows setting options from the command line\. You can use that option as many times as you want\. The option arguments must follow the \fIWITH\fR clause for the source type of the \fBSOURCE\fR specification, as described later in this document\.
|
||||
.
|
||||
.IP "\(bu" 4
|
||||
\fB\-\-set "guc_name=\'value\'"\fR
|
||||
.
|
||||
.IP
|
||||
Allows setting PostgreSQL configuration from the command line\. Note that the option parsing is the same as when used from the \fISET\fR command clause, in particular you must enclose the guc value with single\-quotes\.
|
||||
.
|
||||
.IP "\(bu" 4
|
||||
\fB\-\-field "\.\.\."\fR
|
||||
.
|
||||
.IP
|
||||
Allows setting a source field definition\. Fields are accumulated in the order given on the command line\. It\'s then required to use a \fB\-\-field\fR option per field in the source file\.
|
||||
.
|
||||
.IP "\(bu" 4
|
||||
\fB\-\-cast "\.\.\."\fR
|
||||
.
|
||||
.IP
|
||||
Allows setting a specific casting rule for loading the data\.
|
||||
.
|
||||
.IP "\(bu" 4
|
||||
\fB\-\-type csv|fixed|db3|ixf|sqlite|mysql|mssql\fR
|
||||
.
|
||||
.IP
|
||||
Allows forcing the source type, in case when the \fISOURCE\fR parsing isn\'t satisfying\.
|
||||
.
|
||||
.IP "\(bu" 4
|
||||
\fB\-\-encoding <encoding>\fR
|
||||
.
|
||||
.IP
|
||||
Set the encoding of the source file to load data from\.
|
||||
.
|
||||
.IP "" 0
|
||||
.
|
||||
.SS "MORE DEBUG INFORMATION"
|
||||
To get the maximum amount of debug information, you can use both the \fB\-\-verbose\fR and the \fB\-\-debug\fR switches at the same time, which is equivalent to saying \fB\-\-client\-min\-messages data\fR\. Then the log messages will show the data being processed, in the cases where the code has explicit support for it\.
|
||||
.
|
||||
.SH "BATCHES AND RETRY BEHAVIOUR"
|
||||
@ -1561,6 +1622,9 @@ type int to int when (< precision 10)
|
||||
type int to bigint when (<= 10 precision)
|
||||
.
|
||||
.IP "\(bu" 4
|
||||
type tinyint with extra auto_increment to serial
|
||||
.
|
||||
.IP "\(bu" 4
|
||||
type smallint with extra auto_increment to serial
|
||||
.
|
||||
.IP "\(bu" 4
|
||||
|
||||
@ -3,6 +3,7 @@
|
||||
## SYNOPSIS
|
||||
|
||||
pgloader [<options>] [<command-file>]...
|
||||
pgloader [<options>] SOURCE TARGET
|
||||
|
||||
## DESCRIPTION
|
||||
|
||||
@ -12,18 +13,40 @@ after the loading. It uses the `COPY` PostgreSQL protocol to stream
|
||||
the data into the server, and manages errors by filling a pair of
|
||||
*reject.dat* and *reject.log* files.
|
||||
|
||||
pgloader operates using commands which are read from files:
|
||||
pgloader operates either using commands which are read from files:
|
||||
|
||||
pgloader commands.load
|
||||
|
||||
or by using arguments and options all provided on the command line:
|
||||
|
||||
pgloader SOURCE TARGET
|
||||
|
||||
## OPTIONS
|
||||
|
||||
### INQUIRY OPTIONS
|
||||
|
||||
Use these options when you want to know more about how to use `pgloader`, as
|
||||
those options will cause `pgloader` not to load any data.
|
||||
|
||||
* `-h`, `--help`:
|
||||
Show command usage summary and exit.
|
||||
|
||||
* `-V`, `--version`:
|
||||
Show pgloader version string and exit.
|
||||
|
||||
* `-E`, `--list-encodings`:
|
||||
List known encodings in this version of pgloader.
|
||||
|
||||
* `-U`, `--upgrade-config`:
|
||||
Parse given files in the command line as `pgloader.conf` files with the
|
||||
`INI` syntax that was in use in pgloader versions 2.x, and output the
|
||||
new command syntax for pgloader on standard output.
|
||||
|
||||
|
||||
### GENERAL OPTIONS
|
||||
|
||||
Those options are meant to tweak `pgloader` behavior when loading data.
|
||||
|
||||
* `-v`, `--verbose`:
|
||||
Be verbose.
|
||||
|
||||
@ -51,14 +74,6 @@ pgloader operates using commands which are read from files:
|
||||
A filename where to copy the summary output. When relative, the filename
|
||||
is expanded into `*root-dir`.
|
||||
|
||||
* `-E`, `--list-encodings`:
|
||||
List known encodings in this version of pgloader.
|
||||
|
||||
* `-U`, `--upgrade-config`:
|
||||
Parse given files in the command line as `pgloader.conf` files with the
|
||||
`INI` syntax that was in use in pgloader versions 2.x, and output the
|
||||
new command syntax for pgloader on standard output.
|
||||
|
||||
* `-l <file>`, `--load-lisp-file <file>`:
|
||||
Specify a lisp <file> to compile and load into the pgloader image before
|
||||
reading the commands, allowing to define extra transformation function.
|
||||
@ -72,6 +87,48 @@ pgloader operates using commands which are read from files:
|
||||
machine code) another version of itself, usually a newer one like a very
|
||||
recent git checkout.
|
||||
|
||||
### NO COMMAND FILE OPTIONS
|
||||
|
||||
Those options are meant to be used when using `pgloader` from the command
|
||||
line only, rather than using a command file and the rich command clauses and
|
||||
parser. In simple cases, it can be much easier to use the *SOURCE* and
|
||||
*TARGET* directly on the command line, then tweak the loading with those
|
||||
options:
|
||||
|
||||
* `--with "option"`:
|
||||
|
||||
Allows setting options from the command line. You can use that option as
|
||||
many times as you want. The option arguments must follow the *WITH*
|
||||
clause for the source type of the `SOURCE` specification, as described
|
||||
later in this document.
|
||||
|
||||
* `--set "guc_name='value'"`
|
||||
|
||||
Allows setting PostgreSQL configuration from the command line. Note that
|
||||
the option parsing is the same as when used from the *SET* command
|
||||
clause, in particular you must enclose the guc value with single-quotes.
|
||||
|
||||
* `--field "..."`
|
||||
|
||||
Allows setting a source field definition. Fields are accumulated in the
|
||||
order given on the command line. It's then required to use a `--field`
|
||||
option per field in the source file.
|
||||
|
||||
* `--cast "..."`
|
||||
|
||||
Allows setting a specific casting rule for loading the data.
|
||||
|
||||
* `--type csv|fixed|db3|ixf|sqlite|mysql|mssql`
|
||||
|
||||
Allows forcing the source type, in case when the *SOURCE* parsing isn't
|
||||
satisfying.
|
||||
|
||||
* `--encoding <encoding>`
|
||||
|
||||
Set the encoding of the source file to load data from.
|
||||
|
||||
### MORE DEBUG INFORMATION
|
||||
|
||||
To get the maximum amount of debug information, you can use both the
|
||||
`--verbose` and the `--debug` switches at the same time, which is equivalent
|
||||
to saying `--client-min-messages data`. Then the log messages will show the
|
||||
|
||||
181
src/main.lisp
181
src/main.lisp
@ -4,7 +4,7 @@
|
||||
"Return the internal value to use given the script parameters."
|
||||
(cond ((and debug verbose) :data)
|
||||
(debug :debug)
|
||||
(verbose :info)
|
||||
(verbose :notice)
|
||||
(quiet :warning)
|
||||
(t (or (find-symbol (string-upcase min-message) "KEYWORD")
|
||||
:notice))))
|
||||
@ -42,6 +42,24 @@
|
||||
(("load-lisp-file" #\l) :type string :list t :optional t
|
||||
:documentation "Read user code from files")
|
||||
|
||||
(("with") :type string :list t :optional t
|
||||
:documentation "Load options")
|
||||
|
||||
(("set") :type string :list t :optional t
|
||||
:documentation "PostgreSQL options")
|
||||
|
||||
(("field") :type string :list t :optional t
|
||||
:documentation "Source file fields specification")
|
||||
|
||||
(("cast") :type string :list t :optional t
|
||||
:documentation "Specific cast rules")
|
||||
|
||||
(("type") :type string :optional t
|
||||
:documentation "Force input source type")
|
||||
|
||||
(("encoding") :type string :optional t
|
||||
:documentation "Source expected encoding")
|
||||
|
||||
("self-upgrade" :type string :optional t
|
||||
:documentation "Path to pgloader newer sources")))
|
||||
|
||||
@ -83,7 +101,8 @@
|
||||
|
||||
(defun usage (argv &key quit)
|
||||
"Show usage then QUIT if asked to."
|
||||
(format t "~a [ option ... ] command-file ..." (first argv))
|
||||
(format t "~&~a [ option ... ] command-file ..." (first argv))
|
||||
(format t "~&~a [ option ... ] SOURCE TARGET" (first argv))
|
||||
(command-line-arguments:show-option-help *opt-spec*)
|
||||
(when quit (uiop:quit)))
|
||||
|
||||
@ -142,14 +161,16 @@
|
||||
(command-line-arguments:process-command-line-options *opt-spec* args)
|
||||
(condition (e)
|
||||
;; print out the usage, whatever happens here
|
||||
(declare (ignore e))
|
||||
;; (declare (ignore e))
|
||||
(format t "~a~%" e)
|
||||
(usage argv :quit t)))
|
||||
|
||||
(destructuring-bind (&key help version quiet verbose debug logfile
|
||||
list-encodings upgrade-config
|
||||
((:load-lisp-file load))
|
||||
client-min-messages log-min-messages summary
|
||||
root-dir self-upgrade)
|
||||
root-dir self-upgrade
|
||||
with set field cast type encoding)
|
||||
options
|
||||
|
||||
;; First thing: Self Upgrade?
|
||||
@ -229,40 +250,136 @@
|
||||
;; Now process the arguments
|
||||
(when arguments
|
||||
;; Start the logs system
|
||||
(let ((*log-filename* (log-file-name logfile)))
|
||||
(let ((*log-filename* (log-file-name logfile))
|
||||
(*summary-pathname* (parse-summary-filename summary debug)))
|
||||
|
||||
(with-monitor ()
|
||||
;; tell the user where to look for interesting things
|
||||
(log-message :log "Main logs in '~a'" (probe-file *log-filename*))
|
||||
(log-message :log "Data errors in '~a'~%" *root-dir*)
|
||||
|
||||
;; process the files
|
||||
(loop for filename in arguments
|
||||
do
|
||||
;; The handler-case is to catch unhandled exceptions at the
|
||||
;; top level and continue with the next file in the list.
|
||||
;;
|
||||
;; The handler-bind is to be able to offer a meaningful
|
||||
;; backtrace to the user in case of unexpected conditions
|
||||
;; being signaled.
|
||||
(handler-case
|
||||
(handler-bind
|
||||
((condition
|
||||
#'(lambda (condition)
|
||||
(log-message :fatal "We have a situation here.")
|
||||
(print-backtrace condition debug *standard-output*))))
|
||||
(let ((truename (probe-file filename))
|
||||
(summary-pathname
|
||||
(parse-summary-filename summary debug)))
|
||||
(if truename
|
||||
(run-commands truename
|
||||
:summary summary-pathname
|
||||
:start-logger nil)
|
||||
(log-message :error "Can not find file: ~s" filename)))
|
||||
(format t "~&"))
|
||||
(handler-case
|
||||
;; The handler-case is to catch unhandled exceptions at the
|
||||
;; top level.
|
||||
;;
|
||||
;; The handler-bind is to be able to offer a meaningful
|
||||
;; backtrace to the user in case of unexpected conditions
|
||||
;; being signaled.
|
||||
(handler-bind
|
||||
((condition
|
||||
#'(lambda (condition)
|
||||
(log-message :fatal "We have a situation here.")
|
||||
(print-backtrace condition debug *standard-output*))))
|
||||
|
||||
(condition (c)
|
||||
(when debug (invoke-debugger c))
|
||||
(uiop:quit 1)))))))
|
||||
;; if there are exactly two arguments in the command
|
||||
;; line, try and process them as source and target
|
||||
;; arguments
|
||||
(if (= 2 (length arguments))
|
||||
(let ((type (parse-cli-type type))
|
||||
(source (first arguments))
|
||||
(target (parse-target-string (second arguments))))
|
||||
|
||||
(multiple-value-bind (type source)
|
||||
(if type
|
||||
(parse-source-string-for-type type source)
|
||||
(parse-source-string source))
|
||||
|
||||
;; some verbosity about the parsing "magic"
|
||||
(log-message :info "SOURCE: ~s" source)
|
||||
(log-message :info "TARGET: ~s" target)
|
||||
|
||||
(when (and (null source) (null target))
|
||||
;; we might actually have just 2 load files to
|
||||
;; process
|
||||
(mapcar #'process-command-file arguments))
|
||||
|
||||
;; so, we actually have all the specs for the
|
||||
;; job on the command line now.
|
||||
(let ((type (or type (getf source :type))))
|
||||
(load-data :from source
|
||||
:into target
|
||||
:encoding (parse-cli-encoding encoding)
|
||||
:options (parse-cli-options type with)
|
||||
:gucs (parse-cli-gucs set)
|
||||
:type type
|
||||
:fields (parse-cli-fields type field)
|
||||
:casts (parse-cli-casts cast)))))
|
||||
|
||||
;; process the files
|
||||
(mapcar #'process-command-file arguments)))
|
||||
|
||||
(condition (c)
|
||||
(when debug (invoke-debugger c))
|
||||
(uiop:quit 1))))))
|
||||
|
||||
;; done.
|
||||
(uiop:quit)))))
|
||||
|
||||
(defun process-command-file (filename)
|
||||
"Process FILENAME as a pgloader command file (.load)."
|
||||
(let ((truename (probe-file filename)))
|
||||
(if truename
|
||||
(run-commands truename :start-logger nil)
|
||||
(log-message :error "Can not find file: ~s" filename)))
|
||||
(format t "~&"))
|
||||
|
||||
|
||||
;;;
|
||||
;;; Main API to use from outside of pgloader.
|
||||
;;;
|
||||
(defun load-data (&key ((:from source)) ((:into target))
|
||||
(type (getf source :type))
|
||||
encoding fields options gucs casts start-logger)
|
||||
"Load data from SOURCE into TARGET."
|
||||
(with-monitor (:start-logger start-logger)
|
||||
;; some preliminary checks
|
||||
(when (and (eq source :stdin) (null type))
|
||||
(log-message :fatal "When loading from stdin, option --type is mandatory.")
|
||||
(return-from load-data))
|
||||
|
||||
(when (and (member type '(:csv :fixed)) (null fields))
|
||||
(log-message :fatal "Source type ~a requires --fields arguments." type)
|
||||
(return-from load-data))
|
||||
|
||||
(when (and casts (not (member type '(:sqlite :mysql :mssql))))
|
||||
(log-message :log "Cast rules are not considered for ~s sources." type))
|
||||
|
||||
;; now generates the code for the command
|
||||
(log-message :debug "LOAD DATA ~a FROM ~s" type source)
|
||||
(run-commands
|
||||
(process-relative-pathnames
|
||||
(uiop:getcwd)
|
||||
(case type
|
||||
(:csv (lisp-code-for-loading-from-csv source fields target
|
||||
:encoding encoding
|
||||
:gucs gucs
|
||||
:csv-options options))
|
||||
|
||||
(:fixed (lisp-code-for-loading-from-fixed source fields target
|
||||
:encoding encoding
|
||||
:gucs gucs
|
||||
:fixed-options options))
|
||||
|
||||
(:db3 (lisp-code-for-loading-from-dbf source target
|
||||
:gucs gucs
|
||||
:dbf-options options))
|
||||
|
||||
(:ixf (lisp-code-for-loading-from-ixf source target
|
||||
:gucs gucs
|
||||
:ixf-options options))
|
||||
|
||||
(:sqlite (lisp-code-for-loading-from-sqlite source target
|
||||
:gucs gucs
|
||||
:casts casts
|
||||
:sqlite-options options))
|
||||
|
||||
(:mysql (lisp-code-for-loading-from-mysql source target
|
||||
:gucs gucs
|
||||
:casts casts
|
||||
:mysql-options options))
|
||||
|
||||
(:mssql (lisp-code-for-loading-from-mssql source target
|
||||
:gucs gucs
|
||||
:casts casts
|
||||
:mssql-options options))))
|
||||
:start-logger start-logger)))
|
||||
|
||||
@ -381,23 +381,38 @@
|
||||
(:import-from #:pgloader.sqlite #:*sqlite-default-cast-rules*)
|
||||
(:export #:parse-commands
|
||||
#:run-commands
|
||||
#:with-database-uri))
|
||||
|
||||
;; tools to enable complete CLI parsing in main.lisp
|
||||
#:process-relative-pathnames
|
||||
#:parse-source-string
|
||||
#:parse-source-string-for-type
|
||||
#:parse-target-string
|
||||
#:parse-cli-options
|
||||
#:parse-cli-gucs
|
||||
#:parse-cli-type
|
||||
#:parse-cli-encoding
|
||||
#:parse-cli-fields
|
||||
#:parse-cli-casts
|
||||
|
||||
;; functions to generate lisp code from parameters
|
||||
#:lisp-code-for-loading-from-mysql
|
||||
#:lisp-code-for-loading-from-csv
|
||||
#:lisp-code-for-loading-from-fixed
|
||||
#:lisp-code-for-loading-from-dbf
|
||||
#:lisp-code-for-loading-from-ixf
|
||||
#:lisp-code-for-loading-from-sqlite
|
||||
#:lisp-code-for-loading-from-mssql))
|
||||
|
||||
|
||||
;;
|
||||
;; Main package
|
||||
;;
|
||||
(defpackage #:pgloader
|
||||
(:use #:cl #:pgloader.params #:pgloader.utils)
|
||||
(:use #:cl #:pgloader.params #:pgloader.utils #:pgloader.parser)
|
||||
(:import-from #:pgloader.pgsql
|
||||
#:copy-from-file
|
||||
#:list-databases
|
||||
#:list-tables)
|
||||
(:import-from #:pgloader.parser
|
||||
#:run-commands
|
||||
#:parse-commands
|
||||
#:with-database-uri)
|
||||
(:import-from #:pgloader.sources
|
||||
#:connection-error)
|
||||
(:export #:*version-string*
|
||||
|
||||
@ -9,6 +9,7 @@
|
||||
#:*csv-path-root*
|
||||
#:*root-dir*
|
||||
#:*log-filename*
|
||||
#:*summary-pathname*
|
||||
#:*client-min-messages*
|
||||
#:*log-min-messages*
|
||||
#:*report-stream*
|
||||
@ -91,6 +92,8 @@
|
||||
:type "log")
|
||||
"Main pgloader log file")
|
||||
|
||||
(defparameter *summary-pathname* nil "Pathname where to output the summary.")
|
||||
|
||||
(defparameter *client-min-messages* :notice)
|
||||
(defparameter *log-min-messages* :info)
|
||||
|
||||
|
||||
@ -405,12 +405,14 @@
|
||||
(destructuring-bind (source encoding fields target columns clauses) command
|
||||
`(,source ,encoding ,fields ,target ,columns ,@clauses))))
|
||||
|
||||
(defrule load-csv-file load-csv-file-command
|
||||
(:lambda (command)
|
||||
(bind (((source encoding fields pg-db-uri columns
|
||||
&key ((:csv-options options)) gucs before after) command)
|
||||
((&key dbname table-name &allow-other-keys) pg-db-uri))
|
||||
`(lambda ()
|
||||
(defun lisp-code-for-loading-from-csv (source fields pg-db-uri
|
||||
&key
|
||||
(encoding :utf-8)
|
||||
columns
|
||||
gucs before after
|
||||
((:csv-options options)))
|
||||
(bind (((&key dbname table-name &allow-other-keys) pg-db-uri))
|
||||
`(lambda ()
|
||||
(let* ((state-before ,(when before `(pgloader.utils:make-pgstate)))
|
||||
(summary (null *state*))
|
||||
(*state* (or *state* (pgloader.utils:make-pgstate)))
|
||||
@ -440,4 +442,16 @@
|
||||
(when summary
|
||||
(report-full-summary "Total import time" *state*
|
||||
:before state-before
|
||||
:finally state-after))))))))
|
||||
:finally state-after)))))))
|
||||
|
||||
(defrule load-csv-file load-csv-file-command
|
||||
(:lambda (command)
|
||||
(bind (((source encoding fields pg-db-uri columns
|
||||
&key ((:csv-options options)) gucs before after) command))
|
||||
(lisp-code-for-loading-from-csv source fields pg-db-uri
|
||||
:encoding encoding
|
||||
:columns columns
|
||||
:gucs gucs
|
||||
:before before
|
||||
:after after
|
||||
:csv-options options))))
|
||||
|
||||
@ -55,51 +55,61 @@
|
||||
(destructuring-bind (source target clauses) command
|
||||
`(,source ,target ,@clauses))))
|
||||
|
||||
(defun lisp-code-for-loading-from-dbf (source pg-db-uri
|
||||
&key
|
||||
gucs before after
|
||||
((:dbf-options options)))
|
||||
(bind (((&key dbname table-name &allow-other-keys) pg-db-uri))
|
||||
`(lambda ()
|
||||
(let* ((state-before (pgloader.utils:make-pgstate))
|
||||
(summary (null *state*))
|
||||
(*state* (or *state* (pgloader.utils:make-pgstate)))
|
||||
(state-after ,(when after `(pgloader.utils:make-pgstate)))
|
||||
,@(pgsql-connection-bindings pg-db-uri gucs)
|
||||
,@(batch-control-bindings options)
|
||||
,@(identifier-case-binding options)
|
||||
(source
|
||||
,(bind (((kind url) source))
|
||||
(ecase kind
|
||||
(:http `(with-stats-collection
|
||||
("download" :state state-before)
|
||||
(pgloader.archive:http-fetch-file ,url)))
|
||||
(:filename url))))
|
||||
(source
|
||||
(if (string= "zip" (pathname-type source))
|
||||
(progn
|
||||
(with-stats-collection ("extract" :state state-before)
|
||||
(let ((d (pgloader.archive:expand-archive source)))
|
||||
(make-pathname :defaults d
|
||||
:name (pathname-name source)
|
||||
:type "dbf"))))
|
||||
source))
|
||||
(source
|
||||
(make-instance 'pgloader.db3:copy-db3
|
||||
:target-db ,dbname
|
||||
:source source
|
||||
:target ,table-name)))
|
||||
|
||||
,(sql-code-block dbname 'state-before before "before load")
|
||||
|
||||
(pgloader.sources:copy-from source
|
||||
:state-before state-before
|
||||
,@(remove-batch-control-option options))
|
||||
|
||||
,(sql-code-block dbname 'state-after after "after load")
|
||||
|
||||
;; reporting
|
||||
(when summary
|
||||
(report-full-summary "Total import time" *state*
|
||||
:before state-before
|
||||
:finally state-after))))))
|
||||
|
||||
(defrule load-dbf-file load-dbf-command
|
||||
(:lambda (command)
|
||||
(bind (((source pg-db-uri
|
||||
&key ((:dbf-options options)) gucs before after) command)
|
||||
((&key dbname table-name &allow-other-keys) pg-db-uri))
|
||||
`(lambda ()
|
||||
(let* ((state-before (pgloader.utils:make-pgstate))
|
||||
(summary (null *state*))
|
||||
(*state* (or *state* (pgloader.utils:make-pgstate)))
|
||||
(state-after ,(when after `(pgloader.utils:make-pgstate)))
|
||||
,@(pgsql-connection-bindings pg-db-uri gucs)
|
||||
,@(batch-control-bindings options)
|
||||
,@(identifier-case-binding options)
|
||||
(source
|
||||
,(bind (((kind url) source))
|
||||
(ecase kind
|
||||
(:http `(with-stats-collection
|
||||
("download" :state state-before)
|
||||
(pgloader.archive:http-fetch-file ,url)))
|
||||
(:filename url))))
|
||||
(source
|
||||
(if (string= "zip" (pathname-type source))
|
||||
(progn
|
||||
(with-stats-collection ("extract" :state state-before)
|
||||
(let ((d (pgloader.archive:expand-archive source)))
|
||||
(make-pathname :defaults d
|
||||
:name (pathname-name source)
|
||||
:type "dbf"))))
|
||||
source))
|
||||
(source
|
||||
(make-instance 'pgloader.db3:copy-db3
|
||||
:target-db ,dbname
|
||||
:source source
|
||||
:target ,table-name)))
|
||||
|
||||
,(sql-code-block dbname 'state-before before "before load")
|
||||
|
||||
(pgloader.sources:copy-from source
|
||||
:state-before state-before
|
||||
,@(remove-batch-control-option options))
|
||||
|
||||
,(sql-code-block dbname 'state-after after "after load")
|
||||
|
||||
;; reporting
|
||||
(when summary
|
||||
(report-full-summary "Total import time" *state*
|
||||
:before state-before
|
||||
:finally state-after)))))))
|
||||
&key ((:dbf-options options)) gucs before after) command))
|
||||
(lisp-code-for-loading-from-dbf source pg-db-uri
|
||||
:gucs gucs
|
||||
:before before
|
||||
:after after
|
||||
:dbf-options options))))
|
||||
|
||||
@ -106,38 +106,52 @@
|
||||
(destructuring-bind (source encoding fields target columns clauses) command
|
||||
`(,source ,encoding ,fields ,target ,columns ,@clauses))))
|
||||
|
||||
(defun lisp-code-for-loading-from-fixed (source fields pg-db-uri
|
||||
&key
|
||||
(encoding :utf-8)
|
||||
columns
|
||||
gucs before after
|
||||
((:fixed-options options)))
|
||||
(bind (((&key dbname table-name &allow-other-keys) pg-db-uri))
|
||||
`(lambda ()
|
||||
(let* ((state-before ,(when before `(pgloader.utils:make-pgstate)))
|
||||
(summary (null *state*))
|
||||
(*state* (or *state* (pgloader.utils:make-pgstate)))
|
||||
(state-after ,(when after `(pgloader.utils:make-pgstate)))
|
||||
,@(pgsql-connection-bindings pg-db-uri gucs)
|
||||
,@(batch-control-bindings options))
|
||||
|
||||
(progn
|
||||
,(sql-code-block dbname 'state-before before "before load")
|
||||
|
||||
(let ((truncate ,(getf options :truncate))
|
||||
(source
|
||||
(make-instance 'pgloader.fixed:copy-fixed
|
||||
:target-db ,dbname
|
||||
:source ',source
|
||||
:target ,table-name
|
||||
:encoding ,encoding
|
||||
:fields ',fields
|
||||
:columns ',columns
|
||||
:skip-lines ,(or (getf options :skip-line) 0))))
|
||||
(pgloader.sources:copy-from source :truncate truncate))
|
||||
|
||||
,(sql-code-block dbname 'state-after after "after load")
|
||||
|
||||
;; reporting
|
||||
(when summary
|
||||
(report-full-summary "Total import time" *state*
|
||||
:before state-before
|
||||
:finally state-after)))))))
|
||||
|
||||
(defrule load-fixed-cols-file load-fixed-cols-file-command
|
||||
(:lambda (command)
|
||||
(bind (((source encoding fields pg-db-uri columns
|
||||
&key ((:fixed-options options)) gucs before after) command)
|
||||
((&key dbname table-name &allow-other-keys) pg-db-uri))
|
||||
`(lambda ()
|
||||
(let* ((state-before ,(when before `(pgloader.utils:make-pgstate)))
|
||||
(summary (null *state*))
|
||||
(*state* (or *state* (pgloader.utils:make-pgstate)))
|
||||
(state-after ,(when after `(pgloader.utils:make-pgstate)))
|
||||
,@(pgsql-connection-bindings pg-db-uri gucs)
|
||||
,@(batch-control-bindings options))
|
||||
|
||||
(progn
|
||||
,(sql-code-block dbname 'state-before before "before load")
|
||||
|
||||
(let ((truncate ,(getf options :truncate))
|
||||
(source
|
||||
(make-instance 'pgloader.fixed:copy-fixed
|
||||
:target-db ,dbname
|
||||
:source ',source
|
||||
:target ,table-name
|
||||
:encoding ,encoding
|
||||
:fields ',fields
|
||||
:columns ',columns
|
||||
:skip-lines ,(or (getf options :skip-line) 0))))
|
||||
(pgloader.sources:copy-from source :truncate truncate))
|
||||
|
||||
,(sql-code-block dbname 'state-after after "after load")
|
||||
|
||||
;; reporting
|
||||
(when summary
|
||||
(report-full-summary "Total import time" *state*
|
||||
:before state-before
|
||||
:finally state-after))))))))
|
||||
&key ((:fixed-options options)) gucs before after) command))
|
||||
(lisp-code-for-loading-from-fixed source fields pg-db-uri
|
||||
:encoding encoding
|
||||
:columns columns
|
||||
:gucs gucs
|
||||
:before before
|
||||
:after after
|
||||
:fixed-options options))))
|
||||
|
||||
@ -31,50 +31,60 @@
|
||||
(destructuring-bind (source target clauses) command
|
||||
`(,source ,target ,@clauses))))
|
||||
|
||||
(defun lisp-code-for-loading-from-ixf (source pg-db-uri
|
||||
&key
|
||||
gucs before after
|
||||
((:ixf-options options)))
|
||||
(bind (((&key dbname table-name &allow-other-keys) pg-db-uri))
|
||||
`(lambda ()
|
||||
(let* ((state-before (pgloader.utils:make-pgstate))
|
||||
(summary (null *state*))
|
||||
(*state* (or *state* (pgloader.utils:make-pgstate)))
|
||||
(state-after ,(when after `(pgloader.utils:make-pgstate)))
|
||||
,@(pgsql-connection-bindings pg-db-uri gucs)
|
||||
,@(batch-control-bindings options)
|
||||
,@(identifier-case-binding options)
|
||||
(source
|
||||
,(bind (((kind url) source))
|
||||
(ecase kind
|
||||
(:http `(with-stats-collection
|
||||
("download" :state state-before)
|
||||
(pgloader.archive:http-fetch-file ,url)))
|
||||
(:filename url))))
|
||||
(source
|
||||
(if (string= "zip" (pathname-type source))
|
||||
(progn
|
||||
(with-stats-collection ("extract" :state state-before)
|
||||
(let ((d (pgloader.archive:expand-archive source)))
|
||||
(make-pathname :defaults d
|
||||
:name (pathname-name source)
|
||||
:type "ixf"))))
|
||||
source))
|
||||
(source
|
||||
(make-instance 'pgloader.ixf:copy-ixf
|
||||
:target-db ,dbname
|
||||
:source source
|
||||
:target ,table-name)))
|
||||
|
||||
,(sql-code-block dbname 'state-before before "before load")
|
||||
|
||||
(pgloader.sources:copy-from source
|
||||
:state-before state-before
|
||||
,@(remove-batch-control-option options))
|
||||
|
||||
,(sql-code-block dbname 'state-after after "after load")
|
||||
|
||||
(when summary
|
||||
(report-full-summary "Total import time" *state*
|
||||
:before state-before
|
||||
:finally state-after))))))
|
||||
|
||||
(defrule load-ixf-file load-ixf-command
|
||||
(:lambda (command)
|
||||
(bind (((source pg-db-uri
|
||||
&key ((:ixf-options options)) gucs before after) command)
|
||||
((&key dbname table-name &allow-other-keys) pg-db-uri))
|
||||
`(lambda ()
|
||||
(let* ((state-before (pgloader.utils:make-pgstate))
|
||||
(summary (null *state*))
|
||||
(*state* (or *state* (pgloader.utils:make-pgstate)))
|
||||
(state-after ,(when after `(pgloader.utils:make-pgstate)))
|
||||
,@(pgsql-connection-bindings pg-db-uri gucs)
|
||||
,@(batch-control-bindings options)
|
||||
,@(identifier-case-binding options)
|
||||
(source
|
||||
,(bind (((kind url) source))
|
||||
(ecase kind
|
||||
(:http `(with-stats-collection
|
||||
("download" :state state-before)
|
||||
(pgloader.archive:http-fetch-file ,url)))
|
||||
(:filename url))))
|
||||
(source
|
||||
(if (string= "zip" (pathname-type source))
|
||||
(progn
|
||||
(with-stats-collection ("extract" :state state-before)
|
||||
(let ((d (pgloader.archive:expand-archive source)))
|
||||
(make-pathname :defaults d
|
||||
:name (pathname-name source)
|
||||
:type "ixf"))))
|
||||
source))
|
||||
(source
|
||||
(make-instance 'pgloader.ixf:copy-ixf
|
||||
:target-db ,dbname
|
||||
:source source
|
||||
:target ,table-name)))
|
||||
|
||||
,(sql-code-block dbname 'state-before before "before load")
|
||||
|
||||
(pgloader.sources:copy-from source
|
||||
:state-before state-before
|
||||
,@(remove-batch-control-option options))
|
||||
|
||||
,(sql-code-block dbname 'state-after after "after load")
|
||||
|
||||
(when summary
|
||||
(report-full-summary "Total import time" *state*
|
||||
:before state-before
|
||||
:finally state-after)))))))
|
||||
&key ((:ixf-options options)) gucs before after) command))
|
||||
(lisp-code-for-loading-from-ixf source pg-db-uri
|
||||
:gucs gucs
|
||||
:before before
|
||||
:after after
|
||||
:ixf-options options))))
|
||||
|
||||
@ -124,25 +124,24 @@
|
||||
|
||||
|
||||
;;; LOAD DATABASE FROM mssql://
|
||||
(defrule load-mssql-database load-mssql-command
|
||||
(:lambda (source)
|
||||
(bind (((ms-db-uri pg-db-uri
|
||||
&key
|
||||
gucs casts before after including excluding
|
||||
((:mysql-options options))) source)
|
||||
(defun lisp-code-for-loading-from-mssql (source pg-db-uri
|
||||
&key
|
||||
gucs casts before after
|
||||
((:mssql-options options))
|
||||
(including)
|
||||
(excluding))
|
||||
(bind (((&key ((:dbname msdb)) table-name
|
||||
&allow-other-keys) source)
|
||||
|
||||
((&key ((:dbname msdb)) table-name
|
||||
&allow-other-keys) ms-db-uri)
|
||||
|
||||
((&key ((:dbname pgdb)) &allow-other-keys) pg-db-uri))
|
||||
`(lambda ()
|
||||
((&key ((:dbname pgdb)) &allow-other-keys) pg-db-uri))
|
||||
`(lambda ()
|
||||
(let* ((state-before (pgloader.utils:make-pgstate))
|
||||
(*state* (or *state* (pgloader.utils:make-pgstate)))
|
||||
(state-idx (pgloader.utils:make-pgstate))
|
||||
(state-after (pgloader.utils:make-pgstate))
|
||||
(*default-cast-rules* ',*mssql-default-cast-rules*)
|
||||
(*cast-rules* ',casts)
|
||||
,@(mssql-connection-bindings ms-db-uri)
|
||||
,@(mssql-connection-bindings source)
|
||||
,@(pgsql-connection-bindings pg-db-uri gucs)
|
||||
,@(batch-control-bindings options)
|
||||
,@(identifier-case-binding options)
|
||||
@ -168,5 +167,21 @@
|
||||
(report-full-summary "Total import time" *state*
|
||||
:before state-before
|
||||
:finally state-after
|
||||
:parallel state-idx))))))
|
||||
:parallel state-idx)))))
|
||||
|
||||
(defrule load-mssql-database load-mssql-command
|
||||
(:lambda (source)
|
||||
(bind (((ms-db-uri pg-db-uri
|
||||
&key
|
||||
gucs casts before after including excluding
|
||||
((:mysql-options options)))
|
||||
source))
|
||||
(lisp-code-for-loading-from-mssql ms-db-uri pg-db-uri
|
||||
:gucs gucs
|
||||
:casts casts
|
||||
:before before
|
||||
:after after
|
||||
:mssql-options options
|
||||
:including including
|
||||
:excluding excluding))))
|
||||
|
||||
|
||||
@ -159,54 +159,70 @@
|
||||
|
||||
|
||||
;;; LOAD DATABASE FROM mysql://
|
||||
(defun lisp-code-for-loading-from-mysql (my-db-uri pg-db-uri
|
||||
&key
|
||||
gucs casts views before after
|
||||
((:mysql-options options))
|
||||
((:including incl))
|
||||
((:excluding excl))
|
||||
((:decoding decoding-as)))
|
||||
(bind (((&key ((:dbname mydb)) table-name
|
||||
&allow-other-keys) my-db-uri)
|
||||
|
||||
((&key ((:dbname pgdb)) &allow-other-keys) pg-db-uri))
|
||||
`(lambda ()
|
||||
(let* ((state-before (pgloader.utils:make-pgstate))
|
||||
(*state* (or *state* (pgloader.utils:make-pgstate)))
|
||||
(state-idx (pgloader.utils:make-pgstate))
|
||||
(state-after (pgloader.utils:make-pgstate))
|
||||
(*default-cast-rules* ',*mysql-default-cast-rules*)
|
||||
(*cast-rules* ',casts)
|
||||
,@(mysql-connection-bindings my-db-uri)
|
||||
,@(pgsql-connection-bindings pg-db-uri gucs)
|
||||
,@(batch-control-bindings options)
|
||||
,@(identifier-case-binding options)
|
||||
(source
|
||||
(make-instance 'pgloader.mysql::copy-mysql
|
||||
:target-db ,pgdb
|
||||
:source-db ,mydb)))
|
||||
|
||||
,(sql-code-block pgdb 'state-before before "before load")
|
||||
|
||||
(pgloader.mysql:copy-database source
|
||||
,@(when table-name
|
||||
`(:only-tables ',(list table-name)))
|
||||
:including ',incl
|
||||
:excluding ',excl
|
||||
:decoding-as ',decoding-as
|
||||
:materialize-views ',views
|
||||
:state-before state-before
|
||||
:state-after state-after
|
||||
:state-indexes state-idx
|
||||
,@(remove-batch-control-option options))
|
||||
|
||||
,(sql-code-block pgdb 'state-after after "after load")
|
||||
|
||||
(report-full-summary "Total import time" *state*
|
||||
:before state-before
|
||||
:finally state-after
|
||||
:parallel state-idx)))))
|
||||
|
||||
(defrule load-mysql-database load-mysql-command
|
||||
(:lambda (source)
|
||||
(bind (((my-db-uri pg-db-uri
|
||||
&key
|
||||
gucs casts views before after
|
||||
((:mysql-options options))
|
||||
((:including incl))
|
||||
((:excluding excl))
|
||||
((:decoding decoding-as))) source)
|
||||
|
||||
((&key ((:dbname mydb)) table-name
|
||||
&allow-other-keys) my-db-uri)
|
||||
|
||||
((&key ((:dbname pgdb)) &allow-other-keys) pg-db-uri))
|
||||
`(lambda ()
|
||||
(let* ((state-before (pgloader.utils:make-pgstate))
|
||||
(*state* (or *state* (pgloader.utils:make-pgstate)))
|
||||
(state-idx (pgloader.utils:make-pgstate))
|
||||
(state-after (pgloader.utils:make-pgstate))
|
||||
(*default-cast-rules* ',*mysql-default-cast-rules*)
|
||||
(*cast-rules* ',casts)
|
||||
,@(mysql-connection-bindings my-db-uri)
|
||||
,@(pgsql-connection-bindings pg-db-uri gucs)
|
||||
,@(batch-control-bindings options)
|
||||
,@(identifier-case-binding options)
|
||||
(source
|
||||
(make-instance 'pgloader.mysql::copy-mysql
|
||||
:target-db ,pgdb
|
||||
:source-db ,mydb)))
|
||||
|
||||
,(sql-code-block pgdb 'state-before before "before load")
|
||||
|
||||
(pgloader.mysql:copy-database source
|
||||
,@(when table-name
|
||||
`(:only-tables ',(list table-name)))
|
||||
:including ',incl
|
||||
:excluding ',excl
|
||||
:decoding-as ',decoding-as
|
||||
:materialize-views ',views
|
||||
:state-before state-before
|
||||
:state-after state-after
|
||||
:state-indexes state-idx
|
||||
,@(remove-batch-control-option options))
|
||||
|
||||
,(sql-code-block pgdb 'state-after after "after load")
|
||||
|
||||
(report-full-summary "Total import time" *state*
|
||||
:before state-before
|
||||
:finally state-after
|
||||
:parallel state-idx))))))
|
||||
(destructuring-bind (my-db-uri
|
||||
pg-db-uri
|
||||
&key
|
||||
gucs casts views before after
|
||||
mysql-options including excluding decoding)
|
||||
source
|
||||
(lisp-code-for-loading-from-mysql my-db-uri pg-db-uri
|
||||
:gucs gucs
|
||||
:casts casts
|
||||
:views views
|
||||
:before before
|
||||
:after after
|
||||
:mysql-options mysql-options
|
||||
:including including
|
||||
:excluding excluding
|
||||
:decoding decoding))))
|
||||
|
||||
|
||||
@ -104,7 +104,7 @@
|
||||
(defun run-commands (source
|
||||
&key
|
||||
(start-logger t)
|
||||
((:summary summary-pathname))
|
||||
((:summary *summary-pathname*) *summary-pathname*)
|
||||
((:log-filename *log-filename*) *log-filename*)
|
||||
((:log-min-messages *log-min-messages*) *log-min-messages*)
|
||||
((:client-min-messages *client-min-messages*) *client-min-messages*))
|
||||
@ -128,8 +128,8 @@
|
||||
(parse-commands source)))))))
|
||||
|
||||
;; maybe duplicate the summary to a file
|
||||
(let* ((summary-stream (when summary-pathname
|
||||
(open summary-pathname
|
||||
(let* ((summary-stream (when *summary-pathname*
|
||||
(open *summary-pathname*
|
||||
:direction :output
|
||||
:if-exists :rename
|
||||
:if-does-not-exist :create)))
|
||||
@ -140,3 +140,146 @@
|
||||
|
||||
;; cleanup
|
||||
(when summary-stream (close summary-stream)))))))
|
||||
|
||||
|
||||
;;;
|
||||
;;; Parse an URI without knowing before hand what kind of uri it is.
|
||||
;;;
|
||||
(defvar *db-uri-prefix-list* '((:postgresql . ("pgsql://"
|
||||
"postgres://"
|
||||
"postgresql://"))
|
||||
(:sqlite . ("sqlite://"))
|
||||
(:mysql . ("mysql://"))
|
||||
(:mssql . ("mssql://"))))
|
||||
|
||||
(defun parse-db-uri-prefix (source-string)
|
||||
"See if SOURCE-STRING starts with a known dburi"
|
||||
(loop :for (type . prefixes) :in *db-uri-prefix-list*
|
||||
:for prefix := (loop :for prefix :in prefixes
|
||||
:when (let ((plen (length prefix)))
|
||||
(and (< plen (length source-string))
|
||||
(string-equal prefix source-string :end2 plen)))
|
||||
:return prefix)
|
||||
:when prefix :return type))
|
||||
|
||||
(defvar *data-source-filename-extensions*
|
||||
'((:csv . ("csv" "tsv" "txt" "text"))
|
||||
(:sqlite . ("sqlite" "db"))
|
||||
(:db3 . ("db3" "dbf"))
|
||||
(:ixf . ("ixf"))))
|
||||
|
||||
(defun parse-filename-for-source-type (filename)
|
||||
"Given just an existing filename, decide what data source might be found
|
||||
inside..."
|
||||
(multiple-value-bind (abs paths filename no-path-p)
|
||||
(uiop:split-unix-namestring-directory-components
|
||||
(uiop:native-namestring filename))
|
||||
(declare (ignore abs paths no-path-p))
|
||||
(let ((dotted-parts (reverse (sq:split-sequence #\. filename))))
|
||||
(destructuring-bind (extension name-or-ext &rest parts)
|
||||
dotted-parts
|
||||
(declare (ignore parts))
|
||||
(if (string-equal "tar" name-or-ext) :archive
|
||||
(loop :for (type . extensions) :in *data-source-filename-extensions*
|
||||
:when (member extension extensions :test #'string-equal)
|
||||
:return type))))))
|
||||
|
||||
(defun parse-source-string-for-type (type source-string)
|
||||
"use the parse rules as per xxx-source rules"
|
||||
(let ((source (case type
|
||||
(:csv (parse 'csv-file-source source-string))
|
||||
(:fixed (parse 'fixed-file-source source-string))
|
||||
(:db3 (parse 'filename-or-http-uri source-string))
|
||||
(:ixf (parse 'filename-or-http-uri source-string))
|
||||
(:sqlite (parse 'sqlite-uri source-string))
|
||||
(:postgresql (parse 'pgsql-uri source-string))
|
||||
(:mysql (parse 'mysql-uri source-string))
|
||||
(:mssql (parse 'mssql-uri source-string)))))
|
||||
(values type source)))
|
||||
|
||||
(defun parse-source-string (source-string)
|
||||
"Guess type from SOURCE-STRING then parse it accordingly."
|
||||
(cond ((probe-file (uiop:parse-unix-namestring source-string))
|
||||
(let ((type (parse-filename-for-source-type source-string)))
|
||||
(parse-source-string-for-type type source-string)))
|
||||
|
||||
((and source-string
|
||||
(< (length "http://") (length source-string))
|
||||
(string-equal "http://" source-string :end2 (length "http://")))
|
||||
(let ((type (parse-filename-for-source-type
|
||||
(puri:uri-path (puri:parse-uri source-string)))))
|
||||
(case type
|
||||
(:csv (log-message :fatal "No HTTP support for CSV files yet."))
|
||||
(:fixed (log-message :fatal "No HTTP support for FIXED files yet."))
|
||||
(:sqlite (parse-source-string-for-type :sqlite source-string))
|
||||
(:db3 (parse-source-string-for-type :db3 source-string))
|
||||
(:ixf (parse-source-string-for-type :ixf source-string)))))
|
||||
|
||||
((and source-string (parse-db-uri-prefix source-string))
|
||||
(let* ((type (parse-db-uri-prefix source-string)))
|
||||
(multiple-value-bind (type conn)
|
||||
(parse-source-string-for-type type source-string)
|
||||
(if (eq type (getf conn :type))
|
||||
(values type conn)
|
||||
(log-message :fatal "Parsed a ~s connection string for type ~s")))))
|
||||
|
||||
(t nil)))
|
||||
|
||||
(defun parse-target-string (target-string)
|
||||
(parse 'pgsql-uri target-string))
|
||||
|
||||
|
||||
;;;
|
||||
;;; Command line accumulative options parser
|
||||
;;;
|
||||
(defun parse-cli-gucs (gucs)
|
||||
"Parse PostgreSQL GUCs as per the SET clause when we get them from the CLI."
|
||||
(loop :for guc :in gucs
|
||||
:collect (parse 'generic-option guc)))
|
||||
|
||||
(defrule cli-type (or "csv"
|
||||
"fixed"
|
||||
"db3"
|
||||
"ixf"
|
||||
"sqlite"
|
||||
"mysql"
|
||||
"mssql")
|
||||
(:text t))
|
||||
|
||||
(defun parse-cli-type (type)
|
||||
"Parse the --type option"
|
||||
(when type
|
||||
(intern (string-upcase (parse 'cli-type type)) (find-package "KEYWORD"))))
|
||||
|
||||
(defun parse-cli-encoding (encoding)
|
||||
"Parse the --encoding option"
|
||||
(if encoding
|
||||
(make-external-format (find-encoding-by-name-or-alias encoding))
|
||||
:utf-8))
|
||||
|
||||
(defun parse-cli-fields (type fields)
|
||||
"Parse the --fields option."
|
||||
(loop :for field :in fields
|
||||
:collect (parse (case type
|
||||
(:csv 'csv-source-field)
|
||||
(:fixed 'fixed-source-field))
|
||||
field)))
|
||||
|
||||
(defun parse-cli-options (type options)
|
||||
"Parse options as per the WITH clause when we get them from the CLI."
|
||||
(alexandria:alist-plist
|
||||
(loop :for option :in options
|
||||
:collect (parse (case type
|
||||
(:csv 'csv-option)
|
||||
(:fixed 'fixed-option)
|
||||
(:db3 'dbf-option)
|
||||
(:ixf 'ixf-option)
|
||||
(:sqlite 'sqlite-option)
|
||||
(:mysql 'mysql-option)
|
||||
(:mssql 'mysql-option))
|
||||
option))))
|
||||
|
||||
(defun parse-cli-casts (casts)
|
||||
"Parse additional CAST rules when we get them from the CLI."
|
||||
(loop :for cast :in casts
|
||||
:collect (parse 'cast-rule cast)))
|
||||
|
||||
@ -73,8 +73,8 @@ load database
|
||||
(defrule load-sqlite-optional-clauses (* (or sqlite-options
|
||||
gucs
|
||||
casts
|
||||
including
|
||||
excluding))
|
||||
including-matching
|
||||
excluding-matching))
|
||||
(:lambda (clauses-list)
|
||||
(alexandria:alist-plist clauses-list)))
|
||||
|
||||
@ -84,50 +84,60 @@ load database
|
||||
(destructuring-bind (source target clauses) command
|
||||
`(,source ,target ,@clauses))))
|
||||
|
||||
(defun lisp-code-for-loading-from-sqlite (source pg-db-uri
|
||||
&key
|
||||
gucs casts
|
||||
((:sqlite-options options))
|
||||
((:including incl))
|
||||
((:excluding excl)))
|
||||
(bind (((&key dbname table-name &allow-other-keys) pg-db-uri))
|
||||
`(lambda ()
|
||||
(let* ((state-before (pgloader.utils:make-pgstate))
|
||||
(*state* (pgloader.utils:make-pgstate))
|
||||
(*default-cast-rules* ',*sqlite-default-cast-rules*)
|
||||
(*cast-rules* ',casts)
|
||||
,@(pgsql-connection-bindings pg-db-uri gucs)
|
||||
,@(batch-control-bindings options)
|
||||
(db
|
||||
,(bind (((kind url) source))
|
||||
(ecase kind
|
||||
(:http `(with-stats-collection
|
||||
("download" :state state-before)
|
||||
(pgloader.archive:http-fetch-file ,url)))
|
||||
(:sqlite url)
|
||||
(:filename url))))
|
||||
(db
|
||||
(if (string= "zip" (pathname-type db))
|
||||
(progn
|
||||
(with-stats-collection ("extract" :state state-before)
|
||||
(let ((d (pgloader.archive:expand-archive db)))
|
||||
(make-pathname :defaults d
|
||||
:name (pathname-name db)
|
||||
:type "db"))))
|
||||
db))
|
||||
(source
|
||||
(make-instance 'pgloader.sqlite::copy-sqlite
|
||||
:target-db ,dbname
|
||||
:source-db db)))
|
||||
(pgloader.sqlite:copy-database
|
||||
source
|
||||
:state-before state-before
|
||||
,@(when table-name
|
||||
`(:only-tables ',(list table-name)))
|
||||
:including ',incl
|
||||
:excluding ',excl
|
||||
,@(remove-batch-control-option options))))))
|
||||
|
||||
(defrule load-sqlite-database load-sqlite-command
|
||||
(:lambda (source)
|
||||
(bind (((sqlite-uri pg-db-uri
|
||||
&key
|
||||
gucs
|
||||
casts
|
||||
((:sqlite-options options))
|
||||
((:including incl))
|
||||
((:excluding excl))) source)
|
||||
((&key dbname table-name &allow-other-keys) pg-db-uri))
|
||||
`(lambda ()
|
||||
(let* ((state-before (pgloader.utils:make-pgstate))
|
||||
(*state* (pgloader.utils:make-pgstate))
|
||||
(*default-cast-rules* ',*sqlite-default-cast-rules*)
|
||||
(*cast-rules* ',casts)
|
||||
,@(pgsql-connection-bindings pg-db-uri gucs)
|
||||
,@(batch-control-bindings options)
|
||||
(db
|
||||
,(bind (((kind url) sqlite-uri))
|
||||
(ecase kind
|
||||
(:http `(with-stats-collection
|
||||
("download" :state state-before)
|
||||
(pgloader.archive:http-fetch-file ,url)))
|
||||
(:sqlite url)
|
||||
(:filename url))))
|
||||
(db
|
||||
(if (string= "zip" (pathname-type db))
|
||||
(progn
|
||||
(with-stats-collection ("extract" :state state-before)
|
||||
(let ((d (pgloader.archive:expand-archive db)))
|
||||
(make-pathname :defaults d
|
||||
:name (pathname-name db)
|
||||
:type "db"))))
|
||||
db))
|
||||
(source
|
||||
(make-instance 'pgloader.sqlite::copy-sqlite
|
||||
:target-db ,dbname
|
||||
:source-db db)))
|
||||
(pgloader.sqlite:copy-database
|
||||
source
|
||||
:state-before state-before
|
||||
,@(when table-name
|
||||
`(:only-tables ',(list table-name)))
|
||||
:including ',incl
|
||||
:excluding ',excl
|
||||
,@(remove-batch-control-option options)))))))
|
||||
(destructuring-bind (sqlite-uri
|
||||
pg-db-uri
|
||||
&key gucs casts sqlite-options including excluding)
|
||||
source
|
||||
(lisp-code-for-loading-from-sqlite sqlite-uri pg-db-uri
|
||||
:gucs gucs
|
||||
:casts casts
|
||||
:sqlite-options sqlite-options
|
||||
:including including
|
||||
:excluding excluding))))
|
||||
|
||||
|
||||
@ -42,7 +42,7 @@
|
||||
:stream *standard-output*)
|
||||
*log-messengers*)
|
||||
|
||||
(cl-log:log-message :log "Starting pgloader, log system is ready."))
|
||||
(cl-log:log-message :notice "Starting pgloader, log system is ready."))
|
||||
|
||||
(defun stop-logger ()
|
||||
"Stop the pgloader manager and messengers."
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user