It's not possible to use a comma separator when using more than one
source field option at the same time, and for better readability the
options are to be found enclosed in squared brackets.
Also, it's now possible to spell out "from" and "for" keywords on the
source definitions, making it easier to read and maintain the load file,
as in this full example:
(
a from 0 for 10,
b from 10 for 8,
c from 18 for 8,
d from 26 for 17 [null if blanks, trim right whitespace]
)
As seen in #116, it might be better for the users to be able to ask for
field trimming right in the source definition, like we do for processing
nulls.
The default values for MySQL WITH clause options wasn't really tested
and broke on simple cases, the new set of defaults is known to work in
many cases (most?).
Other combinations of options will need some review work, and we might
need to consider preventing some of them, that's for another patch tho.
To avoid wasting everybody's time when trying to debug --load
command.load, rename the option to be more explicit about what it does.
Also implement some basic guards in the form of testing that the
filename extension is part of a very short whitelist: .lisp, .cl, .lsp
and .asd.
The format string were wrong in that ~@[...~] will not consume the argument
where we need to consume it, so use the more verbose form ~:[~*~;...~]
wherein the ~* rule forces the consuming of the unused argument.
Also, prefer using != and NOT REGEXP operators to the "not ( or )" form,
which was wrong too because spelled "not ( and )".
When default values are used in SQLite they are of course using their
SQLite representation, which might not be compatible with the PostgreSQL
target data type we're casting to. Make it so that the default values
are transformed too, as we already do in the MySQL case.
See #100.
It might be important to be able to use the exact same pgloader commands
file but adapt its source and target depending on the environment where
the command is to be run (production, development, staging, etc).
Introduce the new sub-clause GETENV 'variable-name' to that effect.
The regression test facility that we have now isn't nearly sophisticated
enough to support this, so the feature isn't yet covered.
When using both field level specifications and transformation functions,
the transformation needs to happen on the result of the field processing
rather than on the raw field.
Fixes#92.
That patch is not a principaled approach at fixing the problem but
should allow for not messing up with fully qualified table names.
A proper way to do it would be to have a pgsql object name structure
composed of the catalog, the schema and the name as separate entries,
with assorted API to print that object properly. That's for another day
though.
Several transformation functions where not handling nil value
specifically, preventing the compiler to do its best and producing
WARNING at pgloader run time.
The previous patch didn't take into account the need to retain the case
of the PostgreSQL column names when using double-quotes in the load
command, which is now properly forwarded down in the COPY command.
With this patch it's now possible to "quote" column names and to use
_ (underscore) as the leading character in a field or column name. Of
course it is still unnecessary to quote the column names, but handling
the PostgreSQL quoting rules can't be bad here.
