Dimitri Fontaine 633067a0fd Allow more parallelism in database migrations. ...

The newly added statistics are showing that read+write times are not
enough to explain how long we wait for the data copying, so it must be
the workers setup rather than the workers themselves.

From there, let lparallel work its magic in scheduling the work we do in
parallel in pgloader: rather than doing blocking receive-result calls
for each table, only receive-result at the end of the whole
copy-database processing.

On test data here on the laptop we go from 6s to 3s to migrate the
sakila database from MySQL to PostgreSQL: that's because we have lots of
very small tables, so the cost of waiting after each COPY added up quite
quickly.

In passing, stop sharing the same connection object in between parallel
workers that used to be controlled active in-sequence, see the new API
clone-connection (which takes over new-pgsql-connection).

2015-10-20 22:15:55 +02:00

..

copy-format.lisp

Left pad COPY octal chars with 0, fix #275.

2015-08-20 18:17:18 +02:00

pgsql.lisp

Allow more parallelism in database migrations.

2015-10-20 22:15:55 +02:00

queries.lisp

Allow more parallelism in database migrations.

2015-10-20 22:15:55 +02:00

schema.lisp

Allow more parallelism in database migrations.

2015-10-20 22:15:55 +02:00