mirror of
https://github.com/dimitri/pgloader.git
synced 2026-04-29 16:11:01 +02:00
f86371970f
Refactor file organisation further to allow for adding a “direct stream” option when the on-error-stop behavior has been selected. This happens currently by default for databases sources. Introduce the new WITH option “on error resume next” which forces the classic behavior of pgloader. The option “on error stop” already existed, its implementation is new. When this new behavior is activated, the data is sent to PostgreSQL directly, without intermediate batches being built. It means that the whole operation fails at the first error, and we don't have any information in memory to try replaying any COPY of the data. It's gone. This behavior should be fine for database migrations as you don't usually want to fix the data manually in intermediate files, you want to fix the problem at the source database and do the whole dance all-over again, up until your casting rules are perfect. This patch might also incurr some performance benenits in terms of both timing and memory usage, though the local testing didn't show much of anything for the moment.