mirror of
https://github.com/dimitri/pgloader.git
synced 2025-08-09 15:56:58 +02:00
Move the TODO into its own file.
This commit is contained in:
parent
c21f3f06ff
commit
24c0580e11
71
README.md
71
README.md
@ -143,74 +143,3 @@ file into a manual page or an HTML page thanks to the `pandoc` application:
|
||||
$ apt-get install pandoc
|
||||
$ pandoc pgloader.1.md -o pgloader.1
|
||||
$ pandoc pgloader.1.md -o pgloader.html
|
||||
|
||||
## TODO
|
||||
|
||||
Some notes about what I intend to be working on next.
|
||||
|
||||
### binary distribution
|
||||
|
||||
- prepare an all-included binary for several platforms
|
||||
|
||||
### docs
|
||||
|
||||
- host a proper website for the tool, with use cases and a tutorial
|
||||
|
||||
### error management
|
||||
|
||||
- error management with a local buffer (done)
|
||||
- error reporting (done)
|
||||
- add input line number to log file?
|
||||
|
||||
### data output
|
||||
|
||||
- PostgreSQL COPY Text format output for any supported input
|
||||
|
||||
#### convenience
|
||||
|
||||
- automatic creation of schema even when loading from text files
|
||||
- pre-fetch some rows to guesstimate data types?
|
||||
|
||||
#### performances
|
||||
|
||||
- some more parallelizing options
|
||||
- support for partitionning in pgloader itself
|
||||
|
||||
#### reformating
|
||||
|
||||
Data reformating is now going to have to happen in Common Lisp mostly, maybe
|
||||
offer some other languages (cl-awk etc).
|
||||
|
||||
- raw reformating, before rows are split
|
||||
|
||||
Have a try at something approaching:
|
||||
|
||||
WITH data AS (
|
||||
COPY FROM ...
|
||||
RETURNING x, y
|
||||
)
|
||||
SELECT foo(x), bar(y)
|
||||
FROM data
|
||||
WHERE ...
|
||||
|
||||
A part of that needs to happen client-side, another part server-side, and
|
||||
the grammar has to make it clear what happens where. Maybe add a WHERE
|
||||
clause to the `COPY` or `LOAD` grammar for the client.
|
||||
|
||||
#### filtering
|
||||
|
||||
Add commands to pick different target tables depending on the data found
|
||||
when reading from the source.
|
||||
|
||||
#### UI
|
||||
|
||||
- add a web controler with pretty monitoring
|
||||
- launch new jobs from the web controler
|
||||
|
||||
#### crazy ideas
|
||||
|
||||
- MySQL replication, reading from the binlog directly
|
||||
- plproxy (re-)sharding support
|
||||
- partitioning support
|
||||
- remote archiving support (with (delete returning *) insert into)
|
||||
|
||||
|
82
TODO.md
Normal file
82
TODO.md
Normal file
@ -0,0 +1,82 @@
|
||||
# TODO
|
||||
|
||||
Some notes about what I intend to be working on next. You can sponsor any
|
||||
and all of those ideas if you actually need them today, and you can also
|
||||
sponsor new ideas not on the list yet.
|
||||
|
||||
## Data Formats
|
||||
|
||||
## MySQL Support
|
||||
|
||||
- MATERIALIZE ALL VIEWS
|
||||
- Convert SQL dialect for SQL views
|
||||
- Triggers and Stored Procedures
|
||||
|
||||
## Other
|
||||
|
||||
### binary distribution
|
||||
|
||||
- prepare an all-included binary for several platforms
|
||||
|
||||
### docs
|
||||
|
||||
- host a proper website for the tool, with use cases and a tutorial
|
||||
|
||||
### error management
|
||||
|
||||
- error management with a local buffer (done)
|
||||
- error reporting (done)
|
||||
- add input line number to log file?
|
||||
|
||||
### data output
|
||||
|
||||
- PostgreSQL COPY Text format output for any supported input
|
||||
|
||||
#### convenience
|
||||
|
||||
- automatic creation of schema even when loading from text files
|
||||
- pre-fetch some rows to guesstimate data types?
|
||||
|
||||
#### performances
|
||||
|
||||
- some more parallelizing options
|
||||
- support for partitionning in pgloader itself
|
||||
|
||||
#### reformating
|
||||
|
||||
Data reformating is now going to have to happen in Common Lisp mostly, maybe
|
||||
offer some other languages (cl-awk etc).
|
||||
|
||||
- raw reformating, before rows are split
|
||||
|
||||
Have a try at something approaching:
|
||||
|
||||
WITH data AS (
|
||||
COPY FROM ...
|
||||
RETURNING x, y
|
||||
)
|
||||
SELECT foo(x), bar(y)
|
||||
FROM data
|
||||
WHERE ...
|
||||
|
||||
A part of that needs to happen client-side, another part server-side, and
|
||||
the grammar has to make it clear what happens where. Maybe add a WHERE
|
||||
clause to the `COPY` or `LOAD` grammar for the client.
|
||||
|
||||
#### filtering
|
||||
|
||||
Add commands to pick different target tables depending on the data found
|
||||
when reading from the source.
|
||||
|
||||
#### UI
|
||||
|
||||
- add a web controler with pretty monitoring
|
||||
- launch new jobs from the web controler
|
||||
|
||||
#### crazy ideas
|
||||
|
||||
- MySQL replication, reading from the binlog directly
|
||||
- plproxy (re-)sharding support
|
||||
- partitioning support
|
||||
- remote archiving support (with (delete returning *) insert into)
|
||||
|
Loading…
Reference in New Issue
Block a user