bbkr / uprooted Goto Github PK
View Code? Open in Web Editor NEWExtract subtrees of data from relational databases.
License: Artistic License 2.0
Extract subtrees of data from relational databases.
License: Artistic License 2.0
This is nice to have feature but complicates Writer
a lot - next row of data must have ability to terminate previous set of rows and leaving writing after last row must terminate everything..
It will be implemented if it gets 8 upvotes here :)
To complete proof-of-concept chain from Schema
to Writer
MySQL seems like simplest choice.
Dumping to *.sql
file or cloning to another database are probably two most common dev tasks.
Tree
can be represented as set of Paths
from root Table
to every leaf Table
(intermediate Tables are also considered leafs). This is also contract for Reader
.
It must have circuit breaker when Path
detected loop.
Decide if horse riddle is fatal and not allow Tree
to be constructed.
Or maybe it should accept such Path
but cause error/warning on Reader
level?
To complete proof-of-concept chain from Schema
to Writer
MySQL seems like simplest choice.
UpRooted::Schema::Column.is-nullable
, UpRooted::Writer.use-schema-prefix
, etc.
Less confusing, because in old layout one my expect to provide actual data like UpRooted::Writer.new( schema-prefix => 'foo' )
.
my $writer = UpRooted::Writer::MySQLFile.new( :!use-schema-name );
$writer.write( $reader, id => 1 );
This looks bad for example, named param, positional param and slurpy param in one class.
Not to mention future write
method options may have conflicts with conditions if more params are added.
Should be named params all the way:
my $writer = UpRooted::Writer::MySQLFile.new( :!use-schema-name );
$writer.write( :$reader, conditions => { 'id' => 1 } );
Instead of having separate "factory" actor:
use UpRooted::Cartographer::MySQL;
my $connection = DBIish.connect( 'mysql', host => ..., port => ..., ... );
my $schema = UpRooted::Cartographer::MySQL.new( :$connection ).schema( );
Just extend base Schema
:
use Schema::MySQL;
my $connection = DBIish.connect( 'mysql', host => ..., port => ..., ... );
my $schema = Schema::MySQL.new( :$connection );
To write data from Reader
directly into PostgreSQL database.
Many potential users who want to use UpRooted
may be discouraged by installation of Raku, studying package management and installing needed C libraries on their system.
Docker image with Raku, DBI, C libraries and UpRooted preinstalled will be a good start for those who want to simply get the job done without deep diving into Raku ecosystem.
It should accept Channel from Reader
.
On this Channel there will be Table
followed by set of rows from this Table
. When all Table
s are read Channel will be closed.
This flow will Writer
to prepare for new data set before actual data will arrive, for example open new file, prepare new printf template, prepare new SQL statement, etc.
Decide which database use. Percona is probably most popular today on server environments but MariaDB diverged more from vanilla MySQL. Or maybe use all of them?
Places to check:
Consistent naming pattern:
UpRooted::*
object instance it should go by full name.table
, not Table
(because there is no Table
namespace or object anywhere)This will prevent unwanted descending into data that should not be dumped.
To directly copy UpRooted::Reader
data to another database.
If lucky this will reuse almost whole logic from Writer::MySQLFile
except quoting.
Probably Writer::Helper::SQLQuery
could be created to be shared by both.
Which will also reuse later VALUES batching code.
This was noble idea to provide "safe" common type that UpRooted::Writer
can rely on.
However this is faulty because of following reasons:
UpRooted::Column
is very difficult.UpRooted::Reader
s in many cases.UpRooted::Writer
s may benefit from complex types. Like for example Writer::JSON
(if implemented) will handle PostgreSQL array types without any issues. It is best for Writer
to decide how to handle data from database driver.UpRooted::Writer
that can store this type he should be able to do so without looking at real column type.It should accept Tree
as contract and iterate over Path
s in their order.
Each Path
should be processed by specific Reader
implementation and produce set or rows.
Reader
should expose channel, which will receive Table
, row, row, row, Table
, row, row, row, row, row... pattern.
Channel should be closed when last Path
is processed.
Each set of new root Table
conditions should open new Channel.
Avoid copy-pasting code between MySQL
and PostgreSQL
implementations wherever possible. For example:
If successful we will get PostgreSQL implementations almost for free, only tests will be needed.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.