greenlion / warp Goto Github PK

WARP can use all the parts of an explicit key so remove the logic that enforces the rules for b-tree searches on range scans.

where textcol < 1 can not be evaluated by Fastbit and returns wrong results. Comparisons between strings and integers need to be handled at the MySQL layer.

Create a basic feature test using mysqltest

Investigate the SE storage engine interface for partitioning and implement partitioning for WARP.

n/t

To support faster loading of data, it should be possible to disable UNIQUE and PRIMARY KEY checking. Of course, this could allow duplicate data into tables, because WARP is not index organized and a UNIQUE/PRIMARY KEY is just a hint to check for duplicates at insertion/update time.

This is an on-disk format change, so it should be part of beta 2 so that tables do not need to be repaired/reconstructed in beta3.

n/t

Need to add trx_id to each table row similar to rowid
Need to add visibility support to transactions so that only visible transactions are shown to a transaction.
Add row level locking

If a statement errors out after inserting rows, the rows do not roll back. All statements should be atomic. The same goes for deletions and updates. An operation cancelled in the middle leaves a partially changed base table. This is the same as MyISAM.

Bonus: add transaction support for more than one table.

Since Percona Server supports RocksDB and has other useful enhancements, fork it instead of MySQL.

DROP TABLE works fine, but MySQL does not call handler::delete_table for all tables in a database when the database is dropped.

Fastbit is not collation aware - need to compare strings in the bitmaps using case insensitive collation comparison when ci collation is used.

Make the Fastbit rowid value available to the MySQL table if the special column name $rowid is used in a table. Insertions should ignore any $rowid value passed in during an insert.

MySQL 8.0 can clone a server from another server. Right now only InnoDB is supported by the server for cloning. Cloning WARP based tables should be supported as well. This will require modifying the server and won't be pluggable so it is a semi-non-compatible change to upstream MySQL. WARP could still be loaded into a non WarpSQL server, but cloning would clone WARP tables as empty.

Merging upstream 8.0.20 broke the engine. Merge it again in a branch and investigate why it no longer works.

Because FastBit is an append-only database, WARP does not do in place updates. This means tables with frequent updates will grow substantially since old row versions are maintained until an OPTIMIZE table operation is run on tables. WARP should be able to automatically rebuild tables in the background to remove old versions.

Is a MDX transformer in the roadmap? If so will it be Pentahoo/Mondrian interpretation of the spec of something that conforms to the MS/Power BI spec..

If useful there is a pretty simple implementation of the Mondrian tools available at https://github.com/Wondersoft/olaper

I have been a fan of the swanhart-tools for along time and made good use of "Shard Query" and "Flex Views" over the years. Really glad to see it maturing.

N/t

Use vectorized instructions for aggregate instructions.

Store WARP tables in an alternative data directory by default

n/t

Fasbit has some code to do bitmap index joins. Experiment with pushing down joins to fastbit to see how performance compares vs hash and nl joins.

Figure out why LOAD DATA INFILE can not load in parallel.

Aggregation functions like SUM/STDDEV/etc can overflow on bigint or decimal expressions. Add new data types to prevent this. This is important for scientific computing.

Fastbit support Windows, so the storage engine just needs to have some minor modification for Windows support including using proper path separators, etc.

n/t

TRUNCATE TABLE works correctly, but DELETE w/out WHERE clause does not.

where l_shipdate = '1992-01-01' -- not pushed down
where l_shipdate IS NULL -- wrong results!

Having a single file per column is convenient, but lots of fsync are necessary for ACID compliance. Implement a container format (ie tablespace) that stores all columns in a single file.

The following should be supported without rebuilding the entire table but still require a table lock:
Adding keys
Dropping keys
Adding columns
Dropping columns
Changing default values

The following should require rebuilding a table:
ADD PRIMARY KEY
ADD UNIQUE KEY

Port some shard-query logic into the database directly for parallel query.

When a multi-value insert (or load data) writes data to a table, the inserts are cached in a ibis::tablex object up to a maximum size. Unfortunately it is not possible to query tablex objects, they are only for appending data, so insertions of duplicate keys in an individual batch can not be detected easily. Find a way around this to improve insertion performance with UNIQUE/PRIMARY keys.

Right now the deleted rid bitmap is read in in 8 byte chunks. That is a lot of fread() calls (even though the OS should buffer the reads). A table with 64 million rows only takes 8MB of space to store the bitmap entirely in ram, so it makes sense to read the whole thing in and just point ->bits to the proper offset in the buffer.

N/t

n/t

Current FlexCDC runs as an external daemon. Convert the PHP source base into C++ and integrate as a daemon plugin.

old versions and rolled back transactions consume space in the table - implement background purge operation to remove old data

To improve insertion performance, rowid values (which are 64 bit values) should be handed out in batches.

This could result in partial reads/writes. Wrap read/write system calls with retry logic for interrupted system calls.

Conjuctions with functions should push down the constant evaluation portion, and push the remaining portion to the SQL layer for evaluation.

n/t

Currently the first column is used for count() - instead, find the smallest column (or the first NULL marker) and use that for count() to reduce IO.

n/t

Add the SQL SHIM plugin interface into the current version of WarpSQL Server.
https://swanhart.livejournal.com/139967.html
https://swanhart.livejournal.com/140050.html

This has been a handy way to persist realtime data like logs/stat states. I have made good use of it in telecom with CDR data. https://dev.mysql.com/doc/refman/8.0/en/innodb-memcached.html

Add RLE zlib compression for columnar data.

Allow WARP tables to be created in a different DATA_DIR

In order to support CDC, transaction id values should use the InnoDB transaction id generator. InnoDB is always available in the server because it is the DD storage engine.