WarpSQL Server, an open source OLAP focused distribution of the world's most popular open source database bundled with OLAP performance related plugins such as the WARP storage engine..
where textcol < 1 can not be evaluated by Fastbit and returns wrong results. Comparisons between strings and integers need to be handled at the MySQL layer.
To support faster loading of data, it should be possible to disable UNIQUE and PRIMARY KEY checking. Of course, this could allow duplicate data into tables, because WARP is not index organized and a UNIQUE/PRIMARY KEY is just a hint to check for duplicates at insertion/update time.
Need to add trx_id to each table row similar to rowid
Need to add visibility support to transactions so that only visible transactions are shown to a transaction.
Add row level locking
If a statement errors out after inserting rows, the rows do not roll back. All statements should be atomic. The same goes for deletions and updates. An operation cancelled in the middle leaves a partially changed base table. This is the same as MyISAM.
Bonus: add transaction support for more than one table.
Make the Fastbit rowid value available to the MySQL table if the special column name $rowid is used in a table. Insertions should ignore any $rowid value passed in during an insert.
MySQL 8.0 can clone a server from another server. Right now only InnoDB is supported by the server for cloning. Cloning WARP based tables should be supported as well. This will require modifying the server and won't be pluggable so it is a semi-non-compatible change to upstream MySQL. WARP could still be loaded into a non WarpSQL server, but cloning would clone WARP tables as empty.
Because FastBit is an append-only database, WARP does not do in place updates. This means tables with frequent updates will grow substantially since old row versions are maintained until an OPTIMIZE table operation is run on tables. WARP should be able to automatically rebuild tables in the background to remove old versions.
Is a MDX transformer in the roadmap? If so will it be Pentahoo/Mondrian interpretation of the spec of something that conforms to the MS/Power BI spec..
I have been a fan of the swanhart-tools for along time and made good use of "Shard Query" and "Flex Views" over the years. Really glad to see it maturing.
Aggregation functions like SUM/STDDEV/etc can overflow on bigint or decimal expressions. Add new data types to prevent this. This is important for scientific computing.
Fastbit support Windows, so the storage engine just needs to have some minor modification for Windows support including using proper path separators, etc.
Having a single file per column is convenient, but lots of fsync are necessary for ACID compliance. Implement a container format (ie tablespace) that stores all columns in a single file.
The following should be supported without rebuilding the entire table but still require a table lock:
Adding keys
Dropping keys
Adding columns
Dropping columns
Changing default values
The following should require rebuilding a table:
ADD PRIMARY KEY
ADD UNIQUE KEY
When a multi-value insert (or load data) writes data to a table, the inserts are cached in a ibis::tablex object up to a maximum size. Unfortunately it is not possible to query tablex objects, they are only for appending data, so insertions of duplicate keys in an individual batch can not be detected easily. Find a way around this to improve insertion performance with UNIQUE/PRIMARY keys.
Right now the deleted rid bitmap is read in in 8 byte chunks. That is a lot of fread() calls (even though the OS should buffer the reads). A table with 64 million rows only takes 8MB of space to store the bitmap entirely in ram, so it makes sense to read the whole thing in and just point ->bits to the proper offset in the buffer.
In order to support CDC, transaction id values should use the InnoDB transaction id generator. InnoDB is always available in the server because it is the DD storage engine.