shelnutt2 / crunch Goto Github PK
View Code? Open in Web Editor NEWMariaDB storage engine based on cap'n proto storage
License: GNU Lesser General Public License v3.0
MariaDB storage engine based on cap'n proto storage
License: GNU Lesser General Public License v3.0
Replace stderr with my_error()/print_error() calls
Initial transaction support was added in #40 . This issue is to add the XA transaction support:
https://mariadb.com/kb/en/library/xa-transactions/
Update documentation - Create docs folder.
For info() we can compute data file size by adding a new parameter to the data_struct that has the file size. We can then also maintain a class variable for total size. This can be updated on each run of findTableFiles.
Improve handling of consolidation:
Add cap'n proto as vendor submodule and CMake check for if system version is missing or not >= v 0.6.1
Implement row level locking.
Consider changing on disk cap'n proto schema so that each column is stored as a field struct. There is a lot of metadata we are not storing right now, such as nullability of a column, or default values. Default values could be stored if they are a constant but if they are an expression we must rely on mariadb for storing these values in the .frm file.
In order to do auto table discovery, we have to store everything needed in the table data (data files or schema files) ourselves.
The limitation of inplace later tables when adding a column with a default and nullable (and it always being null for existing rows) is a because we are not storing this information.
The downside here, is right now each row is stored simply, does it really make sense to pack all this extra information into each and every single row? It would great increase disk space, and processing of data that is constant (for the schema version).
Perhaps we introduce a new data file that goes along with the schema, and contains the metadata of the table? We keep the "rows" struct compact and simple, and we create new struct to present the table and all the metadata. The advantage this way is the data files stay compact, we only have to write "table metadata" once per schema change. The downside is in any single row data file you'd be missing the data required for logic, such as if a column has no value but uses a default expression. New struct/metadata files also increases the complexity. Right now storage is simple. The downside is virtual columns are not supported.
I'm leaning toward new struct and new data files. It does not make sense to expand the row structure so that every column is a field struct and contains all metadata, it'd be a massive amount of duplication, and we have no compression (yet)!
CI builds can be speed up if we limit what we build with mariadb.
Add support for bit datatype
Need to add support for a testing framework. We need the ability to add mysql tests for this storage engine.
myisam handles with with test executables, however I'd like to use the mysql testing platform, so tests can be run inside the database to ensure all functions are implemented correctly.
Add support for year datatype
The alter table test with multiple inplace alters that changes a mediumint to a bigint is failing in travis ci. This failure can not be preproduced anywhere else but travis ci. It was disabled, but should be investigated.
Sample failure: https://travis-ci.org/Shelnutt2/crunch/builds/343135301
exception on rnd_next ./test/t1: capnp/layout.c++, line: ../storage/crunch/src/crunch.cpp:346, exception_line: 2159, type: 0, e.what(): expected boundsCheck(segment, ptr, ref->structRef.wordSize()); Message contained out-of-bounds struct pointer.
2018-02-18 22:32:29 139966787450624 [ERROR] mysqld: Got error -44 "Internal error < 0 (Not system error)" from storage engine Crunch
Add support for non-sequential reads, rnd_pos and position
Cap'n proto does not support underscore in field names and enforces camel case. Need to convert column names which often have underscores to camel case.
in #50 online (inplace) alter table support was added for column additions, name changes and dropping columns.
Changing column datatypes to non-compatible datatype (e.g. int to a string) requires a rebuild. This should be handled with online alters. The basic requirements are to create conversion process from one data type to another, i.e itoa, or atoi. Then on reads this will be converted on the fly. When a consolidation happens the data on disk should be updated so the conversion is no longer required.
Properly set engine table flags
Add support for auto increment column
OPTIMIZE TABLE Should Run Consolidation
Switch CI tests to use docker image for build dependencies. This will prevent having to compile capnp proto for every test, which adds 5-10 minutes to the testing. We can also skip the ubuntu dependencies and increase testing to other gcc/clang version with little overhead.
Initial delete support was added in #18. Initial writes in #1.
This issue is to extend the data/delete support to support multiple files containing the data and deletes. The basic idea is anytime an update or delete statement is called, we will end writing new rows to the data file, start a new data file and move deletes to the new file.
Add support for geometry data types.
Add support for storing date, datetime in struct fields.
Add support for spatial indexing. Blocked by #29
We should move data files into data folder. This would make it easier to atomically consolidate files. Right now we rename/move all existing files into a "consolidate folder", then we rename the new data file from transactions to the main folder than we delete the consolidate folder. Its too make operations, and not atomic enough in case of failure.
What we want to do is create data_dirs and then just symlink from data_dir_X to data. This can be done atomically:
http://blog.moertel.com/posts/2005-08-22-how-to-change-symlinks-atomically.html
Add support for handling of null field
Add transaction support.
This involves storing transaction data for deletes and writes in new files. On commit of a transaction the files are closed and moved from a transactions working directory to the main table folder.
Need to clear out transaction folder on startup.
Right now unsigned integers are not supported. This is simple because we are only checking the field type in getCapnpTypeFromField and build_row. We need to check if unsigned bit is set and create schema to store appropriately.
Add support for basic inserts
Add travis ci support for mariadb 10.2, linux and osx.
Segfault from freeing pointer on closing table
Add primary key support
Right now table locks are all that is supported. When #13 is implemented, we will have more fine grain control. This issue will happen after #13.
The basic idea is that we can used shared read only locks for everything but deletes. For deletes, if we want to implement the possibility of read only locks, we'd need to keep a mapping of old file, old position to new file, new position, so we can port the deletes that happen while consolidation is running. The commit phase always has to be an exclusive lock.
By nature of the using write once files per transaction, we only have up to two atomic operations per table during transaction commits. One rename of the data file, one rename of the delete file.
From a single table perspective (simplest transaction), if a crash were to occur after only a single atomic operation were to complete, on start up we must rollback and delete this partial transaction. The transaction can be determined by comparing the data folder to the transactions folder. Since all transaction files have the same name and only differ in extension it is possible to find the partially committed transaction.
This must be done only on crash recovery as during normal operation it is easily possible to encounter this half committed transaction if a transaction is mid-flight.
Transactions can span across multiple tables, and in that case the same basic operation can occur. All tables get the same transaction id (epoch_nanosections+uuid). Thus checking across all tables for any partial commits is doable. This however only works if the transaction involved two or more crunch tables. If this is cross engine, we can not do this effectively without XA commits.
The idea is to add support for online table alters by creating new version of the schema. We can then produce a migration of data on read. This would put the work on reads instead of causing a table rebuild when a column is changed. During file consolidation older data files can be upgraded to the latest version of the schema.
If column order is what is being changed nothing has to be done. Even if ondisk storage and capnp proto schema is in different order, the fields are set using the field interface, so no changes are needed.
If column name changes, then we can just create a new capn proto schema with the new names, but as long as order and data types are the same, then the data on disk doesn't need to change.
Changes are needed for:
Implement:
We should use fswatch for a cross platform file monitoring solution so we can mremap the data file when it changes, or reload the schema.
https://github.com/emcrisostomo/fswatch
This will need to be run from a separate thread and use a lock to prevent changes during active reads.
Marko was able to show me how innodb sets the default value in their instant add column. They basically call ->set_default on the field, then they read the field value. We should be able to do the same.
Capn'proto expects all text to be in utf-8. Right now when we get blob or char/varchar data we assume it is in utf-8. Instead of assume, a conversion should be done.
Add support for a daemon plugin which can manipulate a table in order to perform maintenance operations. Initially this will be limited to file consolidation.
Two strategies that need to be examined are having the daemon use the handler interface to interact with the table structure, similar to handler_socket plugin. The second strategy is to link directly to ha_crunch library and manipulate the table outside of mariadb handler structure. The main issue with the second method is locks still need to be acquired in order to safely consolidate the files.
Timestamp to Datetime conversions that happen in message upgrades are not correct. Timestamps are stored in UTC and mysql converts it to server timezone after select. During a normal offline alter table timestamps are converted to server timezone then written to the datetime field. We should do the same.
Add support for storing blobs in blob type of capnp proto.
Records, currentRowNum need to be 64bit
Add support for consolidating data and delete files by reading non-deleted rows into a consolidated file.
The approach to be discussed here is to perform partial table scans and consolidate in chunks.
Add order by for tests to work around travis ci failures. Sometimes the resulting transaction files are ending in a different order in the travis ci vm. I am unable to reproduce this locally, in docker or in kvms. For now to prevent the false test failures, we will add order bys to test with multiple results.
See ha_maria::idx_cond_push
Switch travis ci compilation to use ninja-build in order to reduce test time.
Delete support is needed. Deletes can be done in multiple ways.
"1)" or "2) "are pretty equivalent. The advantage to 2 is during a table scan one does not have to parse the message only to find it has been deleted. It might also be that option 1 makes roll back easier. However with 1 or 2 we still need to maintain a list of the ongoing rows touched in the transactions.
"3)" Does not seem to have a large benefit. If we keep the rows separate, then we just have to read that into memory and still do a comparison. The only upside compared to 1, is we don't have to parse capnp proto message to see if it is deleted or not, we can store the file offset and skip that way.
With option 2) We can also have a daemon process that periodically reorganizes a table that is closed, so zero'ed out space that is not a message is removed, truncating the file size.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.