Code Monkey home page Code Monkey logo

bitkv-rs's Introduction

Bitkv

github Build Codecov GitHub License

An efficient key-value storage engine, designed for fast reading and writing, which is inspired by Bitcask.

See Introduction, Installation and Usages for more details.

Introduction

Bitkv is a high-performance key-value storage system written in Rust. It leverages a log-structured design with an append-only write approach to deliver exceptional speed, reliability, and scalability.

Features

  • Efficient Key-Value Storage: Optimized for fast read and write operations with minimal overhead.
  • Diverse Index: Support BTree, Skiplist, BPlusTree index for multiple index strategies.
  • MemMap files for efficient I/O: To achieve rapid index reconstruction and enhance startup speeds
  • Low latency per item read or written: Benchmarks run on a Macintosh with Apple M1 Core:
    • Write latency: ~ 3.3 µs
    • Read latency: ~ 370 ns
  • Concurrency Support: fine-grained locking minimizes contentions.
  • WriteBatch transaction: commit a batch of writes to ensure atomicity.

Installation

To use bitkv in your project, add it as a dependency in your Cargo.toml file:

[dependencies]
bitkv-rs = "0.2.1"

Then, run cargo build to download and compile bitkv-rs and its dependencies.

For more detailed setup and compilation instructions, visit the Bitkv-rs GitHub repository.

Usages

Please see examples.

For detailed usage and API documentation, refer to the bitkv-rs Documentation.

TODO

  • Basic error handling
  • Merge files during compaction
  • Configurable compaction triggers and thresholds
  • WriteBactch transaction
  • Use mmap to read data file that on disk.
  • Optimize hintfile storage structure to support the memtable build faster
  • Http api server
  • Tests
  • Benchmark
  • Documentation
  • Increased use of flatbuffers option to support faster reading speed
  • Extend support for Redis Data Types

Contribution

Contributions to this project are welcome! If you find any issues or have suggestions for improvements, please raise an issue or submit a pull request.

License

Bitkv-rs is licensed under the [MIT license](https://github.com/example/bitkv-rs/blob/main/LICENSE-MIT), permitting use in both open source and private projects.
This license grants you the freedom to use bitkv-rs in your own projects, under the condition that the original license and copyright notice are included with any substantial portions of the Bitkv-rs software.

bitkv-rs's People

Contributors

kevinzh0a avatar dependabot[bot] avatar

Stargazers

V. Can Keklik avatar Jasper Bekkers avatar  avatar  avatar xy avatar yanzx avatar 你男朋友技术不错 avatar  avatar

Watchers

 avatar

bitkv-rs's Issues

Introduce multiple indexing methods #

After implementing the B+ tree as the default index for the engine, I've encountered an issue where batch writes and reads exceed the expected runtime, despite the absence of apparent errors in checking. If anyone has insights or suggestions on potential causes or solutions for this performance issue, I would greatly appreciate your advice. Insights or recommendations on possible reasons or corrective measures for this performance anomaly would be highly valued. Expert advice is sought to aid in refining and optimizing the indexing implementation. Thank you in advance for any suggestions!

Development Roadmap

Phase 1: Basic

Develop

  • Develop memory-disk architecture
  • Data read/write process
  • Design a database startup procedure
  • Data deletion process

Test

  • Unit tests
  • DB test

Phase 2: Core Functionalities

  • data file management strategy for efficient data write and retrieval
  • LogRecord encoding/decoding
  • Close and sync strategies
  • Implement WriteBatch atomic writes

Phase 3: Performance Optimization

Develop

  • #12
  • Implement MMap for enhanced file I/O operations
  • Establish data merge optimization strategies
  • Develop HTTP APIs for database interaction over the web.

Test

  • benchmark tests
  • HTTP request tests
  • Integrate Continuous Integration (CI) on GitHub Actions automate testing workflows
  • Code coverage over 80%

Documentation

  • Documentation 100%

Phase 4: Advanced Features and Integration

  • Increased use of flat buffer option to support faster reading speed
  • Integrate support for Redis data formats, enabling seamless migration and interoperability.

Roadmap Objectives and Milestones

  • Q1-Q2: Complete architectural foundation and core functionalities setup.
  • Q3: Focus on performance optimizations with advanced indexing and I/O strategies.
  • Q4: Broaden database capabilities with advanced features.
  • Q1 (Next): Extend Redis RESP protocol.

This roadmap is a living document and is subject to change as the project evolves. Welcome to write feedback or contributions to help shape the future of the project.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.