Code Monkey home page Code Monkey logo

kekbit's Introduction

Kekbit

Cargo Documentation Rust 1.40+ GitHub Build Clippy codecov

Mean and lean components for working with ultralight persistent data channels in rust. Channels could be used for communication, transaction journaling, live replication of an application state or as a backend for persisting software system images.

Basic Concepts

Persistent data channels

  • A mechanism to sequentially persist data at a very fast rate
  • They are writer bound - it is a writer which creates them and specify the particular structure of a channel such size, maximum record length, or timeout
  • They have a fixed predefined capacity.
  • Once a channel is closed, is full, or is abandoned it will never be used again for writing.
  • They are byte-oriented sinks.
  • They are backed by a file; using a RAM disk for storage such as tempfs or /dev/shm could provide blazing fast speeds.
  • They always use little endian byte order.

Writers and Readers

  • Writers are components which push data into a persistent channel. For each channel there is only one writer.
  • Write operation can be done in stages using a chain of handlers.
  • Readers are components which poll data from a channel. Data available in the channel could be consumed multiple times, sequential or in parallel by multiple readers.
  • The default implementations for both readers and writers are non-blocking.
  • Readers also offer a straight Iterator API.

Usage

Add this to your Cargo.toml:

[dependencies]
kekbit = "0.3.4"

See the Examples for detailed usage.

Compatibility

The minimum supported Rust version is 1.31. Any change to this is considered a breaking change.

License

Licensed under MIT license (LICENSE or http://opensource.org/licenses/MIT)

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, shall be licensed as above, without any additional terms or conditions.

kekbit's People

Contributors

motoras avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

kekbit's Issues

ShmWriter allows sending non-Send type across threads

Hello fellow Rustacean,
we (Rust group @sslab-gatech) found a memory-safety/soundness issue in this crate while scanning Rust code on crates.io for potential vulnerabilities.

Issue Description

unsafe impl<H: Handler> Send for ShmWriter<H> {}

ShmWriter implements Send trait regardless of the inner type parameter H. This definition allows safe Rust code to send non-Send type across threads, which potentially causes a data race or undefined behavior.

H: Send trait bound should probably be added to ShmWriter's Send implementation. If all handlers are expected to be Send, then Send bound can be added to Handler trait's definition instead.

Reproduction

Below is an example program that shows non-Send type can be sent across threads using safe APIs of kekbit.

Show Detail

#![forbid(unsafe_code)]

use std::marker::PhantomData;
use std::thread;

use kekbit::api::Handler;
use kekbit::core::{shm_writer, Metadata, TickUnit::Nanos};

// non-Send type that panics when dropped in a wrong thread
struct NonSend {
    created_thread: thread::ThreadId,
    // Ensure `NonSend` type does not implement `Send` trait
    _marker: PhantomData<*mut ()>,
}

impl NonSend {
    pub fn new() -> Self {
        NonSend {
            created_thread: thread::current().id(),
            _marker: PhantomData,
        }
    }
}

impl Drop for NonSend {
    fn drop(&mut self) {
        if thread::current().id() != self.created_thread {
            panic!("NonSend destructor is running on a wrong thread!");
        }
    }
}

impl Handler for NonSend {}

fn main() {
    // Example code from: https://docs.rs/kekbit/0.3.3/kekbit/core/fn.shm_writer.html#examples
    const FOREVER: u64 = 99_999_999_999;
    let writer_id = 1850;
    let channel_id = 42;
    let capacity = 3000;
    let max_msg_len = 100;
    let metadata = Metadata::new(writer_id, channel_id, capacity, max_msg_len, FOREVER, Nanos);
    let test_tmp_dir = tempdir::TempDir::new("kekbit").unwrap();

    let writer = shm_writer(&test_tmp_dir.path(), &metadata, NonSend::new()).unwrap();

    let handle = thread::spawn(move || {
        // `NonSend` is sent to another thread via `ShmWriter`
        drop(writer);
    });

    handle.join().unwrap();
}

Output:

thread '<unnamed>' panicked at 'NonSend destructor is running on a wrong thread!', src/main.rs:44:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Any', src/main.rs:68:19

Return code 101

Tested Environment

  • Crate: kekbit
  • Version: 0.3.3
  • OS: Ubuntu 20.04.1 LTS
  • Rustc version: rustc 1.48.0 (7eac88abb 2020-11-16)
  • 3rd party dependencies:
    • tempdir = { version = "0.3.7" }

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.