Code Monkey home page Code Monkey logo

reduct-cpp's Introduction

ReductStore

A high-performance time series database for blob data

GitHub release (latest SemVer) GitHub Workflow Status Docker Pulls GitHub all releases Discord

ReductStore is a time series database that is specifically designed for storing and managing large amounts of blob data. It boasts high performance for both writing and real-time querying, with the added benefit of batching data. This makes it an ideal solution for edge computing, computer vision, and IoT applications where network latency is a concern. For more information, please visit https://www.reduct.store/.

Why Does It Exist?

There are numerous time-series databases available in the market that provide remarkable functionality and scalability. However, all of them concentrate on numeric data and have limited support for unstructured data, which may be represented as strings.

On the other hand, S3-like object storage solutions could be the best place to keep blob objects, but they don't provide an API to work with data in the time domain.

There are many kinds of applications where we need to collect unstructured data such as images, high-frequency sensor data, binary packages, or huge text documents and provide access to their history. Many companies build a storage solution for these applications based on a combination of TSDB and Blob storage in-house. It might be a working solution; however, it is a challenging development task to keep data integrity in both databases, implement retention policies, and provide data access with good performance.

The ReductStore project aims to solve the problem of providing a complete solution for applications that require unstructured data to be stored and accessed at specific time intervals. It guarantees that your data will not overflow your hard disk and batches records to reduce the number of critical HTTP requests for networks with high latency.

All of these features make the database the right choice for edge computing and IoT applications if you want to avoid development costs for your in-house solution.

Features

  • Storing and accessing unstructured data as time series
  • No limit for maximum size of blob
  • Real-time FIFO bucket quota based on size to avoid disk space shortage
  • HTTP(S) API
  • Append-only replication
  • Optimized for small objects (less than 1 MB)
  • Labeling data for annotation and filtering
  • Iterative data querying
  • Batching records in an HTTP response for write and read operations
  • Embedded Web Console
  • Token authorization for managing data access

Get Started

The quickest way to get up and running is with our Docker image:

docker run -p 8383:8383 -v ${PWD}/data:/data reduct/store:latest

Alternatively, you can opt for Cargo:

apt install protobuf-compiler
cargo install reductstore
RS_DATA_PATH=./data reductstore

For a more in-depth guide, visit the Getting Started and Download sections.

After initializing the instance, dive in with one of our Client SDKs to write or retrieve data. To illustrate, here's a Python sample:

import time
import asyncio
from reduct import Client, Bucket

async def main():
    # Create a client for interacting with a ReductStore service
    async with Client("http://localhost:8383") as client:
        # Create a bucket and store a reference to it in the `bucket` variable
        bucket: Bucket = await client.create_bucket("my-bucket", exist_ok=True)

        # Write data to the bucket
        ts = time.time_ns() / 1000
        await bucket.write("entry-1", b"Hey!!", ts)

        # Read data from the bucket
        async with bucket.read("entry-1", ts) as record:
            data = await record.read_all()
            print(data)

# Run the main function
loop = asyncio.get_event_loop()
loop.run_until_complete(main())

Client SDKs

ReductStore is built with adaptability in mind. While it comes with a straightforward HTTP API that can be integrated into virtually any environment, we understand that not everyone wants to interact with the API directly. To streamline your development process and make integrations smoother, we've developed a series of client SDKs tailored for different programming languages and environments. These SDKs wrap around the core API, offering a more intuitive and language-native way to interact with ReductStore, thus accelerating your development cycle. Here are the client SDKs available:

Tools

ReductStore is not just about data storage; it's about simplifying and enhancing your data management experience. Along with its robust core features, ReductStore offers a suite of tools to streamline administration, monitoring, and optimization. Here are the key tools you can leverage:

  • CLI Client - a command-line interface for direct interactions with ReductStore
  • Web Console - a web interface to administrate a ReductStore instance

Feedback & Contribution

Your input is invaluable to us! ๐ŸŒŸ If you've found a bug, have suggestions for improvements, or want to contribute directly to the codebase, here's how you can help:

  • Discord: Join our Discord community to discuss, share ideas, and collaborate with fellow ReductStore users.
  • Feedback & Bug Reports: Open an issue on our GitHub repository. Please provide as much detail as possible so we can address it effectively.
  • Contribute: ReductStore is an open-source project. We encourage and welcome contributions.

Get Involved

We believe in the power of community and collaboration. If you've built something amazing with ReductStore, we'd love to hear about it! Share your projects, experiences, and insights on our Discord community.

If you find ReductStore beneficial, give us a โญ on our GitHub repository.

Your support fuels our passion and drives us to keep improving.

Together, let's redefine the future of blob data storage! ๐Ÿš€

Frequently Asked Questions (FAQ)

Q1: What sets ReductStore apart from other time-series databases?

A1: ReductStore is specially designed for storing and managing large amounts of blob data, optimized for both high performance and real-time querying. Unlike other databases that focus primarily on numeric data, ReductStore excels in handling unstructured data, making it ideal for various applications like edge computing and IoT.

Q2: How do I get started with ReductStore?

A2: You can easily set up ReductStore using our Docker image or by using cargo. Detailed instructions are provided in the Getting Started section.

Q3: Is there any size limitation for the blob data?

A3: While ReductStore is optimized for small objects (less than 1 MB), there's no hard limit for the maximum size of a blob.

Q4: Can I integrate ReductStore with my current infrastructure?

A4: Absolutely! With our variety of client SDKs and its adaptable HTTP API, ReductStore can be integrated into almost any environment.

Q5: I'm facing issues with the installation. Where can I get help?

A5: We recommend checking out our documentation. If you still face issues, feel free to join our Discord community or raise an issue on our GitHub repository.

reduct-cpp's People

Contributors

anthonycvn avatar atimin avatar toge avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Forkers

mounte toge

reduct-cpp's Issues

Feature: Support Reduct Storage HTTP API 0.7

Current problem

We don't support features of HTTP API 0.7:

  • We don't need to hash API token. We could remove SHA256 encoding
  • Querying data

Possible solution

For querying data, we could do something like this:

client->Query("entry-1", [](const Record& record) {
  auto data = record.Read();
  // or in chunks
  [&os](auto data) {
    fos << data;
    return true;
  }
});

Additional context

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct

Test cmake installation in CI

Current ubuntu version in GitHub action is 20.04, we should wait for the upgrade and make installation test in CI action

Feature: Support labels

Current problem

Since version 1.3, ReductStore supports labels for records and allows querying them by labels. The C++ SDK should support it.

Possible solution

Example of API:

// write a record with labels
auto err = bucket->Write("entry-1", {}, {{"quality":"good"}}, [](auto rec) { rec->WriteAll("some_data2"); });

// read a record whith labels
err = bucket->Read("entry-1", ts, [](auto rec) {
        std::cout << "Labels: " <<  rec->labels << std::endl;
});

// query records with include and exclude filters
err = bucket->Query("entry-1", start, IBucket::Time::clock::now(), {.inlude = {...}, .exclude={...}} [](auto&& record) {
        std::cout << "Labels: " <<  rec->labels << std::endl;
});

Additional context

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct

CMake Integration

Currently, the library can be built only with conan and there is no install script.

  1. use find_package to use dependencies
  2. add install command to CMakeLists.txt
  3. Make FindReductCpp macros and add to install script
  4. Add CI a test to check installation.

Feature: Support Reduct Storage v1.0

Current problem

In v1.0.0 the storage engine changed the API paths, see reductstore/reductstore#178

Possible solution

We should change the API paths in the SDK too and remove deprecated endpoints

Additional context

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct

Doesn't work with FetchContent

When I try to use the library with FetchContent:

include(FetchContent)

FetchContent_Declare(
        reduct_cpp
        URL https://github.com/reduct-storage/reduct-cpp/archive/refs/tags/v0.6.0.zip
        URL_HASH MD5=8be8b5154193e451fc1599b9ecfeb701
)

FetchContent_MakeAvailable(reduct_cpp)

I have this error:

ERROR: Conanfile not found at /home/atimin/Projects/flipback/reduct-storage/cli-client/conanfile.py
CMake Error at cmake-build-debug/conan.cmake:631 (message):
  Conan install failed='1'
Call Stack (most recent call first):
  cmake-build-debug/_deps/reduct_cpp-src/cmake/InstallDependencies.cmake:14 (conan_cmake_install)
  cmake-build-debug/_deps/reduct_cpp-src/CMakeLists.txt:19 (include)

Read/write records by chunks

Current API provides methods to read records to string:

class IBucket {
  public:
  virtual Error Write(std::string_view entry_name, std::string_view data,
                      Time ts = Time::clock::now()) const noexcept = 0;

  virtual Result<std::string> Read(std::string_view entry_name,
                                   std::optional<Time> ts = std::nullopt) const noexcept = 0;
}

To integrate it to Reduct Storage, we should add asynchronous read/write methods

class IBucket {
public:
  using Sender = std::function<bool(/*TBD*/)>;
  using Receiver = std::function<bool(/*TBD*/)>;
  
  virtual Error Write(std::string_view entry_name, Sender sender) const noexcept = 0;
  virtual Error Read(std::string_vew entry_name, Recevier receiver) const noexcept = 0;
 
}

For implementation see:

https://github.com/yhirose/cpp-httplib#receive-content-with-a-content-receiver-1
https://github.com/yhirose/cpp-httplib#send-content-with-a-content-provider

Feature: Use `Record` struct for methods `Bucket.Read` and `Bucket.Write`

Current problem

For querying, we have Record struct to read a record:

  auto err = bucket->Query("entry-1", start, IBucket::Time::clock::now(), std::nullopt, [](auto&& record) {
    std::string blob;
    auto read_err = record.Read([&blob](auto chunk) {
      blob.append(chunk);
      return true;  // return false if you want to aboard the receiving
    });

    if (!read_err) {
      std::cout << "Read blob: " << blob;
    }

    return true; // return false if you want stop iterating
  });

We can use the same approach for Bucket.Read and Bucket.Write methods.

Possible solution

  struct ReadableRecord {
    Time timestamp;
    size_t size;
    bool last;

    /**
     * Function to receive data in chunks
     */
    std::function<Error(ReadCallback)> Read;
  };

struct WrtiableRecord {
    Time timestamp;
    size_t size;

    std::function<Error(WriteCallback)> Write;
}

class   IClient {
public:
  virtual Error<ReadableRecord> Read(std::string_view entry_name, std::optional<Time> ts) const noexcept = 0;

  virtual Error Write(std::string_view entry_name, WritableRecord record) noexcept = 0;

Additional context

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct

Refactor tests

Currently, the tests create buckets with random names and many tests can't be done precisely. We should refactor them by creating and removing a bucket for each test. Like it is in reduct-js project.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.