zsongshu / elastic-stream Goto Github PK

ElasticStream is an open-source, elastic streaming platform. ElasticStream is designed to stand on shoulders of cloud infra-structure, excelling itself in scalability, fault-tolerance, cost-effectiveness and security.

ElasticStream has three components: placement-driver, range-server and clients.

Primary design goal of range-server is to scale linearly with addition of hardware: including CPU and SSD. To achieve this, threads of it must be as independent from one another as possible, to avoid software locks and even atomic instructions. With that said, instead of centralizing shared data in a global location that all threads access after acquiring a lock, each thread of range-server has its own data. When other threads want to access the data, they pass a message to the owning thread to perform the operation on their behalf through lockless ring-buffers.

Range Server has 3 categories of threads: worker threads, IO thread and miscellaneous auxiliary threads. The former two kinds follows Thread-per-Core architecture while auxiliary threads use traditional multi-threads parallelism, with CPU affinity set.

Each range-server, by default, has one IO thread, taking up a dedicated core, running an io_uring instance with SQ_POLLING and IO_POLLING. This configuration offers best latency performance and makes most the modern storage capabilities.

Multiple independent workers runs in parallel within a range-server. Each of them are backed by an io_uring instance with FAST_POLL feature set. A worker also takes up a dedicated core and is in charge of a group of streams. Inter-worker communication is fulfilled by way of message passing, aka, lockless ring buffer.

This project employs several distinct programming languages. The Placement Driver is developed utilizing Go, whereas the Range Server leverages the Rust language. In addition, we furnish two SDK implementations with Java and Rust.

For the parts crafted in Go and Java, you can locate the corresponding build commands in the ./pd and ./sdks/frontend-java directories respectively.

Considering the necessity for unified build and testing process, streamlined dependency management, and consistent project-wide coding practices, we've placed all our Rust crates within one workspace.

Particularly, some crates might have additional dependencies when building. please run ./scripts/install_deps.sh before you kick things off. Given that the top-level directory of the project doubles as a crate root, you're empowered to run commands below to build it.

# debug mode.
cargo build
# release mode.
cargo build --release

cargo llvm-cov

With HTML report,

cargo llvm-cov --html

or

cargo llvm-cov --open

If wishing to execute cargo run instead of cargo test, use run sub-command:

cargo llvm-cov run

Read more

Sample integration with github action.

We welcome contribution of various types: documentation, bug reporting, feature request, and pull requests. Before proposing a major large-scale change, it is advised to discuss your proposal first.

Store module is built on top of io-uring directly. The Server module, however, is built using tokio-uring, following thread-per-core paradigm, which as a result is fully async. Reading and writing records between these two modules involve communication between async and sync code, as shall comply with the following guideline

Sometimes you have to deal with low-level operations, for example, interacting with DMA requires page alignment memory. Unsafe code is required to handle these cases and address sanitizer would be helpful to maintain memory safety.

RUSTFLAGS=-Zsanitizer=address cargo test test_layout -Zbuild-std --target x86_64-unknown-linux-gnu

Read the following link for more advanced usage.

cargo run --bin range-server -- start --log etc/range-server-log.yaml