integrandio / lucidmq Goto Github PK

On startup, allow for the user to pass in a flag to indicate log level. This will be useful for reducing logs in CI and enabling them during local development.

In order to increase velocity and correctness, we need to be able to run integration tests during our CI stage. We will leverage docker and docker compose to accomplish this.

1. Create dockerfile that compiles and runs LucidMQ
2. Create a dockerfile that runs the integration tests in lucidmq-py
3. Create a docker-compose file that executes the complete workflow.
4. Create a github workflow step that runs the docker-compose file.

Right now producer in LucidMQ-py only sends one message. Let's create a clean API for sending multiple message including a batching solution for many that are sent in short succession.

There is a lot of code duplication in our LucidMQ and LucidMQ-cli code. We can reduce code by creating a shared library and importing it across both packages.

A consumer calls poll which waits for messages to come through the commit-log. Since it is using a RWLock there can be lock contention on the same topic/commit-log. If a consumer gives a high timeout, other operations could be locked out due to this timeout period. We need a less contentious heavy solution to no block all producers/consumers to the same topic.

In order to make our Python client easy to use, we want it pushed to PyPI.

1. Come up with a proper versioning scheme so protocols don't break unexpectedly
2. Add CI process for pushing to PyPi

Tests cover happy path. However, when a client sends bytes that cannot be deserialized to a protocol message, LucidMQ panics/breaks. We want to gracefully handle this and add to our test suite.

There is no standalone library for LucidMQ in rust. All that code is in the cli. Let's refactor it out and creat a client library.

Add a golang client library to interact with the LucidMQ instance.

1. All commands supported
2. protocol definitions in common protocol directory
3. build steps documented and automated with build scripts
4. Integration test suite built and supported similar to python suite

Utilize raft to make LucidMQ fault tolerant and distributed.

Library:
https://github.com/tikv/raft-rs

Here is an example of someone actually using the library:
https://www.hyperledger.org/blog/2019/01/11/floating-the-sawtooth-raft-implementing-a-consensus-algorithm-in-rust

The all top is message using the topic request message. This requires a topic name. Seems unnecessary to even send those bytes across the wire. We want a better solution for this as well as tests for this in out integ test suite.