echovault / echovault Goto Github PK

EchoVault is currently strongly consistent. This change will aim to make this policy configurable.

At the moment, echovault has a memberlist layer and a raft layer. The memberlist layer is mainly used to establish a cluster skeleton, essentially the "chassis" of the cluster. Memberlist nodes will communicate with other nodes to ask permission to join the raft cluster using the gossip protocol. Once the gossiped message reaches the raft cluster leader, the leader will add the node to the raft cluster.

We must allow a user to configure this functionality using the config flags. The option will enable the user to forgo the raft layer and use the memberlist layer for gossip-based state propagation to allow for eventual consistency. Otherwise, the user can opt for the strong consistency of the raft layer.

Use aof package in the raft module instead of BoltDB for logging and snapshotting

Use a stream cypher to encrypt the AOF and Snapshot files in standalone mode. The user will have to configure an encryption key.

Create test cases that cover all the commands in the ACL module.

Create test suite for custom snapshot engine.

Improve compatibility with Redis clients to allow EchoVault to be a potential drop-in replacement for Redis. The following considerations need to be made:

1. Handshake with Redis client.
2. Make sure each command returns a valid RESP response.
3. Make Pub/Sub module compatible with Redis client.

Embedded Lua Engine that allows users to build Lua modules that can be used to extend commands supported by EchoVault.

Generate docusaurus website for echovault's web home.

In order to keep memory usage in control, it's essential to have eviction policies that allow us to purge keys. As part of this issue, we would like to have the following eviction policies implemented:

1. noeviction - Do not evict any keys regardless of memory usage.
2. allkeys-lru - Use an LRU algorithm to remove the least recently used keys.
3. allkeys-lfy - Use an LFU algorithm to remove the least frequently used keys.
4. volatile-lru - Use an LRU algorithm to remove the least recently used keys with the expire property set to true.
5. volatile-lfu - Use an LFU algorithm to remove the least frequently used keys with the expire property set to true.
6. allkeys-random - Randomly remove any of the keys.
7. volatile-random - Randomly remove any of the keys that have the expire property set to true.
8. volatile-ttl - Remove keys with the shortest remaining ttl.

Allow the user to set up a shards using consistent hashing.

This feature should support:

1. Shards containing multiple single nodes.
2. Shards containing multiple replication clusters.

Reference: https://github.com/buraksezer/consistent

The handleCommand method checks if the caller is authorized to execute the provided command. This makes sense for clients connected over TCP. However, this check must be skipped for embedded clients as there's no need to authorize embedded API call commands.

Make EchoVault embeddable so any application can enhance its feature set with EchoVault features by simply importing EchoVault.

Bonus: Allow projects to import some modules from EchoVault. For example, import only the Append-Only engine or the Snapshot engine.

At the moment, HandlerFunc accepts a parameter that implements the EchoVault interafce. This causes the command modules to depend on the the echovault package, forcing the command modules to be placed in the pkg folder which is not idea.

The fix is to use Inversion of control to pass in all the functions, context, command and connection needed by the command handler, thus removing the dependence on the echovault package in any of the command modules.

This fix should also allow us to move the modules folder from /pkg to /internal. It should also make it easier to implement shared object file extension of echovault commands.

Expose a method in the embedded api to add a new command. This is similar to extension via .so files.

Created Godoc documentation for publishing a GO-embedded library on the v0.3.0 release.

As a user, I should be able to pass a list of .so files to be loaded on startup in order to extend EchoVault functionality with custom commands.

The plugins should be available in both client-server and embedded mode. For embedded mode, provide a generic function that accepts a custom []string to trigger custom plugin commands. It should return an error or a []byte representing the RESP response from the command handler.

Test all the command handlers in the pubsub module.

The ACL layer uses a command's KeyExtractionFunc to extract all the keys the command accesses.

KeyExtractionFunc returns a slice containing all the keys (read and write). However, the ACL layer cannot differentiate which keys are read and which ones are write.

This forces the ACL layer to check all returned keys against all allowed/excluded read and write keys.

TODO:

KeyExtractionFunc should return a struct containing a slice of read keys and a slice of write keys to allow the ACL layer to conduct a more granular check.

How can I run this project and make it reproducible?

error output

cd /root; cd octavia/diskimage-create/; ./diskimage-create.sh -a amd64 -d jammy -i ubuntu-minimal -o amphora-x64-haproxy-jammy.qcow2 -s 3

# go version
go version go1.22.2 linux/amd64

# git log -p -2
commit 2528082d4158680c7b59d7e526aec2b62885d8c4 (HEAD -> main, origin/main, origin/HEAD)
Author: Kelvin Mwinuka <[email protected]>
Date:   Thu Apr 11 11:01:45 2024 +0800

    Update README.md

diff --git a/README.md b/README.md
index 33bd7bd..e581f7a 100644
--- a/README.md
+++ b/README.md
@@ -7,9 +7,6 @@
 <br/>
 [![Go Reference](https://pkg.go.dev/badge/github.com/echovault/echovault.svg)](https://pkg.go.dev/github.com/echovault/echovault)
 <br/>
-[![Discord](https://img.shields.io/discord/1211815152291414037?style=flat&label=Discord&color=%235865F2)
-](https://discord.gg/aasBFwDm)
-<br/>
 <hr/>

 <img alt="echovault_logo" src="./images/EchoVault GitHub Cover.png" width="5062" />

commit c6b246efa455caaea53a6ba7a83115db826d36e1
Merge: 9b7d93f 64bbda6
Author: Kelvin Mwinuka <[email protected]>
Date:   Sat Apr 6 02:51:27 2024 +0800

    Merge pull request #21 from EchoVault/chore/fix-race-condition

    Fixed race conditions in acl module tests
#

# make build
env CC=x86_64-linux-musl-gcc GOOS=linux GOARCH=amd64 DEST=bin/linux/x86_64 make build-server
make[1]: вход в каталог «/root/EchoVault»
CC=x86_64-linux-musl-gcc GOOS=linux GOARCH=amd64 go build -o bin/linux/x86_64/server ./cmd/main.go
# runtime/cgo
cgo: C compiler "x86_64-linux-musl-gcc" not found: exec: "x86_64-linux-musl-gcc": executable file not found in $PATH
make[1]: *** [Makefile:2: build-server] Ошибка 1
make[1]: выход из каталога «/root/EchoVault»
make: *** [Makefile:5: build] Ошибка 2

Create a test suite for the Custom AOF Engine.