Code Monkey home page Code Monkey logo

echovault's Introduction

Go Go Report Card codecov
GitHub Release License
Go Reference


echovault_logo

What is EchoVault?

EchoVault is a highly configurable, distributed, in-memory data store and cache implemented in Go. It can be imported as a Go library or run as an independent service.

EchoVault aims to provide a rich set of data structures and functions for manipulating data in memory. These data structures include, but are not limited to: Lists, Sets, Sorted Sets, Hashes, and much more to come soon.

EchoVault provides a persistence layer for increased reliability. Both Append-Only files and snapshots can be used to persist data in the disk for recovery in case of unexpected shutdowns.

Replication is a core feature of EchoVault and is implemented using the RAFT algorithm, allowing you to create a fault-tolerant cluster of EchoVault nodes to improve reliability. If you do not need a replication cluster, you can always run EchoVault in standalone mode and have a fully capable single node.

EchoVault aims to not only be a server but to be importable to existing projects to enhance them with EchoVault features, this capability is always being worked on and improved.

Features

Features offered by EchoVault include:

  1. TLS and mTLS support for multiple server and client RootCAs.
  2. Replication cluster support using the RAFT algorithm.
  3. ACL Layer for user Authentication and Authorization.
  4. Distributed Pub/Sub functionality with consumer groups.
  5. Sets, Sorted Sets, Hashes, Lists and more.
  6. Persistence layer with Snapshots and Append-Only files.
  7. Key Eviction Policies.
  8. Command extension via shared object files.
  9. Command extension via embedded API.

We are working hard to add more features to EchoVault to make it much more powerful. Features in the roadmap include:

  1. Sharding
  2. Streams
  3. Transactions
  4. Bitmap
  5. HyperLogLog
  6. Lua Modules
  7. JSON
  8. Improved Observability

Usage (Embedded)

Install EchoVault with: go get github.com/echoVault/echoVault. Run go mod tidy to pull all of EchoVault's dependencies.

Here's an example of using EchoVault as an embedded library. You can access all of EchoVault's commands using an ergonomic API.

func main() {
	server, err := echovault.NewEchoVault()

	if err != nil {
		log.Fatal(err)
	}

	_, _ = server.Set("key", "Hello, world!", echovault.SETOptions{})
	
	v, _ := server.Get("key")
	fmt.Println(v) // Hello, world!

	wg := sync.WaitGroup{}

	// Subscribe to multiple EchoVault channels.
	readMessage := server.Subscribe("subscriber1", "channel_1", "channel_2", "channel_3")
	wg.Add(1)
	go func() {
		wg.Done()
		for {
			message := readMessage()
			fmt.Printf("EVENT: %s, CHANNEL: %s, MESSAGE: %s\n", message[0], message[1], message[2])
		}
	}()
	wg.Wait()

	wg.Add(1)
	go func() {
		for i := 1; i <= 3; i++ {
			// Simulating delay.
			<-time.After(1 * time.Second)
			// Publish message to each EchoVault channel.
			_, _ = server.Publish(fmt.Sprintf("channel_%d", i), "Hello!")
		}
		wg.Done()
	}()
	wg.Wait()

	// (Optional): Listen for TCP connections on this EchoVault instance.
	server.Start()
}

An embedded EchoVault instance can still be part of a cluster, and the changes triggered from the API will be consistent across the cluster.

Usage (Client-Server)

Homebrew

To install via homebrew, run:

  1. brew tap echovault/echovault
  2. brew install echovault/echovault/echovault

Once installed, you can run the server with the following command: echovault --bind-addr=localhost --data-dir="path/to/persistence/directory"

Binaries

You can download the binaries by clicking on a release tag and downloading the binary for your system.

Clients

EchoVault uses RESP, which makes it compatible with existing Redis clients.

Documentation

https://echovault.io

echovault's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

echovault's Issues

Differentiate Read and Write Keys when Authorising commands in ACL Layer

The ACL layer uses a command's KeyExtractionFunc to extract all the keys the command accesses.

KeyExtractionFunc returns a slice containing all the keys (read and write). However, the ACL layer cannot differentiate which keys are read and which ones are write.

This forces the ACL layer to check all returned keys against all allowed/excluded read and write keys.

TODO:

KeyExtractionFunc should return a struct containing a slice of read keys and a slice of write keys to allow the ACL layer to conduct a more granular check.

Skip ACL Authorisation for Embedded API Calls

The handleCommand method checks if the caller is authorized to execute the provided command. This makes sense for clients connected over TCP. However, this check must be skipped for embedded clients as there's no need to authorize embedded API call commands.

Refactor command modules to use dependency injection of required parameters for command handlers

At the moment, HandlerFunc accepts a parameter that implements the EchoVault interafce. This causes the command modules to depend on the the echovault package, forcing the command modules to be placed in the pkg folder which is not idea.

The fix is to use Inversion of control to pass in all the functions, context, command and connection needed by the command handler, thus removing the dependence on the echovault package in any of the command modules.

This fix should also allow us to move the modules folder from /pkg to /internal. It should also make it easier to implement shared object file extension of echovault commands.

Allow shared object (.so) plugins to be loaded into commands list

As a user, I should be able to pass a list of .so files to be loaded on startup in order to extend EchoVault functionality with custom commands.

The plugins should be available in both client-server and embedded mode. For embedded mode, provide a generic function that accepts a custom []string to trigger custom plugin commands. It should return an error or a []byte representing the RESP response from the command handler.

Create eviction policies

In order to keep memory usage in control, it's essential to have eviction policies that allow us to purge keys. As part of this issue, we would like to have the following eviction policies implemented:

  1. noeviction - Do not evict any keys regardless of memory usage.
  2. allkeys-lru - Use an LRU algorithm to remove the least recently used keys.
  3. allkeys-lfy - Use an LFU algorithm to remove the least frequently used keys.
  4. volatile-lru - Use an LRU algorithm to remove the least recently used keys with the expire property set to true.
  5. volatile-lfu - Use an LFU algorithm to remove the least frequently used keys with the expire property set to true.
  6. allkeys-random - Randomly remove any of the keys.
  7. volatile-random - Randomly remove any of the keys that have the expire property set to true.
  8. volatile-ttl - Remove keys with the shortest remaining ttl.

Compatibility with Redis Clients

Improve compatibility with Redis clients to allow EchoVault to be a potential drop-in replacement for Redis. The following considerations need to be made:

  1. Handshake with Redis client.
  2. Make sure each command returns a valid RESP response.
  3. Make Pub/Sub module compatible with Redis client.

Make EchoVault Embeddable

Make EchoVault embeddable so any application can enhance its feature set with EchoVault features by simply importing EchoVault.

Bonus: Allow projects to import some modules from EchoVault. For example, import only the Append-Only engine or the Snapshot engine.

build does not work

How can I run this project and make it reproducible?

error output

cd /root; cd octavia/diskimage-create/; ./diskimage-create.sh -a amd64 -d jammy -i ubuntu-minimal -o amphora-x64-haproxy-jammy.qcow2 -s 3

# go version
go version go1.22.2 linux/amd64

# git log -p -2
commit 2528082d4158680c7b59d7e526aec2b62885d8c4 (HEAD -> main, origin/main, origin/HEAD)
Author: Kelvin Mwinuka <[email protected]>
Date:   Thu Apr 11 11:01:45 2024 +0800

    Update README.md

diff --git a/README.md b/README.md
index 33bd7bd..e581f7a 100644
--- a/README.md
+++ b/README.md
@@ -7,9 +7,6 @@
 <br/>
 [![Go Reference](https://pkg.go.dev/badge/github.com/echovault/echovault.svg)](https://pkg.go.dev/github.com/echovault/echovault)
 <br/>
-[![Discord](https://img.shields.io/discord/1211815152291414037?style=flat&label=Discord&color=%235865F2)
-](https://discord.gg/aasBFwDm)
-<br/>
 <hr/>

 <img alt="echovault_logo" src="./images/EchoVault GitHub Cover.png" width="5062" />

commit c6b246efa455caaea53a6ba7a83115db826d36e1
Merge: 9b7d93f 64bbda6
Author: Kelvin Mwinuka <[email protected]>
Date:   Sat Apr 6 02:51:27 2024 +0800

    Merge pull request #21 from EchoVault/chore/fix-race-condition

    Fixed race conditions in acl module tests
#

# make build
env CC=x86_64-linux-musl-gcc GOOS=linux GOARCH=amd64 DEST=bin/linux/x86_64 make build-server
make[1]: вход в каталог «/root/EchoVault»
CC=x86_64-linux-musl-gcc GOOS=linux GOARCH=amd64 go build -o bin/linux/x86_64/server ./cmd/main.go
# runtime/cgo
cgo: C compiler "x86_64-linux-musl-gcc" not found: exec: "x86_64-linux-musl-gcc": executable file not found in $PATH
make[1]: *** [Makefile:2: build-server] Ошибка 1
make[1]: выход из каталога «/root/EchoVault»
make: *** [Makefile:5: build] Ошибка 2

Allow configuration of consistency policy

EchoVault is currently strongly consistent. This change will aim to make this policy configurable.

At the moment, echovault has a memberlist layer and a raft layer. The memberlist layer is mainly used to establish a cluster skeleton, essentially the "chassis" of the cluster. Memberlist nodes will communicate with other nodes to ask permission to join the raft cluster using the gossip protocol. Once the gossiped message reaches the raft cluster leader, the leader will add the node to the raft cluster.

We must allow a user to configure this functionality using the config flags. The option will enable the user to forgo the raft layer and use the memberlist layer for gossip-based state propagation to allow for eventual consistency. Otherwise, the user can opt for the strong consistency of the raft layer.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.