Code Monkey home page Code Monkey logo

jredisbloom's Introduction

GitHub issues CircleCI Dockerhub codecov

RedisBloom: Probabilistic Data Structures for Redis

Forum Discord

logo

Overview

RedisBloom adds a set of probabilistic data structures to Redis, including Bloom filter, Cuckoo filter, Count-min sketch, Top-K, and t-digest. Using this capability, you can query streaming data without needing to store all the elements of the stream. Probabilistic data structures each answer the following questions:

  • Bloom filter and Cuckoo filter:
    • Did value v already appear in the data stream?
  • Count-min sketch:
    • How many times did value v appear in the data stream?
  • Top-k:
    • What are the k most frequent values in the data stream?
  • t-digest:
    • Which fraction of the values in the data stream are smaller than a given value?
    • How many values in the data stream are smaller than a given value?
    • Which value is smaller than p percent of the values in the data stream? (What is the p-percentile value?)
    • What is the mean value between the p1-percentile value and the p2-percentile value?
    • What is the value of the nᵗʰ smallest/largest value in the data stream? (What is the value with [reverse] rank n?)

Answering each of these questions accurately can require a huge amount of memory, but you can lower the memory requirements drastically at the cost of reduced accuracy. Each of these data structures allows you to set a controllable trade-off between accuracy and memory consumption. In addition to having a smaller memory footprint, probabilistic data structures are generally much faster than accurate algorithms.

RedisBloom is part of Redis Stack.

How do I Redis?

Learn for free at Redis University

Build faster with the Redis Launchpad

Try the Redis Cloud

Dive in developer tutorials

Join the Redis community

Work at Redis

Setup

You can either get RedisBloom setup in a Docker container or on your own machine.

Docker

To quickly try out RedisBloom, launch an instance using docker:

docker run -p 6379:6379 -it --rm redis/redis-stack-server:latest

Build it yourself

You can also build RedisBloom on your own machine. Major Linux distributions as well as macOS are supported.

First step is to have Redis installed, of course. The following, for example, builds Redis on a clean Ubuntu docker image (docker pull ubuntu):

mkdir ~/Redis
cd ~/Redis
apt-get update -y && apt-get upgrade -y
apt-get install -y wget make pkg-config build-essential
wget https://download.redis.io/redis-stable.tar.gz
tar -xzvf redis-stable.tar.gz
cd redis-stable
make distclean
make
make install

Next, you should get the RedisBloom repository from git and build it:

apt-get install -y git
cd ~/Redis
git clone --recursive https://github.com/RedisBloom/RedisBloom.git
cd RedisBloom
./sbin/setup
bash -l
make

Then exit to exit bash.

Note: to get a specific version of RedisBloom, e.g. 2.4.5, add -b v2.4.5 to the git clone command above.

Next, run make run -n and copy the full path of the RedisBloom executable (e.g., /root/Redis/RedisBloom/bin/linux-x64-release/redisbloom.so).

Next, add RedisBloom module to redis.conf, so Redis will load when started:

apt-get install -y vim
cd ~/Redis/redis-stable
vim redis.conf

Add: loadmodule /root/Redis/RedisBloom/bin/linux-x64-release/redisbloom.so under the MODULES section (use the full path copied above).

Save and exit vim (ESC :wq ENTER)

For more information about modules, go to the Redis official documentation.

Run

Run redis-server in the background and then redis-cli:

cd ~/Redis/redis-stable
redis-server redis.conf &
redis-cli

Give it a try

After you setup RedisBloom, you can interact with it using redis-cli.

Create a new bloom filter by adding a new item:

# 127.0.0.1:6379> BF.ADD newFilter foo
(integer) 1

Find out whether an item exists in the filter:

# 127.0.0.1:6379> BF.EXISTS newFilter foo
(integer) 1

In this case, 1 means that the foo is most likely in the set represented by newFilter. But recall that false positives are possible with Bloom filters.

# 127.0.0.1:6379> BF.EXISTS newFilter bar
(integer) 0

A value 0 means that bar is definitely not in the set. Bloom filters do not allow for false negatives.

Client libraries

Project Language License Author Stars Package Comment
jedis Java MIT Redis Stars Maven
redis-py Python MIT Redis Stars pypi
node-redis Node.JS MIT Redis Stars npm
nredisstack .NET MIT Redis Stars nuget
redisbloom-go Go BSD Redis Stars GitHub
rueidis Go Apache License 2.0 Rueian Stars GitHub
rebloom JavaScript MIT Albert Team Stars GitHub
phpredis-bloom PHP MIT Rafa Campoy Stars GitHub
phpRebloom PHP MIT Alessandro Balasco Stars GitHub
vertx-redis-client Java Apache License 2.0 Eclipse Vert.x Stars GitHub
rustis Rust MIT Dahomey Technologies Stars GitHub

Documentation

Documentation and full command reference at redisbloom.io.

Mailing List / Forum

Got questions? Feel free to ask at the RedisBloom mailing list.

License

RedisBloom is licensed under the Redis Source Available License 2.0 (RSALv2) or the Server Side Public License v1 (SSPLv1).

jredisbloom's People

Contributors

bsbodden avatar chayim avatar dengliming avatar dependabot[bot] avatar dvirsky avatar gkorland avatar joyang1 avatar mnunberg avatar sazzad16 avatar snyk-bot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

jredisbloom's Issues

Support for T-Digest data structure

Given RedisBloom/RedisBloom#285 there is the following set of commands we should enable:

  • TDIGEST.CREATE: Allocate a new histogram
  • TDIGEST.RESET: Empty out a histogram and re-initialize it
  • TDIGEST.ADD: Add a value to the t-Digest with the specified count
  • TDIGEST.MERGE: Merge one t-Digest into another
  • TDIGEST.CDF: Returns the fraction of all points added which are ≤ x.
  • TDIGEST.QUANTILE: Returns an estimate of the cutoff such that a specified fraction of the data added to the t-Digest would be less than or equal to the cutoff.
  • TDIGEST.MIN: Get the minimum value from the histogram. Will return DBL_MAX if the histogram is empty
  • TDIGEST.MAX: Get the maximum value from the histogram. Will return DBL_MIN if the histogram is empty
  • TDIGEST.INFO : Returns compression, capacity, total merged and unmerged nodes, the total compressions
    made up to date on that key, and merged and unmerged weight.

With the in-depth params in https://oss.redislabs.com/redisbloom/master/TDigest_Commands/

Issue: connect pool

redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Failed connecting to host

java.lang.NoClassDefFoundError: redis/clients/jedis/util/Pool

I got an Error like this:
java.lang.NoClassDefFoundError: redis/clients/jedis/util/Pool

my jrebloom version is 2.1.0, jedis version is 2.9.3
Client.class in io.rebloom.client like this:
import redis.clients.jedis.util.Pool . But my jedis folder is redis.clients.util, so it cannot find path(redis.clients.jedis.util.Pool)
should i change my jrebloom or jedis to a suitable version?

run cfScanDump method result :WRONGTYPE Operation against a key holding the wrong kind of value

package com.ghy.www.test9;

import com.ghy.www.f.F;
import io.rebloom.client.Client;
import io.rebloom.client.ReserveParams;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import redis.clients.jedis.JedisPoolConfig;

import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;

public class Bloom_6 {
private static JedisPool pool = new JedisPool(new JedisPoolConfig(), F.IP_1, F.PORT_1, F.TIMEOUT_1, F.PASSWORD_1);

public static void main(String[] args) {
    Jedis jedis = null;
    Client client = null;
    try {
        jedis = pool.getResource();
        jedis.flushDB();

        client = new Client(pool);

        ReserveParams params = new ReserveParams();
        client.bfReserve("mykey", 0.001, 10, params);
        client.add("mykey", "a");
        client.add("mykey", "b");
        client.add("mykey", "c");
        client.add("mykey", "d");
        client.add("mykey", "e");

        Map backupMap = new HashMap();

        Map.Entry<Long, byte[]> mykey = client.cfScanDump("mykey", 0);
        while (mykey.getKey() != 0) {
            long cursor = mykey.getKey();
            byte[] dataArray = mykey.getValue();
            backupMap.put(cursor, dataArray);
            mykey = client.cfScanDump("mykey", cursor);
        }

        jedis.flushDB();

        Set<Map.Entry> set = backupMap.entrySet();
        Iterator<Map.Entry> iterator = set.iterator();
        while (iterator.hasNext()) {
            client.cfLoadChunk("mykey", iterator.next());
        }

        System.out.println();
        Map<String, Object> mykeyInfo = client.info("mykey");
        for (String eachKey : mykeyInfo.keySet()) {
            System.out.println("key=" + eachKey + " value=" + mykeyInfo.get(eachKey));
        }

    } catch (Exception e) {
        e.printStackTrace();
    } finally {
        if (jedis != null) {
            jedis.close();
        }
        if (client != null) {
            client.close();
        }
    }
}

}

redis.clients.jedis.exceptions.JedisDataException: WRONGTYPE Operation against a key holding the wrong kind of value

what is the version of jedis ?

In the Client class, i get that you import redis.clients.jedis.util.Pool;but my project occour this Exception:java.lang.ClassNotFoundException: redis.clients.jedis.util.Pool;I am sure it's the version of jedis(2.9.0) in my project cause this problem .So,what is your version of jedis? Thx.

Unable to delete rebloom key

I am building a daily bloom filter where I make a new bloom-filter and remove the old filter via scheduler, but the delete command doesn't seem to be able to remove key.
Eg:

Client client1 = new Client("127.0.0.1", 6379);
String[] bloomFilterKeyList = {"a", "b"};
client1.addMulti("testBloom", bloomFilterKeyList);
client1.delete("testBloom");

Even after running the command testBloom key/filter still doesn't get deleted.

Issue with createFilter

Hello,

I am following the example on how to create a filter, but I am running into the error below:

redis.clients.jedis.exceptions.JedisDataException: ERR unknown command `BF.RESERVE`, with args beginning with: `specialBloom`, `0.01`, `500000`, 
	at redis.clients.jedis.Protocol.processError(Protocol.java:132)
	at redis.clients.jedis.Protocol.process(Protocol.java:166)
	at redis.clients.jedis.Protocol.read(Protocol.java:220)

Looking at the documentation here: https://oss.redislabs.com/redisbloom/Bloom_Commands/
it seems that it should work with those args --> BF.RESERVE {key} {error_rate} {capacity}

Here is how I am initializing it (I have my local redis instance on port 6379):

       redisClient = new Client("localhost", 6379);
       redisClient.createFilter("specialBloom", 500000, 0.01);

Any help would be appreciated!

return type bug

  String rep = sendCommand(conn, Command.RESERVE, name, errorRate + "", initCapacity + "").getStatusCodeReply();

will casue Byte caste long error

why add 3, return boolean[] length is 4 ???

package com.ghy.www.test9;

import com.ghy.www.f.F;
import io.rebloom.client.Client;
import io.rebloom.client.InsertOptions;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import redis.clients.jedis.JedisPoolConfig;

import java.util.Map;

public class Bloom_3 {
    private static JedisPool pool = new JedisPool(new JedisPoolConfig(), F.IP_1, F.PORT_1, F.TIMEOUT_1, F.PASSWORD_1);

    public static void main(String[] args) {

        Jedis jedis = null;
        Client client = null;
        try {
            jedis = pool.getResource();
            jedis.flushDB();

            client = new Client(pool);
            {
                InsertOptions insertOptions = new InsertOptions();
                boolean[] insert = client.insert("mykey", insertOptions, "a", "b", "c");
                for (int i = 0; i < insert.length; i++) {
                    System.out.println(insert[i]);
                }
                System.out.println();
                Map<String, Object> mykey = client.info("mykey");
                for (String eachKey : mykey.keySet()) {
                    System.out.println("key=" + eachKey + " value=" + mykey.get(eachKey));
                }
            }
            System.out.println();
            System.out.println();
            {
                InsertOptions insertOptions = new InsertOptions();
                boolean[] insert = client.insert("mykey", insertOptions, "x", "y", "z");
                for (int i = 0; i < insert.length; i++) {
                    System.out.println(insert[i]);
                }
                System.out.println();
                Map<String, Object> mykey = client.info("mykey");
                for (String eachKey : mykey.keySet()) {
                    System.out.println("key=" + eachKey + " value=" + mykey.get(eachKey));
                }
            }
            System.out.println();
            System.out.println();
            {
                InsertOptions insertOptions = new InsertOptions();
                insertOptions.nocreate();
                client.insert("nononokey", insertOptions, "x", "y", "z");
            }
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            if (jedis != null) {
                jedis.close();
            }
            if (client != null) {
                client.close();
            }
        }
    }
}

true
true
true
false

key=Expansion rate value=2
key=Number of filters value=1
key=Capacity value=100
key=Number of items inserted value=3
key=Size value=290


true
true
true
false

key=Expansion rate value=2
key=Number of filters value=1
key=Capacity value=100
key=Number of items inserted value=6
key=Size value=290


redis.clients.jedis.exceptions.JedisDataException: ERR not found

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.