Code Monkey home page Code Monkey logo

bloomfilter's Introduction

A Bloom filter is a space-efficient probabilistic data structure used to determine whether an element belongs to a set or not. The Bloom filter allows false positives (maybe in the set) but never false negatives (definitely not in set) . If you are new to bloomfilters, give Bloom Filters by Example a read.

The bloomfilter package is suitable for caching filtering, decentralized aggregation, search large chemical structure databases and many other applications. More specifically, we use this package in production with KrakenD to distributedly reject JWT tokens as it allows us to perform massive rejections with very little memory consumption. For instance, 100 million tokens of any size consume around 0.5GB RAM (with a rate of false positives of 1 in 999,925,224 tokens), and lookups are completed in constant time (k number of hashes). These numbers are impossible to get with a key-value or a relational database.

Implementations

This repository contains several bloomfilter implementations that you can use to solve different distributed computing problems. The solution starts from an optimized local implementation that adds rotation, RPC coordination, and generic rejecters. The packages are:

  • bitset: Implementations of bitsets for basic sets.
  • bloomfilter: Optimized implementation of the bloomfilter.
  • rotable: Implementation over the BF with 3 rotating buckets.
  • rpc: Implementation of an RPC layer over rotable.
  • krakend: Integration of the rpc package as a rejecter for KrakenD

bloomfilter's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

bloomfilter's Issues

What happens on a restart?

What happens when a krakend pod with auth/revoker setup. Gets restarted.
To my understanding as bloomfilter is in memory only. All data will get purged.

How do i ensure that on rollout/restart, blocked token doesn't become valid again?

error on adding bloomfilter

hello
i am using bloomfilter with krakend. it is work well but after few days raise an error on adding token to bloom filter with message "error on adding bloomfilter: connection is shut down".

and all the other adds after it, raise this message and bloom filter not work properly.

whats the problem .... ?

thank you .

image

Bloom filter unused or expired tokens clearing or garbage collector

Is there currently any way to clear or to remove old tokens, for example tokens which are already expired, from the bloom filter? I didn't find any notes or documentation about this. It would be nice to have a script which might be running on regular times to remove all the tokens based on some criteria e.g where the date is expired.

Bloomfilter Server pass to use json codification instead Gob one

Is your feature request related to a problem? Please describe.
Bloomfilter server is encoded using gob as the net/rpc package is using it, so to revoke a token we only can use a client in Golang.

Describe the solution you'd like
Change the encoding to json with net/rpc/jsonrpc.

Describe alternatives you've considered
How can I generate a Java client to call bloomfilter server instead.

Question: how use with krakend?

Could you give us an example of using with krakend? I need to filter an array of objects by the value of one of the fields. Is it possible?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.