Code Monkey home page Code Monkey logo

basenine's Introduction

Basenine

GitHub Latest Release GitHub License GitHub Workflow Tests Code Coverage (Codecov)

Schema-free, document-oriented streaming database that optimized for monitoring network traffic in real-time.

Featured Aspects

  • Has the fastest possible write speed.
  • Has a read speed that scales linearly.
  • Schema-free.
  • Only allows create and read.
  • Accepts JSON as the record format.
  • Let's you query based on JSONPath.
  • Has a rich filtering syntax for querying.
  • Defines a TCP-based protocol.
  • Has long lasting TCP connections.
  • Watches the database and streams back the new records.

Server

Run the server:

make && ./basenine -port 9099

Protocol

The database server has these connection modes:

  • Insert mode is a long lasting TCP connection to insert data into the data_*.db binary files on server's directory. A client can elevate itself to insert mode by sending /insert command.

  • Insertion filter mode is a short lasting TCP connection that lets you set an insertion filter which is executed right before the insertion of each individual record. The default value of insertion filter is an empty string.

  • Query mode lets you filter the records in the database based on a filtering syntax named BFL. Query mode streams the results to the client and is able to keep up where it left off even if the database have millions of records. The TCP connection in this mode is long lasting as well. The filter cannot be changed without establishing a new connnection. The server also streams the query progress through /metadata command to the client.

  • Single mode is a short lasting TCP connection that returns a single record from the database based on the provided index value.

  • Fetch mode is a short lasting TCP connection mode for fetching N number of records from the database, starting from a certain offset, supporting both directions.

  • Validate mode checks the query against syntax errors. Returns the error if it's syntactically invalid otherwise returns OK.

  • Macro mode lets you define a macro for the query language like http~proto.name == "http".

  • Limit mode allows you to set a hard-limit for the database size in bytes like 100000000 (100MB). The disk usage ranges between 50000000 (50MB) and 100000000 (100MB). So the actual effective limit is the half of this value.

  • Flush mode is a short lasting TCP connection mode that removes all the records in the database.

  • Reset mode is a short lasting TCP connection mode that removes all the records in the database and resets the core into its initial state.

Query

Querying achieved through a filter syntax named Basenine Filter Language (BFL). It enables the user to query the traffic logs efficiently and precisely.

http and request.method == "GET" and request.path != "/example" and (request.query.a > 42 or request.headers["x"] == "y")

Please see the syntax reference for more info.

Client

Go

Insert

// Establish a new connection to a Basenine server at localhost:9099
c, err := NewConnection("localhost", "9099")
if err != nil {
    panic(err)
}

// Elevate to INSERT mode
c.InsertMode()

// There can be many Send and SendText calls
c.SendText(`{"brand":{"name":"Chevrolet"},"model":"Camaro","year":2019}`)
c.Send([]byte(`{"brand":{"name":"Chevrolet"},"model":"Camaro","year":2020}`))
c.SendText(`{"brand":{"name":"Chevrolet"},"model":"Camaro","year":2021}`)

// Close
c.Close()

Single

// Retrieve the record with ID equals to 42 with an empty query
// The 4th argument query, is only effective in case of
// record altering helpers like `redact` are used.
// Please refer the BFL syntax reference for more info.
data, err := Single("localhost", "9099", fmt.Sprintf("%024d", 42), "")
if err != nil {
    panic(err)
}

Fetch

// Retrieve up to 20 records starting from offset 100, in reverse direction (-1),
// with query `brand.name == "Chevrolet"` and with a 5 seconds timeout.
// Returns a slice of records, first meta and the latest meta state.
data, firstMeta, lastMeta, err := Fetch("localhost", "9099", 100, -1 `brand.name == "Chevrolet"`, 20, 5*time.Second)
if err != nil {
    panic(err)
}

Query

// Establish a new connection to a Basenine server at localhost:9099
c, err := NewConnection("localhost", "9099")
if err != nil {
    panic(err)
}

// Make []byte channels to recieve the data and the meta
data := make(chan []byte)
meta := make(chan []byte)

// Clean up
defer func() {
    data <- []byte(CloseChannel)
    meta <- []byte(CloseChannel)
    c.Close()
}()

// Define a function to handle the data stream
handleDataChannel := func(wg *sync.WaitGroup, c *Connection, data chan []byte) {
    defer wg.Done()
    for {
        bytes := <-data

        if string(bytes) == CloseChannel {
            return
        }

        // Do something with bytes
    }
}

// Define a function to handle the meta stream
handleMetaChannel := func(c *Connection, meta chan []byte) {
    for {
        bytes := <-meta

        if string(bytes) == CloseChannel {
            return
        }

        // Do something with bytes
    }
}

var wg sync.WaitGroup
go handleDataChannel(&wg, c, data)
go handleMetaChannel(c, meta)
wg.Add(1)

// The first argument can be an id, an empty string or "latest"
// to start the streaming from the latest record.
c.Query("", `brand.name == "Chevrolet"`, data, meta)

wg.Wait()

Validate

err := Validate("localhost", "9099", `brand.name == "Chevrolet"`)
if err != nil {
    // err should be nil, otherwise a connection error or a syntax error
}

Macro

// Define a macro `chevy` expands into `brand.name == "Chevrolet"`
err := Macro("localhost", "9099", "chevy", `brand.name == "Chevrolet"`)
if err != nil {
    // err can only be a connection error
}

Insertion Filter

// Set the insertion filter to `brand.name == "Chevrolet" and redact("year")`
err := InsertionFilter("localhost", "9099", `brand.name == "Chevrolet" and redact("year")`)
if err != nil {
    // err can only be a connection error
}

Limit

// Set the database size limit to 100MB
err := Limit("localhost", "9099", 100000000)
if err != nil {
    // err can only be a connection error
}

Flush

// Remove all the records
err := Flush("localhost", "9099")
if err != nil {
    // err can only be a connection error
}

Reset

// Reset the database into its initial state
err := Reset("localhost", "9099")
if err != nil {
    // err can only be a connection error
}

basenine's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

basenine's Issues

[Bug] Redaction of json path using recursive descent duplicates the property name

Describe the bug
Redaction of json path using recursive descent duplicates the property name.

To Reproduce
Steps to reproduce the behavior:

  1. Query a json path using recursive descent - For example: redact("response.content.text.json()...href")
  2. See that the property name got duplicated (in the example case href property name)
{
  "user": {
    "firstName": "User",
    "lastName": "Name",
    "username": "user",
    "id": "57a98d98e4b00679b4a830b2",
    "_links": {
      "addresses": { "href": "[REDACTED]" },
      "cards": { "href": "[REDACTED]" },
      "customer": { "href": "[REDACTED]" },
      "self": { "href": "[REDACTED]" },
      "href": "[REDACTED]"
    },
    "href": "[REDACTED]"
  },
  "href": "[REDACTED]"
}

Expected behavior
Redact only the value of the property name and don't duplicate it

Screenshots
image

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.