Code Monkey home page Code Monkey logo

roadmap's Introduction

Go Reference

Gravwell Open-Source Code

This repository contains open-sourced libraries and commands developed by Gravwell.

There are a selection of Gravwell-specific libraries and tools:

  • ingest/ contains the ingest library, which is used to connect to a Gravwell indexer and upload data.
  • ingesters/ contains the source code for Gravwell ingesters.
  • generators/ is a collection of tools that generate artificial data for testing Gravwell or any other log analytics system.
  • manager/ provides a very simple init command which we use in Docker containers.
  • chancacher/ implements a caching library we use for ingesters.

There are also a few libraries which may be of use outside Gravwell-specific applications:

  • filewatch/ is a library that can monitor files on the filesystem for changes; we use this in the FileFollow ingester.
  • timegrinder/ is a timestamp extraction library we use to extract timestamps from arbitrary data
  • ipexist/ contains a library for efficiently storing and checking for the existence of an IPv4 set with high density sets.
  • winevent/ is a library which can interact with the Windows Event subsystem to extract XML rendered events.

roadmap's People

Contributors

ashnwade avatar

roadmap's Issues

Integrated Metrics

Feature Description

Set up a dedicated metrics store that can more efficiently store aggs.
Some basic aggs should just be part of ingest, ingest counts, ingest size, rate per tag, size per tag.
Allow users to create custom aggs that occur at ingest.

24h Clock

Feature Description

Allow individual users to choose which time format they prefer to work with: 12h or 24h

New Feature: Alerts

Feature Description

  • A unified view for defining and listing alerts
  • Asynchronous flows that react to a triggering event
  • Perform common configuration for multiple alerts (reduce the need for duplication)
  • Define "criticality"
  • Provide schema validation

Disable Flow nodes

Feature Description

Allow Flow nodes to be marked as disabled.
This will halt the execution of a flow without requiring the user to insert a node for blocking.

Visualizations

Feature Description

Currently visualizations are very tightly coupled. There is coupling with specific libraries and also with specific search modules. The goal is to decouple components so that there is a lot more flexibility.

  • Use different libraries for different visualizations
  • Change visualizations without having re-execute a search
  • Allow for custom renderers

List Component Refactor

Feature Description

Improve the consistency, performance, and interactions with the list page for various assets.

Considerations

Considerations for improvement include but are not limited to:

  • Filtering
  • Searching
  • Grouping
  • Admin Interactions
  • Hidden Assets
  • Mass Operations
  • Label Visibility
  • Paged Responses

Query Streaming

Feature Description

Enable live searches for up-to-date streaming results. Right now we have "near real time" results, because we re-run the searches every x seconds.

eval Improvements

Feature Description

Completed transition to a new, native eval module. This includes full support for all EV types, several builtin functions, and significant performance improvements.

General Changes

  • error recovery
  • diagnostics
  • persistent variables
  • variable initializer support
  • return
  • break/continue
  • true/false keywords
  • bool cast

Built-ins

Printing

  • printf

Hashing

  • md5
  • sha1
  • sha256
  • sha512

Encoding

  • base64
  • json
  • hex

HTML

  • htmlescape
  • htmlunescape

Logging

  • gravwell log

Math

  • abs
  • ceil
  • floor
  • log
  • log10
  • log2
  • max
  • min
  • mod
  • modf
  • pow
  • remainder
  • round
  • roundtoeven
  • sqrt
  • trunc

Random numbers

  • rand
  • randfloat
  • normal
  • [ ] seed

Paths

  • base
  • clean
  • dir
  • ext
  • isabs
  • join
  • match
  • [ ] split

regex

  • match

Strings

  • contains
  • count
  • hasprefix
  • hassuffix
  • index
  • replace
  • tolower
  • toupper
  • totitle
  • trim
  • trimleft
  • trimprefix
  • trimright
  • trimsuffix
  • trimspace

Comparison/conditional

  • in

Share write access with groups

Feature Description

Improve collaboration by allowing write access to be shared with a group.

Shareable assets that require consideration:

  • Query Library
  • Scheduled Searches
  • Flows
  • Alerts
  • Scripts
  • Playbooks
  • Kits
  • Macros
  • Extractors
  • Resources
  • Templates
  • Actionables
  • Files
  • Secrets
  • Dashboards

Assets that are checked off in this list already have write access for groups in the latest production release. This will be completed in stages.

Gravwell Command Bar

Feature Description

Gravwell allows users to create/use many different kinds of assets (dashboards, templates, queries, flows, etc). Though there are clear paths for users to find their assets, it can be inconvenient to:

  • click through to find them
  • remember their type (e.g. "was it a template or a saved query?...")
  • pivot between them

To make access to assets more convenient, this proposes adding an application-wide, multi-function search bar to the Gravwell UI.

Table Renderer Refactor

Feature Description

Replace our current table implementation with a new table renderer that is easier to maintain and extend.
Improvements may include but are not limited to:

  • better initial column sizes (allow horizontal scrolling)
  • improve column resizing
  • improve wrapping and un-wrapping table content
  • consider changes for the context menu
  • lazy loading

Vega Rendering

Feature Description

Support Vega rendering for custom renderer plugins.

WASM

Feature Description

A WASM based execution environment for more flexible hosted agents and ingesters. The system will provide an ingest and query interface via WASI1 so that users can write their own ingesters and/or query agents using the language of their choosing. Each WASM kernel will be run in a secure isolated container and provide pre-authenticated access to the ingest and query APIs.

Other type of comments

Feature Description

Other type of C-style comments inside queries.

Right now we support strict C-style block comments

/* comment */

The ask is for the other type of C-style comments:

// comment

The functionality is that the // will comment out the rest of the light, stopping at the first newline it sees

Remove websockets

Feature Description

The websockets that currently support search functionality will be removed and replaced with a REST API.
This will provide a simple and more robust implementation. This will also support future improvements with renderers and Dashboards.

Allow one Extractor to map to multiple Tags

Feature Description

Allow one extractor to map to multiple tags. This will:

  • reduce the number of extractors needed by avoiding duplication for like tags.
  • make it easier to find and update one extractor for like tags.
  • improve performance on the extractors page if there are fewer items to load.

System Configuration Interface in the GUI

Feature Description

Allow configuration for the entire Gravwell system to occur through the GUI, not just config files. Config files are hard. It can also be difficult to maintain multiple config files, especially for distributed ingesters.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.